ESSTRA Utility

ESSTRA Utility is a Python script for accessing metadata embedded in binary files by ESSTRA Core.

This version of ESSTRA Utility includes features named show, rm, update and shrink as described below.

How to Use

See the following sections in the README.md file in the top-level directory for a quick overview of how to use the tool:

• How to Build and Install
• Accessing Metadata

Commands

The first argument of esstra is a command, and the second or subsequent arguments are the arguments for the command:

$ esstra <command> <arg> [<arg>...]

Supported commands in this version are as follows:

• show: outputs metadata in binary files
• rm: removes metadata from binary files
• update: updates metadata in binary files by specifying additional information about the source files
• shrink: reduces the size of binary files by removing duplication in metadata

Command "show"

A command line:

$ esstra show <binary> [<binary>...]

outputs metadata embedded in the specified binary files in YAML format. For example, passing a binary file hello built from hello.c as in Sample "hello" to the command:

$ esstra show hello

would give you an output as follows:

Headers:
  ToolName: ESSTRA Core
  ToolVersion: 0.5.0
  DataFormatVersion: 0.1.0
  InputFileNames:
  - hello.c
  BinaryFile: /home/snagao/esstra/samples/hello/hello
SourceFiles:
- Directory: /home/snagao/esstra/samples/hello
  Files:
  - File: hello.c
    SHA1: 4bbee85215cbcb6a4f1625e4851cca19b0d3f6e2
- Directory: /usr/include
  Files:
  - File: features-time64.h
    SHA1: 57c3c8093c3af70e5851f6d498600e2f6e24fdeb
  - File: features.h
    SHA1: d8725bb98129d6d70ddcbf010021c2841db783f7
  - File: stdc-predef.h
    SHA1: 2fef05d80514ca0be77efec90bda051cf87d771f
          :
        (snip)
          :

As the output of this command is in YAML format, you can parse it with any YAML processors. Below is an example of how to use the command yq with a pipe to convert the output to JSON:

$ esstra show hello | yq -oj
{
  "Headers": {
    "ToolName": "ESSTRA Core",
    "ToolVersion": "0.5.0",
    "DataFormatVersion": "0.1.0",
    "InputFileNames": [
      "hello.c"
    ],
    "BinaryFile": "/home/snagao/esstra/samples/hello/hello"
  },
  "SourceFiles": [
    {
      "Directory": "/home/snagao/esstra/samples/hello",
      "Files": [
        {
          "File": "hello.c",
          "SHA1": "4bbee85215cbcb6a4f1625e4851cca19b0d3f6e2"
        }
      ]
      "Directory": "/usr/include",
      "Files": [
        {
          "File": "features-time64.h",
          "SHA1": "57c3c8093c3af70e5851f6d498600e2f6e24fdeb"
        },
        {
          "File": "features.h",
          "SHA1": "d8725bb98129d6d70ddcbf010021c2841db783f7"
        },
        {
          "File": "stdc-predef.h",
          "SHA1": "2fef05d80514ca0be77efec90bda051cf87d771f"
        },
            :
          (snip)
            :

Please note that the show command always displays shrunk data even if the binary file's metadata itself is not shrunk.

Command "rm"

The rm command completely removes metadata embedded in binary files by ESSTRA Core during compilation:

$ esstra rm <binary> [<binary> ...]

When a binary file with metadata removed is specified for ESSTRA Utility, an error occurs as shown below:

$ esstra rm hello
* processing 'hello'...
* done.
$ esstra show hello
* section not found: '.esstra'

Command "update"

The update command in the current version attaches "license information" to each file's information in the binary files' metadata.

We believe that other kinds of information could also be helpful, such as copyright information, CVE numbers and so on. Since ESSTRA is at an early stage in development, we have developed a feature that attaches license information as a sort of feasibility study.

To attach license information, you need to prepare an SPDX tag-value file with a minimum version of SPDX 2.2 including LicenseInfoInFile: (a scanner's detected licenses) and/or LicenseConcluded: (a reviewer's concluded license) tags. Some license scanners like FOSSology and ScanCode toolkit can generate such files.

The two are stored in separate metadata fields: detected licenses go to LicenseDetected (a list) and the concluded license goes to LicenseConcluded (a single value). Keeping them apart lets a reader of the binary tell a scanner's guess from a reviewer's decision. NOASSERTION / NONE values are skipped.

Files in the binary's metadata are matched against the SPDX entries by checksum, using the strongest hash algorithm present on both sides (SHA-256 preferred over SHA-1).

A typical usage is:

$ esstra update <binary> -i <spdx-tv-file>

If you want to update two or binary files with two or more license information files at once, you can specify them all on the command line:

$ esstra update <binary> [<binary> ...] -i <spdx-tv-file> [<spdx-tv-file> ..]

On re-run (multiple SPDX tag-value files, or metadata that already carries license information), the two fields behave differently:

• Detected licenses are accumulated (appended, deduplicated). For example, if the metadata already records MIT as detected for a file and an SPDX tag-value file adds BSD-3-Clause for the same file, the result is LicenseDetected: [MIT, BSD-3-Clause].
• The concluded license is replaced (last write wins), since a file has a single concluded license.

For more details on the update command, please refer to the document Sample "hello2".

Command "shrink"

ESSTRA Core cannot avoid data duplication, which especially occurs when a binary file is built from two or more source files.

The shrink command is meant to be used in such situations. It reduces the size of the binary files by removing duplication in the metadata:

$ esstra shrink <binary> [<binary> ...]

For more details, refer to the case study document Demo of ESSTRA Usage on OpenSSL Package.

Why Duplication Occurs

Duplication occurs due to limitations in how GCC plugins work.

ESSTRA Core operates as a GCC plugin, collecting metadata from source and header files during compilation and embedding it into object files. When the linker combines these object files into a binary, the metadata is merged as-is.

Since GCC compiles each source file independently, the plugin cannot share context across files. As a result, common headers included in multiple files lead to duplicated metadata in the final binary.

To address this, ESSTRA Utility provides the shrink command, which removes redundant metadata and reduces binary size.

Common Options

The following options are common to all commands:

• -v, --verbose: Run in verbose mode.
• -s, --silent: Run in silent mode. All messages, including errors, will be suppressed.
• -S, --show-error: Output errors even when muted by the --silent option.
• -D, --debug: Output debug messages and all other types of messages.

Tests

See the tests README for instructions on running the functional tests.

License

See the LICENSE file.