FLiT Command-Line

Prev | Table of Contents | Next

FLiT comes with a command-line tool called flit. This command-line tool is simply a symbolic link to flit.py. In the repository, it is located in scripts/flitcli/flit.py. When installed, it is installed in <DESTDIR><PREFIX>/share/flit/scripts/flit.py with the symbolic link at <DESTDIR><PREFIX>/bin/flit.

This command is split up into many subcommands. Most of it is self documented. For more information, simply call:

flit --help

Core functionality subcommands:

• flit help: Display help for a specific subcommand
• flit init: Initializes a flit test directory for use
• flit update: Updates the Makefile based on flit-config.toml
• flit make: Run flit tests locally and add results to the database
• flit import: Imports test results into an SQLite3 database
• flit bisect: Assign variability blame to files and functions

Extra functionality subcommands:

• flit disguise: Anonymizes project-specific data from text files
• flit experimental: Access to experimental features

Possibly of interest:

• Adding More Subcommands

flit help

This can display the help documentation for a specific subcommand. This is just a convenience subcommand.

flit help init

is identical to

flit init --help

This is the preferred method for getting the latest documentation for the flit command-line tool. The documentation within the tool is more likely to be up to date than this markdown description.

flit init

Initializes a flit test directory for use. It will initialize the directory by copying the default configuration file into the given directory. The config file is called flit-config.toml.

To see how to modify flit-config.toml, see the documentation for FLiT Configuration File.

There are a few other files copied over:

• flit-config.toml: the configuration file
• custom.mk: a custom file that is included at the end of Makefile
• Makefile: an autogenerated Makefile using flit-config.toml
• main.cpp: a minimal main function. No modification necessary, but you may if you want. You are free to rename this file as necessary, just make sure it is included in custom.mk in the SOURCE variable.
• tests/Empty.cpp: This is a template for how to write tests in this framework

If you want to play with the litmus tests in this directory, you can pass the --litmus-tests flag which will copy over the litmus tests into the tests directory for use.

Note: Only the litmus tests will be in this folder, the Empty.cpp file is not copied.

flit update

Updates the Makefile based on flit-config.toml. The Makefile is autogenerated and should not be modified manually. If there are things you want to replace or add, you can use custom.mk which is included at the end of the Makefile. So, you may add rules, add to variables, or override variables.

You do not actually need to call this explicitly since the Makefile will automatically call flit update to update itself if flit-config.toml has changed. You should only call it directly if you want to manually examine the autogenerated Makefile.

flit make

Run the full set of tests through the autogenerated Makefile. You could do this manually, but this does a few things conveniently for you, such as

1. Do the build for the run in parallel, automatically choosing a good parallel job number
2. Perform the run, by default with only one job, since in general we cannot assume that your tests are reentrant.
3. Import the results into the database.

Again, you can do these things manually and have more control. But for most users, this will be the most direct and convenient way to run locally.

If you use the Makefile directly, which is totally fine, then there are a few things to know.

make help

will output help documentation on what targets are available and what they mean.

The output from the Makefile will be short by default. You can output all of the details by defining VERBOSE=1 or VERBOSE=true either as an argument to make or as an environment variable.

make VERBOSE=1 ...

flit import

Imports flit results into the database configured in flit-config.toml. It can either import a CSV file created from a FLiT test executable (using the --output flag) or it can import another SQLite3 database file created from another call to flit import.

The following is an example of how to do a full run of the FLiT litmus tests and import them into a database file.

flit init --directory litmus-test-run --litmus-tests
cd litmus-test-run
rm tests/Empty.cpp
make run -j8
flit import --new-run results/*.csv

These commands will populate by default a file called results.sqlite within the created litmus-test-run directory. You are free to examine the results using the sqlite3 interactive tool or any other method you have for running queries on an SQLite3 database.

There is an option to specify a different database file than the one in flit-config.toml. Here are three reasons you may want to use this option:

1. run flit import outside of an initialized flit test directory
2. create a database file different from the one in flit-config.toml
3. use the import functionality on a system that does not have the python toml package installed

If one of those applies to you, then you can use the --dbfile argument to flit import.

flit import --dbfile temporary.sqlite backup/results/*.csv

flit bisect

After FLiT identifies compilations that cause some tests to exhibit variability, one may want to investigate further and understand where the compiler introduced overly aggressive optimizations.

The flit bisect tool is capable of assigning blame to individual source files and can often go further to assign blame to individual functions within the blamed source files. You can run flit bisect directly giving it a specific compilation, precision, and test case, or you can tell it to automatically run for all differences in a given SQLite3 database.

FLiT Bisect depends on binutils to extract symbols, filenames, and line numbers from compiled object files.

Here is an example of giving a single test case (named subnormal) known to show variability:

flit init --directory litmus-test-run --litmus-tests
cd litmus-test-run
flit bisect \
  --precision double \
  "g++ -O3 -funsafe-math-optimizations" \
  subnormal

Note: if the compiler given (e.g., g++) is found in the compiler section of flit-config.toml, then the fixed_compile_flags specified there will be used when compiling object files from this compilation under test. However, the fixed_link_flags will not be used since the link step is performed by default with the baseline compilation's compiler. This behavior can be overridden with the --ldflags, --use-linker, and --add-ldflags options.

And here is an example of giving a full SQLite3 database

flit init --directory litmus-test-run --litmus-tests
cd litmus-test-run
rm tests/Empty.cpp
make run -j8
flit import --new-run results/*.csv
flit bisect --auto-sqlite-run results.sqlite --parallel 8 --jobs 8

In this mode, the compiler type will be inferred by matching the compiler in the sqlite database with flit-config.toml. The same is true for a single bisect invocation if the given compiler is found within flit-config.toml.

Call flit bisect --help for more documentation.

flit experimental

There may or may not be any experimental features currently available. These features are available by the flit experimental command. Please call flit experimental to see the documentation of what experimental features are available.

You may also read the documentation on experimental features.

flit disguise

This command disguises (a.k.a., anonymizes) text or log files. Fields that can be disguised are source file names, test names, and function names. To accomplish this feat, a mapping csv file can either be provided by the user (with --disguise-map) or will be autogenerated as "disguise.csv" (default behavior) from the contents of the Makefile and the object files for the baseline compilation. The mapping is applied as a very simple search and replace. To undo the disguise, use the --undo flag either allowing the default "disguise.csv" file to be used or specifying one with --disguise-map.

Adding More Subcommands

The FLiT command-line structure is extremely modular. If you create a file with the correct naming scheme, put it in the correct directory, and implement three things, it will automatically be picked up and become available as a subcommand from the flit command-line tool.

See the flit command-line README.md for more information.

Prev | Table of Contents | Next