Replay Compilation

The GraalVM compiler can record the inputs to a compilation task, serialize these inputs into a replay file, and reproduce the compilation using the same inputs. Replay compilation is based on instrumenting the JVM Compiler Interface (JVMCI). Truffle compilations are currently not supported. It is not a goal to execute the replayed code.

This file is a manual from the user's perspective. To learn how replay compilation is implemented, start by reading ReplayCompilationSupport.java and CompilerInterfaceDeclarations.java.

Example

Recording is enabled with the option -Djdk.graal.RecordForReplay=*. The value is a method filter selecting the methods to be recorded. The syntax of method filters is explained MethodFilter.java.

The below command records and serializes every compilation into ./replay-files/replaycomp/<compile-id>.replay. The directories are created if they do not exist. To use the legacy JSON format instead, use -Djdk.graal.ReplayFileFormat=Json.

mx benchmark renaissance:scrabble -- -Djdk.graal.RecordForReplay='*' -Djdk.graal.DumpPath=$PWD/replay-files -- -r 1

It is recommended to select specific methods rather than * to avoid slowing the compiler down. The compile speed overhead for the methods in the binary format can be around 5x. The size of a typical compilation unit is around 300 kB and compression reduces it by 75%. Note that if the VM exits during an ongoing compilation, some of the replay files may be incomplete.

The mx replaycomp command finds all replay files found in a given directory (and subdirectories) and invokes the replay compiler on each. It supports the default binary .replay format and the legacy .json format. The command also accepts a path to a single file.

mx replaycomp ./replay-files

Debugging

A replayed compilation can be debugged using a standard Java debugger (see Debugging.md).

mx -d replaycomp ./replay-files

Record Crashed Compilations

Using the -Djdk.graal.CompilationFailureAction=Diagnose option, the compiler retries and records crashed compilations. The command below exercises this behavior by forcing a crash. The replay file can then be found in the replaycomp directory inside the diagnostic zip archive.

mx benchmark renaissance:scrabble -- -Djdk.graal.CompilationFailureAction=Diagnose -Djdk.graal.CrashAt=hashCode -- -r 1

It is possible to not record retried compilations with the option -Djdk.graal.DiagnoseOptions=RecordForReplay=.

When a recorded compilation ends with an exception, the type and stack trace of the exception is saved in the replay file. During replay, the launcher verifies that the replayed compilation throws an exception of the same type. Use the --verbose=true option to print the stack trace of the recorded exception.

mx replaycomp --verbose=true ./replay-files

Replay with JVM Arguments

JVM arguments, including compiler options, can be passed directly to the replaycomp command.

mx replaycomp -Djdk.graal.Dump=:1 ./replay-files

Any -ea, -esa, and -X arguments from the command line are passed to the JVM as well.

Jargraal vs. Libgraal

Jargraal can replay both jargraal and libgraal compilations. Libgraal can replay only libgraal compilations.

To replay on libgraal, build libgraal first. Then, pass the --libgraal argument to mx replaycomp, which invokes the native launcher. With the below commands, all replay related processing (including replay-file parsing) is performed by libgraal code.

mx -p ../vm --env libgraal build
mx replaycomp --libgraal ./replay-files

It is possible to specify a different JDK for the replay with the --jdk-home argument. The below command runs replay using the libgraal image that is part of the built JDK in $GRAALVM_HOME.

GRAALVM_HOME=$(mx -p ../vm --env libgraal graalvm-home)
mx replaycomp --jdk-home $GRAALVM_HOME ./replay-files

Replay Options

--verbose=true prints additional information for every compilation, including:

• the system properties of the recorded compilation (the options include the VM command from the recorded run),
• the final canonical graph of the recorded/replayed compilation,
• the stack trace of the exception thrown during the recorded/replayed compilation.

mx replaycomp --verbose=true ./replay-files

--compare-graphs=true compares the final canonical graph of the replayed compilation to the recorded one, which is included in the replay file. If there is a mismatch, the command exits with a non-zero status.

mx replaycomp --compare-graphs=true ./replay-files

If the replayed compilation diverges from the recorded one, the compiler may query JVMCI information that is not recorded in the replay file. The default behavior is to return default values, which may not cause a compiler crash. The -Djdk.graal.ReplayDivergenceIsFailure=true argument prevents using default values and instead causes a crash.

mx replaycomp -Djdk.graal.ReplayDivergenceIsFailure=true ./replay-files

Performance Counters

When replaying with the --benchmark command on the AMD64 Linux platform, the replay launcher can count hardware performance events via the PAPI library. To enable this, it is necessary to set up PAPI and build the optional PAPI bridge library. Note that recent architectures may not be supported; see the list here: https://github.com/icl-utk-edu/papi/wiki/Supported-Architectures.

PAPI Setup

PAPI can usually be installed with the package manager (papi-devel on Fedora, libpapi-dev on Ubuntu). The PAPI bridge links against the PAPI available on the system.

To monitor hardware events, it may be necessary to lower the system's restrictions for accessing hardware performance counters like shown below.

sudo sysctl kernel.perf_event_paranoid=-1

Additionally, the performance monitoring library (libpfm) may fail to select the appropriate performance monitoring unit (PMU). The selection can be forced to amd64 using an environment variable.

export LIBPFM_FORCE_PMU=amd64

To discover the available counters, use the papi_avail and papi_native_avail commands, which are part of the PAPI installation. Verify that a particular event like PAPI_TOT_INS (retired instruction count) is counted using the papi_command_line utility.

papi_avail
papi_native_avail
papi_command_line PAPI_TOT_INS

PAPI Bridge

The below command builds the PAPI bridge library using the PAPI library available on the system.

ENABLE_PAPI_BRIDGE=true mx build --dependencies PAPI_BRIDGE

The launcher accepts a comma-separated list of event names. The event counts are collected for every replayed compilation separately and the per-iteration totals are reported.

ENABLE_PAPI_BRIDGE=true mx replaycomp ./replay-files --benchmark --event-names PAPI_TOT_INS

With --benchmark --results-file=RESULTS_FILE, the launcher writes a JSON array. Each replayed compilation produces a compilation record with identifying fields, metrics, and optional PAPI event counters. Each benchmark iteration also ends with an iteration_total record containing the aggregate metrics and events for that iteration.