Automatic benchmark generation suite

June 24, 2026 · View on GitHub

Suite to generate benchmarks across different WaterLily versions, Julia versions (using juliaup), backends, cases, and cases sizes using the benchmark.sh script. Profiling tools are also provided, see README_profiling.md.

TL;DR

Default benchmark for quick validations on the TGV and Jelly test cases.

sh benchmark.sh
julia --project compare.jl

NOTE: For Ubuntu users: The default shell (/bin/sh) in Ubuntu points to dash, and not bash. Executing the sh benchmark.sh calls dash and results in a [Bad Substitution Error](benchmark.sh: 2: Bad substitution). The simple workaround is to call the shell script directly ./benchmark.sh followed by the desired arguments. This applies to all the examples below.

NOTE: For Windows users: Use git-bash to run the benchmarks. Run git-bach or open Git Bash from the Windows search bar and then you can follow the other instructions in this readme.

Detailed usage example

sh benchmark.sh -v "release 1.11" -t "1 4" -b "Array CuArray" -c "tgv jelly" -p "6,7 5,6" -s "25 25" -ft "Float32 Float64"
julia --project compare.jl --data_dir="data" --plot_dir="plots" --patterns="tgv jelly" --sort=1

runs both the TGV and jelly benchmarks (-c) using the current WaterLily version available in $WATERLILY_DIR, using 2 different Julia versions (latest release version and latest 1.11, noting that these need to be available in juliaup), 3 different backends (CPUx01, CPUx04, CUDA). Note that $WATERLILY_DIR needs to be available in the environmental variables, or otherwise the argument -wd "my/waterlily/dir" can be specified. The cases size -p, number of time steps -s, and float type -ft are bash (ordered) arrays which specify each benchmark case (respectively). They must either be equally sized to -c (one value per case) or be a single value that is propagated to every case; omitted ones use their defaults. In this example, the TGV case is run with Float32 precision and the jelly case in Float64. The default benchmarks launch (sh benchmark.sh) is equivalent to:

sh benchmark.sh -wd "$WATERLILY_DIR" -w "" -v "release" -t "4" -b "Array CuArray" -c "tgv jelly" -p "6,7 5,6" -s "25 25" -ft "Float32 Float32"

Note that -w or --waterlily can be used to pass different WaterLily versions by using commit hashes, tags, or branch names, eg. -w "master v1.2.0". An empty -w argument will benchmark the current state of $WATERLILY_DIR.

Benchmarks are then post-processed using the compare.jl script. Plots can be generated by passing the --plot_dir argument. Note that --patterns can be passed to post-process only certain benchmarks. Alternatively, benchmark files can also be passed directly as arguments with

julia --project compare.jl --plot_dir="plots" $(find data/ \( -name "tgv*json" -o -name "jelly*json" \) -printf "%T@ %Tc %p\n" | sort -n | awk '{print \$7}')

Usage information

The accepted command line arguments are (parenthesis for short version):

  • Backend arguments: --waterlily(-w), --waterlily_dir(-wd), --data_dir(-dd), --versions(-v), --backends(-b), --threads(-t), --update(-u). Respectively: List of WaterLily git hashes/tags/branches to test, WaterLily.jl directory, path to store the benchmark data, Julia version, backend types, number of threads (when --backends contains Array), and whether to update the Julia environment before each run (Pkg.develop $WATERLILY_DIR + Pkg.update). --update accepts true/false/1/0 and defaults to false; set -u true (or -u 1) when the WaterLily/Julia version changes between runs. The version/backend/threads arguments accept a list of different parameters, for example: 
    -w "fae590d e22ad41" -v "1.8.5 1.9.4" -b "Array CuArray" -t "1 6"
    would generate benchmark for all these combinations of parameters.
  • Case arguments: --cases(-c), --log2p(-p), --max_steps(-s), --ftype(-ft). The --cases argument specifies which cases to benchmark, and it can be again a list of different cases. The name of the cases needs to be defined in cases.jl, for example tgv or jelly. The current available cases are "tgv sphere cylinder donut jelly". Hence, to add a new case first define the function that returns a Simulation in cases.jl, and then it can be called using the --cases(-c) list argument. Case size, number of time steps, and float data type are then defined for each case (-p, -s, -ft, respectively). Each of -p, -s, -ft may either provide one value per case (an array matching the length of -c), or a single value that is broadcast to every case. Omitted arguments use the defaults at the top of benchmark.sh: the per-case size map DEF_LOG2P (currently tgv→6,7, jelly→5,6, sphere→3,4, cylinder→4,5) and the uniform DEF_MAXSTEPS (25) and DEF_FTYPE (Float32). So -c "tgv sphere" -ft "Float64" runs tgv at 6,7 and sphere at 3,4, both in Float64. A case with no entry in DEF_LOG2P (e.g. donut) errors on an omitted -p, with a hint to add it there or pass -p explicitly; -s/-ft always default. Omitting -c runs only the default sweep (tgv jelly).

The following command

sh benchmark.sh -v "release" -t "1 3 6" -b "Array CuArray" -c "tgv sphere" -p "6,7,8 5,6" -s "10 100" -ft "Float64 Float32"

would allow running benchmarks with 4 backends: CPUx01 (serial), CPUx03, CPUx06, GPU-NVIDIA. Additionally, two benchmarks would be tested, tgv and sphere, with different sizes, number of time steps, and float type, each. This would result into 1 Julia version x (3 Array + 1 CuArray) backends x (3 TGV sizes + 2 jelly sizes) = 20 benchmarks.

Benchmarks are saved in JSON format with the following nomenclature: casename_sizes_maxsteps_ftype_backend_waterlilyHEADhash_juliaversion.json. Benchmarks can be finally compared using compare.jl as follows

julia --project compare.jl benchmark_1.json benchmark_2.json benchmark_3.json ...

or by using pattern syntax

julia --project compare.jl --data_dir="data/benchmark" --patterns="tgv*CPU"

for which only TGV benchmarks on a CPU backend found in the "data/benchmark" directory would be processed. The following syntax would produce equivalent results:

julia --project compare.jl $(find data/benchmark -name "tgv*CPU.json" -printf "%T@ %Tc %p\n" | sort -n | awk '{print \$7}') --sort=1

by taking the tgv JSON files, sort them by creation time, and pass them as arguments to the compare.jl program. Multiple ppaterns can also be specified with --patterns="tgv jelly" for example.

The --speedup_base="<backend>,<waterlily hash/ref>,<julia version>" argument (or a subset of it, ie. "<backend>,<julia version>") can be passed to reference speed-ups: speedup_x = time(benchmark_<backend>) / time(benchmark_x). Tokens are matched case-sensitively against the tags stored in the JSON files, so use CPUx04 (not cpux04); if no match is found the script prints the available (Backend, WaterLily ref, Julia, FP) quadruples and the unmatched tokens. The --sort=<1 to 12> argument can also be used when running the comparison. It will sort the benchmark table rows by the values corresponding to the column index passed as argument. --sort=1 corresponds to sorting by backend alphabetically; --sort=7 sorts by minimum time; --sort=11 sorts by Δ; --sort=12 sorts by speedup. The speedup baseline row is highlighted in blue, and the fastest run per backend is highlighted in green. The --gc (or --gc=1) flag shows the GC fraction column, which is hidden by default. Last, a --backend_color=<colorscheme> can be passed to use a certain color scheme if plotting results (ie. passing the --plot_dir=<plot_dir> argument).

Measurement methodology

Each benchmark runs sim_step! for max_steps iterations (default 25) as a single evals=1 unit, then repeats that unit samples=5 times. The reported table shows:

  • Min [s]: minimum across samples — used as the robust point estimate (insensitive to OS jitter / GC) for speedup and cost-per-DOF.
  • Med [s] / Max [s]: expose the spread. A large gap between Min and Max flags a noisy run — rerun or widen the warmup if you see this consistently.
  • Alloc [k]: number of allocations in the minimum-time sample, divided by 1000.
  • Cost [ns/DOF/dt]: min_time / DOF / max_steps, normalized cost per cell per time step.
  • Δ [%]: cost delta against the same-backend reference row matching the speedup baseline's (WaterLily ref, Julia, FP). The reference row itself prints -. Useful to compare WaterLily versions, Julia versions, or precisions while holding the backend fixed.
  • Speedup: time(speedup_base) / min_time. The speedup baseline defaults to the first row; set explicitly with --speedup_base.
  • GC [%] (hidden by default; show with --gc): GC fraction of the minimum-time sample.

Each case is preceded by 50 warmup steps to absorb the JIT tail and let slow transients (e.g. TGV viscous decay) settle into quasi-steady state, and GC.gc() is called before the run; gcsample=true triggers GC between samples so per-sample GC bias is minimized.

Because samples are invoked on the same Simulation object, late samples run from a later physical time than early ones — for transient cases (e.g. sphere wake, jelly gait) this shows up as spread. Treat per-row Δ within ±5% as noise unless the Min/Max spread is tight.