Performance checks

August 11, 2026 · View on GitHub

MICM ships a small micro-benchmark used to catch hot-path regressions in the Rosenbrock Solve() function. This document explains how to run the same checks the CI uses.

test/benchmark/micm_bench.cpp — the benchmark itself. Selects one mechanism and one solver configuration, then scales the mechanism up so per-Solve() work dominates over per-call overhead. Run build/micm_bench list to print every valid combination.
test/benchmark/bench_harness.hpp — the shared timing loop, the callgrind scoping, and the table that maps a combination to a run. Each solver configuration is generated, not written out by hand.
test/benchmark/chapman_mechanism.hpp — the 7-reaction Chapman mechanism, the same system as test_chapman_integration. This is the default and the one the CI charts track.
test/benchmark/ts1_mechanism.hpp — MOZART-TS1, 210 species and 547 reactions, every one written out as ordinary micm types. Use it to see how the solver scales with mechanism size. This file is generated. Do not edit it.
test/benchmark/import_ts1.py — generates the header above. micm has no mechanism-configuration parser, so the script uses the musica Python API to read the mechanism and emits the micm calls that build it. The header is checked in, so a normal build needs neither musica nor a network. Re-run the script only when the mechanism changes:
```
pip install musica
test/benchmark/import_ts1.py          # musica ships the TS1 configuration
test/benchmark/import_ts1.py --source path/to/musica/configs/v1/ts1
```
CreateGasPhase and CreateProcesses are plain functions rather than templates, so every solver configuration shares one definition of the mechanism.
scripts/bench_micm.sh — wall-clock timing driver (noisy; use for quick local development iteration).
scripts/profile_micm.sh — callgrind driver that produces deterministic instruction counts (used by CI regression gates).
scripts/compare_micm.py — diffs two profile outputs and exits non-zero when the PR adds instructions for any matrix ordering (standard, vector1,2,4,8,128).
.github/workflows/perf-regression.yml — CI workflow that runs the profile against main on every PR.

Building the benchmark

MICM_ENABLE_BENCHMARK defaults to OFF, so a normal build never compiles the benchmark. It registers no ctest, and only the jobs that measure performance turn it on. The option depends on MICM_ENABLE_TESTS: with the tests off, the benchmark stays off.

cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D MICM_ENABLE_BENCHMARK=ON
cmake --build build --target micm_bench --parallel

The binary lands at build/micm_bench. It always holds both mechanisms.

Three workflows pass the option: benchmark-charts.yml, perf-regression.yml, and runner.yml. No other CI job compiles the benchmark.

Why every grid cell holds the same values

Both mechanisms give every cell the same concentrations, custom rate parameters, temperature, and pressure. That is deliberate.

The Rosenbrock solver carries one step size H for the whole state, and NormalizedError is a single RMS over every cell and every variable (see include/micm/solver/rosenbrock.inl). Every cell therefore takes the same internal steps whatever its contents hold. Cell-to-cell variety does not make the benchmark more realistic; it only changes how many internal steps the whole grid needs, and it couples the cells, so one hard cell can stop the entire solve.

A spread of 0.2% on the TS1 concentrations or on its temperature is enough to end a 256-cell solve in NaNDetected or Convergence Exceeded Max Steps at dt=30. Keep the cells uniform.

Wall-clock timing

Good for quick local iteration; not for accept/reject decisions because wall-clock is affected by CPU load, thermals, and turbo.

Both drivers take the same positional arguments:

BUILD_DIR  CELLS  STEPS  BACKEND  LU_TYPE  LU_ALGORITHM  MECHANISM  [MATRIX...]

Every argument has a default, but a later argument needs every earlier one. The trailing MATRIX list is variadic, so it comes last.

scripts/bench_micm.sh                                            # defaults: 10000 cells, 30 steps
scripts/bench_micm.sh build 20000 50                             # 20k cells, 50 steps
scripts/bench_micm.sh build 10000 30 cpu in-place mozart chapman vector4   # just vector4
scripts/bench_micm.sh build 10000 30 cpu in-place mozart ts1      # the TS1 mechanism

CI runs both mechanisms at the same CELLS and STEPS, so the two charts differ only in the mechanism. TS1 has 547 reactions against Chapman's 7, so it takes far longer at the same size: about 280 s for one pass over the six orderings at 10000 cells and 30 steps, against about 3 s for Chapman.

Keep CELLS a multiple of 128: a vector128 group holds 128 cells, so any other count makes that one ordering pad its last group and solve more cells than the other five. The CI counts (2000 and 10000) are not multiples of 128, so vector128 solves 2048 and 10112. Both mechanisms carry the same bias, so they stay comparable to each other.

Sample output:

mechanism = chapman; backend = cpu; LU = mozart / in-place
# mechanism=chapman
kind         best_ms
standard      676.85
vector1       676.51
vector2       428.13
vector4       354.28
vector8       292.94
vector128     213.87

Instruction counts (deterministic)

profile_micm.sh runs the benchmark under valgrind --tool=callgrind. The benchmark itself flips callgrind instrumentation on only around the timed Solve() loop (via CALLGRIND_START_INSTRUMENTATION / CALLGRIND_TOGGLE_COLLECT), so the reported instruction count covers exactly Rosenbrock Solve() and everything it calls. Instruction counts are deterministic across runs, machines, and CPU generations.

sudo apt-get install valgrind            # ships the client-request header too
scripts/profile_micm.sh                  # defaults: 2000 cells, 5 steps
scripts/profile_micm.sh build 2000 5 cpu in-place mozart chapman vector4   # just vector4

Sample output (TSV — copy into a spreadsheet if you like):

mechanism = chapman; backend = cpu; LU = mozart / in-place
# mechanism=chapman
kind              instructions
standard             611346438
vector1              613011430
vector2              406257254
vector4              314970571
vector8              263484576
vector128            220045464

Callgrind is much slower than a native run, so the defaults use smaller CELLS and STEPS. Any real hot-path change is still visible at that size.

TS1 runs at the same CELLS and STEPS. It costs much more per cell, and callgrind multiplies that again, so allow far more time: about 10 s native for one pass over the six orderings, and roughly 50 to 100 times that under callgrind.

scripts/profile_micm.sh build 2000 5 cpu in-place mozart ts1

The script leaves the raw callgrind files in $OUT (default /tmp), named cg_<mechanism>_<kind>.out. Set OUT to keep two runs apart.

Comparing two runs

Both files must come from the same mechanism. Each file carries a # mechanism=<name> marker line, and compare_micm.py exits 2 when the two markers disagree. A file written before the marker existed carries none, and the check passes.

scripts/profile_micm.sh build 2000 5 > base.txt
# ... make changes and rebuild ...
scripts/profile_micm.sh build 2000 5 > pr.txt
scripts/compare_micm.py base.txt pr.txt

Output:

mechanism: chapman

kind                 base              pr           delta   delta%
standard      510,009,297     611,346,438    +101,337,141  +19.87%  <-- regression
vector1       627,328,869     613,011,430     -14,317,439   -2.28%
vector128     227,174,334     220,045,464      -7,128,870   -3.14%
vector2       457,832,087     406,257,254     -51,574,833  -11.27%
vector4       344,510,309     314,970,571     -29,539,738   -8.57%
vector8       276,683,770     263,484,576     -13,199,194   -4.77%
...

FAIL: PR adds instructions to the hot path for: standard

Exit code is 0 when nothing regressed and 1 otherwise. Use --tolerance N to allow a small absolute instruction-count increase per ordering; the default is 0, which is the value CI passes.

Typical workflow when refactoring a hot-path function

Before you start, capture a baseline against main:

git checkout main
cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D MICM_ENABLE_BENCHMARK=ON
cmake --build build --target micm_bench --parallel
OUT=/tmp/cg-base scripts/profile_micm.sh build > /tmp/base.txt

Refactor.

Rebuild and compare:

cmake --build build --target micm_bench --parallel
OUT=/tmp/cg-pr scripts/profile_micm.sh build > /tmp/pr.txt
scripts/compare_micm.py /tmp/base.txt /tmp/pr.txt

If any ordering regressed, run callgrind_annotate /tmp/cg-pr/cg_chapman_<kind>.out on the file the profile script left behind. It gives you a per-source-line breakdown. Diff the "hot lines" between base and PR — the delta is always attributable to specific lines because instruction counts are deterministic. Separate OUT directories are what keep both sides' files available.

CI: `.github/workflows/perf-regression.yml`

The workflow runs on every PR to main:

Checks out the PR head and the base branch side-by-side.
Overlays the PR's benchmark harness onto the base checkout (so both revisions run the exact same benchmark code — otherwise, changing the benchmark itself could mask regressions).
Builds micm_bench from both.
Runs profile_micm.sh against both, once for Chapman and once for TS1, at the same CELLS and STEPS.
Runs compare_micm.py once per mechanism. It marks any matrix ordering that has strictly more instructions on the PR.

Chapman shows per-call overhead and TS1 shows how the solver scales with mechanism size, so a change can move one without moving the other. Read both comparisons.

The comparison reports; it does not gate. Each compare step carries continue-on-error: true, so a marked ordering shows as a failed step inside a passing workflow, and the PR stays mergeable. Read the step's log before you merge a change to a hot path.

To make a comparison block a merge instead, remove continue-on-error from that "Compare instruction counts" step. The comparison already runs at tolerance 0, so expect it to catch deliberate additions too; raise --tolerance in the workflow when a change adds instructions on purpose.