Benchmarking vkgltfrenderer

For contributors and agents. How to measure performance — headless timing and the scripted sequencer. The exact flags, log fields, and comparison rules are owned by the code (src/benchmarking.cpp, utils/benchmark/); this doc explains the workflow.

Two workflows:

1. Headless timing (recommended) — post-warmup render throughput for N frames at 1 or 5 samples per pixel; compare logs between builds.
2. Scripted sequencer — multi-step .cfg matrix with per-stage GPU profiler stats (optional, heavier).

Headless timing (simple)

Render 500 frames (or any count) with path tracing. Set --frames and --maxFrames to the same value so every app frame accumulates samples (default maxFrames is already 500).

One-off CLI

./vk_gltf_renderer --headless --size 1920 1080 \
  --scenefile shader_ball.gltf \
  --hdrfile std_env.hdr \
  --frames 500 --maxFrames 500 \
  --ptSamples 1 --ptAdaptiveSampling 0 \
  --renderSystem 0 --envSystem 1

While rendering, periodic progress lines confirm the run is not stuck (every 50 frames or 5 seconds):

HEADLESS_START frames=500 maxFrames=500 ptSamples=1
HEADLESS_PROGRESS app_frame 50/500 (10%) elapsed_ms=1234.5 ms_per_frame=24.69
...
HEADLESS_SUMMARY frames=500 maxFrames=500 ptSamples=1 effective_spp=500 measured_effective_spp=499 resolution=1920x1080 wall_ms=12320.987 ms_per_frame=24.691 total_wall_ms=12345.678 total_ms_per_frame=24.691 warmup_frames=1 measured_frames=499 throughput_MSps=84.0 spp_per_sec=40.49
BENCHMARK_JSON {"schema":1,"type":"headless_summary",...}

• app_frame — headless loop index (--frames).
• In headless mode, main() raises --maxFrames to at least --frames if you set it lower, so every app frame can accumulate samples during timing runs.
• wall_ms — measured post-warmup render time. The first completed frame is excluded so one-time setup such as shader specialization is not charged to throughput.
• total_wall_ms — full headless render-loop wall time, including warmup and any synchronous setup.
• ms_per_frame — wall_ms / measured_frames.
• effective_spp — total final-image accumulation: min(frames, maxFrames) × ptSamples.
• measured_effective_spp — accumulation covered by the measured post-warmup window.
• throughput_MSps — measured mega pixel-samples per second (resolution × measured_effective_spp / wall_s / 10⁶; higher is faster).
• spp_per_sec — measured measured_effective_spp / wall_s at this resolution (how fast quality accumulates; higher is faster).

Repeat with --ptSamples 5 for 5 spp per frame (effective_spp=2500).

Batch helper (1 spp and 5 spp)

python utils/benchmark/benchmark.py headless --scene resources/shader_ball.gltf --frames 500 --spp 1 5

Logs: utils/benchmark/output/headless_<scene>_spp<N>.log CSV: utils/benchmark/output/headless_results.csv

Compare two builds (same scene/spp, different executable):

python utils/benchmark/benchmark.py headless-compare \
  utils/benchmark/output/headless_shader_ball_spp1_baseline.log \
  utils/benchmark/output/headless_shader_ball_spp1_candidate.log

(Use one log from build A and one from build B.)

---

Scripted sequencer (optional)

Scripted benchmarks measure GPU frame time and VRAM usage across scenes, cameras, and renderer settings. Output is designed for regression testing when comparing builds or branches.

Quick start (sequencer)

Build the sample, then from the project directory:

# Fast smoke benchmark (one scene)
python utils/benchmark/benchmark.py run quick.cfg --scene resources/shader_ball.gltf --hdr std_env.hdr

# Full matrix (edit utils/benchmark/scenes.example.txt for your asset paths)
python utils/benchmark/benchmark.py run matrix.cfg \
  --scenes-file utils/benchmark/scenes.example.txt \
  --scenes-root . \
  --csv-name benchmark_results.csv

Results land in utils/benchmark/output/ (logs per scene + CSV). See also utils/benchmark/README.md.

How it works

1. --benchmark 1 turns off vsync, hides side panels (keeps a fullscreen viewport with the tonemapped image), and drains the scene load pipeline synchronously each frame. Scripted sequencer runs are interactive-window runs (minimal viewport) — they do not pass --headless; use the headless workflow above for windowless timing.
2. ElementSequencer steps through a .cfg script (SEQUENCE "name" blocks).
3. After each sequence, ProfilerManager logs ParameterSequence blocks at log level eSTATS (GPU/CPU timer averages).
4. benchmarkAdvance() records Scene and PathTracer/Rasterizer VRAM stats.
5. When the script finishes, the app closes automatically.

Benchmark script format

Example — one sequence from utils/benchmark/quick.cfg (the file defines several: Warmup, PT 1spp, Rasterizer; see quick.cfg and matrix.cfg for the full scripts):

SEQUENCE "PT 1spp"
--sequenceframes 256
--sequenceaverages 64
--sequenceresetframes 8
--renderSystem 0
--ptSamples 1
--maxFrames 1
--ptAdaptiveSampling 0
--gltfCamera 0
--updateData

Token	Meaning
SEQUENCE "..."	Starts a new measured step
--sequenceframes	Frames to run this step
--sequenceaverages	Frames averaged for profiler report
--sequenceresetframes	Warmup frames after parameter changes (0 = measure immediately)
Other --flags	Any registered CLI parameter (renderer, path tracer, tonemapper, etc.)

Path tracer note: Set --maxFrames to match --ptSamples when measuring convergence cost. Use --maxFrames 1 with --ptSamples 1 for per-frame interactive GPU time.

Comparing versions

# Baseline build
python utils/benchmark/benchmark.py run matrix.cfg --scene my_scene.gltf --csv-name baseline.csv

# Candidate build (rebuild executable first)
python utils/benchmark/benchmark.py run matrix.cfg --scene my_scene.gltf --csv-name candidate.csv

python utils/benchmark/benchmark.py compare baseline.csv candidate.csv --output diff.csv --regression-threshold-pct 5

compare marks Regression when candidate GPU time is more than N% slower than baseline or when the candidate's Scene VRAM peak exceeds the baseline by more than the VRAM threshold (default 64 MB). Negative delta % means faster. See compare_csv in utils/benchmark/benchmark_results.py for the exact rules.

Log parsing

utils/benchmark/benchmark.py reads stable BENCHMARK_JSON records first, with legacy text parsing as a fallback:

• BENCHMARK_JSON {"schema":1,"type":"headless_summary",...}
• BENCHMARK_JSON {"schema":1,"type":"sequence_memory",...}
• ParameterSequence N "name" = { Timer "..."; GPU; avg ...; CPU; avg ...; }
• BENCHMARK_ADV N { Memory Scene; ... Memory PathTracer; ... }

Auto-generated log: log_<executable>.txt next to the binary (Logger behavior).

Tips

• Use fixed resolution (--size 1920 1080, set in utils/benchmark/benchmark_runner.py) for comparable numbers.
• Disable validation layers for performance runs (--vvl off by default in Release).
• Add scenes to utils/benchmark/scenes.example.txt (name + relative path per line).
• Multi-camera scenes: add sequences with --gltfCamera 0, --gltfCamera 1, etc.
• After large setting changes, use --updateData or --resetFrame (no value; bool triggers) and non-zero --sequenceresetframes.