PHP profiler overhead benchmarks

The numbers cited in docs/comparison.md are generated by the scripts in this directory. The goal is a like-for-like measurement of target-process wall time under each profiler at a common configuration, on the same hardware in the same shell session.

What we measure

For each (profiler, benchmark) pair we record the target script's own self-reported wall time (microtime(true) delta), printed to STDERR. We use the script's own timing rather than time(1) so we don't see the profiler's own startup cost in the number.

This is the "what does my application's request latency look like with the profiler turned on?" metric. It is not the same as "total CPU across the system" — for external samplers (reli, phpspy) the sampler process has its own CPU cost on top of the target's wall time.

Files

Path	Purpose
scripts/fib.php	Recursive fibonacci. High call frequency, shallow stack — stresses per-call hook cost.
scripts/sort.php	Repeated sort() over a 100k-int array. Almost all time is in C internals.
scripts/mandelbrot.php	Pure-CPU inner loops, zero PHP function calls. Per-call hooks have nothing to fire on, so per-call-cost profilers see no overhead from this workload.
scripts/depth.php	Synthesised tail recursion at variable depth. Lets us vary stack depth at fixed total work.
scripts/run_with_delay.php	Wrapper that sleeps $BENCH_ATTACH_DELAY_MS before requiring the inner script. Used so that external samplers have time to attach before timed code starts.
profilers/*.php	auto_prepend_file wrappers that turn each in-process profiler on for the entire script run.
run.sh	Orchestrator for in-process profilers (xdebug / xhprof / excimer / spx / datadog). Uses php -n plus per-profiler -d flags so the baseline run is genuinely unaffected by anything in php.ini.
run-external.sh	Orchestrator for external samplers (phpspy / reli).
summarize.php	Reads the CSV outputs, computes the median per (config, bench), and prints the result table relative to baseline.
results/raw.csv	Per-run wall-time output from run.sh.
results/external.csv	Per-run output from run-external.sh.
results/summary.txt	Median + multiplier table fed into docs/comparison.md.

Running

# Run from repo root.
RUNS=3 bash docs/bench/run.sh > docs/bench/results/raw.csv 2>docs/bench/results/run.log
RUNS=3 bash docs/bench/run-external.sh > docs/bench/results/external.csv 2>docs/bench/results/external.log

# Long targets (1-3s baseline, n=10) — better signal-to-noise, but
# only for sampling/light tools (full instrumentation would take
# 10+ minutes per cell).
RUNS=10 bash docs/bench/run-long.sh > docs/bench/results/long.csv 2>docs/bench/results/long.log

# JIT-on suite (target opcache + tracing JIT, plus reli's own PHP
# 8.5 JIT for the relijit row). Two-tier: light tools at long, heavy
# tools at short. SPX is excluded under JIT — see "Profiler / JIT
# compatibility quirks" below.
RUNS_LIGHT=10 RUNS_HEAVY=3 bash docs/bench/run-jit.sh > docs/bench/results/jit.csv 2>docs/bench/results/jit.log

php -n docs/bench/summarize.php docs/bench/results/raw.csv docs/bench/results/external.csv > docs/bench/results/summary.txt

Setup pitfalls

A reproduction on Ubuntu / Debian default kernel settings will silently produce wrong numbers without these two checks.

• ptrace_scope must be 0 (or the bench user must have CAP_SYS_PTRACE). On Ubuntu the default is 1, which lets a process ptrace only its own descendants. phpspy attaches by PID — to a sibling — so process_vm_readv returns EPERM on every sample. The bench scripts redirect phpspy stderr to /dev/null, so you don't see the copy_proc_mem: Failed to copy executor_globals; err=Operation not permitted message; instead you get a 0-byte /tmp/bench-phpspy.out and a "phpspy at 1.0× baseline" cell that's actually a baseline run with a no-op sampler. Fix:

  sudo sh -c 'echo 0 > /proc/sys/kernel/yama/ptrace_scope'
  

  Reverts on reboot. reli's --cmd args form forks the target, so reli itself is unaffected — only phpspy cells fail.

• Sanity-check actual sample counts before trusting any cell. The same silent failure mode applies to anything else that pokes another process's address space, and external samplers don't flag dropped samples in the output stream. After a run, check wc -l /tmp/bench-phpspy.out is non-zero, or count rbt samples via php ./reli rbt:analyze < /tmp/bench-reli.rbt | grep 'samples'. The cpu-rates.csv and cpu-samples.csv outputs already include captured-sample counts per cell — non-zero is the basic gate.

JIT on vs JIT off

The original suite (run.sh / run-long.sh) uses php -n which ignores php.ini and conf.d — so opcache isn't loaded and JIT is off in every cell. That gives a clean tools-only comparison but is not how a typical production CLI / FPM is configured.

run-jit.sh runs the same workloads with opcache + tracing JIT explicitly enabled. The picture changes in two important ways:

• Sampling tools' multiplier shrinks. The "1.27× on depth-100 long" cell from the JIT-off long suite collapses to 1.01× under JIT-on because the baseline gets ~2.7× faster while the per-sample walk cost is unchanged. Headline: at 1 ms sampling, sampling tools add ~1% target overhead under realistic JIT-on conditions.
• Full-instrumentation tools' multiplier grows. xhprof on fib-32 goes from 635× under JIT-off to 1640× under JIT-on: the same xhprof absolute time (~62 s) is compared against a baseline that JIT has made 3× faster. Tools that override zend_execute_ex (xhprof, Xdebug profile) consistently show this 3× multiplier inflation. xhprof on depth-100 reaches 1271×.

If you only cite one set of numbers, the JIT-on numbers (docs/bench/results/jit.csv) are the more production-realistic "what does this cost in a typical CLI / FPM configuration?" answer.

What we don't yet measure

• Profiler-side CPU. External samplers (reli, phpspy) move work off the target into a separate process. The numbers here only show the target half of that cost. A second pass that captures sampler-side getrusage would round it out.
• Sample-rate ceiling. We pick 1 ms as a representative sampling period. We don't yet measure what each sampler can sustain at higher rates before becoming the bottleneck.
• Memory profilers. The benchmarks here are time-axis only. Memory-axis comparison is qualitative in docs/comparison.md rather than numeric.
• Real applications. Composer install / Laravel route handling would be more representative than the synthetic benchmarks here. Adding these is on the to-do list.
• Multiple PHP versions. All measurements are on PHP 8.4 NTS on one host. Comparing to PHP 8.5, ZTS, or older versions is on the to-do list.

Reproducibility caveats

The numbers were measured on a single machine and a single PHP build. They should be read as orders of magnitude rather than precise universal constants. The directional pattern (sampling profilers ~baseline; full-instrumentation profilers can be hundreds of times slower on call-dense code; selective instrumentation lands in between) is robust across runs and is consistent with the published literature cited in the comparison doc.

Notes on individual tools

• Xdebug is configured in mode=profile. We don't measure debug mode (which is very different) or trace mode (which is closer to the function-trace data Xdebug uses for memory).
• xhprof is longxinH/xhprof 2.3.10 from PECL/apt. Wrapper turns on XHPROF_FLAGS_CPU | XHPROF_FLAGS_MEMORY.
• Excimer is configured at 1 ms wall-time sampling.
• SPX is run twice: once in default (per-call instrumentation) mode, once with SPX_SAMPLING_PERIOD=1000 (1 ms) to show what the "sampling" option does to overhead. Note that SPX's hooks still fire on every call in both modes; the option thins the recorded data.
• Datadog profiler is the OSS datadog-profiling.so from dd-library-php, with datadog.trace.enabled=0 so we measure the profiler in isolation. No data is sent anywhere. Note: the Datadog profiler is loaded as a regular PHP extension=, not zend_extension= — the sury Debian package ships extension = datadog-profiling.so in 98-ddtrace.ini. An earlier version of run.sh used zend_extension= and PHP silently failed to load the module ("doesn't appear to be a valid Zend extension" on stderr, but the script still runs). The pre-fix Datadog cells in earlier raw.csv revisions were therefore actually measuring baseline; the current scripts load it correctly.
• phpspy is built from upstream HEAD, attached at 1 ms sampling.
• reli runs from this repo on PHP 8.5 (the runner floor in composer.json is ^8.4; these benchmarks were captured on 8.5), attached to the PHP 8.4 target via the -- cmd args form so reli's startup happens before the timed code begins. The 1 ms sampling period is set with --sleep-ns 1000000.

Profiler / JIT compatibility quirks observed

While running the JIT-on suite we ran into one outright bug worth recording:

• SPX + PHP 8.4 tracing JIT = output inconsistent with source. With opcache and tracing JIT enabled, SPX (in either default or SPX_SAMPLING_PERIOD=1000 mode) produces values that don't match what the script computes, and exits in ~1 ms. Concretely: fib(25) returns 1407295 instead of 75025; fib(37) returns 11456770687 instead of 24157817. The script reports exit 0 and writes a profile, but neither the profile nor the script's printed result reflects the source code. PHP doesn't print the usual "JIT is incompatible with third party extensions that override zend_execute_ex()" warning that xhprof / Xdebug trigger, suggesting SPX hooks the engine via a path that doesn't trip PHP's JIT-disable check. Likely cause: SPX hasn't migrated to the Zend Observer API (the JIT-aware hook PHP 8.0 introduced for profilers); modern dd-trace-php moved to it, and the New Relic agent moved to it on PHP 8.0+ as a prerequisite for the JIT support it shipped at agent 10.18.0.8. The same issue is reported upstream at NoiseByNorthwest/php-spx#215, noting that just loading SPX (even with auto_start=0) is enough to corrupt JIT execution. Reproducer in docs/bench/run-jit.sh's comments. The JIT-on suite excludes SPX as a result.

For comparison, in our tests:

• Excimer + JIT: works correctly, no warning, JIT stays enabled.
• Datadog + JIT: works correctly, no warning, JIT stays enabled.
• xhprof + JIT: correct values, but PHP silently disables JIT at runtime. No warning printed.
• Xdebug profile + JIT: correct values, PHP prints the documented "JIT disabled" warning and falls back to interpreter.

The Dockerfile in this directory describes a containerised version of the same setup. It was the first attempt; we ended up using host installs because the sandbox used to author this work could not reach public container registries. The Dockerfile is left in place because it is a complete, runnable description of the install steps.