face-detect.cpp comparative benchmark

Generated by scripts/bench_compare.py (N=20, warmup excluded). speedup = reference / ggml (>1 means ggml is faster). Absolute numbers are machine-specific.

• Machine: AMD Ryzen 9 9950X3D 16-Core Processor (Zen5, 2 CCDs)
• ggml side: facedetect-cli bench, device cpu, build -DGGML_NATIVE=OFF (AVX2/FMA + tinyBLAS)
• Reference: CPU onnxruntime (intra-op = threads, inter-op = 1), cv2.setNumThreads(threads), matched to the ggml --threads
• Versions: insightface 1.0.1, onnxruntime 1.27.0, opencv 4.13.0, numpy 2.4.6
• Image: tests/fixtures/face_a.jpg

insightface FaceAnalysis.get ALSO runs the genderage + 2d106/1k3d68 landmark heads, so the full-pipeline reference does strictly more work than the ggml detect->align->embed pipeline (noted in the table).

The change: default thread count 1 -> min(hw, 8)

The process-global backend used to default to 1 thread (fd::global_backend() created Backend(1)), below even ggml's own default of 4. These conv/matmul forwards are compute-bound and scale near-linearly to a handful of threads, so every library/CLI consumer that did not explicitly pass --threads was paying a ~3.3x latency tax. The default is now min(hardware_concurrency, 8) with a FACEDETECT_THREADS env override; fd::set_num_threads() / CLI --threads still win at runtime. Parity is unchanged (ggml matmul reduction order is thread-count-independent; detector boxes/landmarks <= 1 px and embedding cosine 1.000000 hold at any thread count - re-gated at 8 threads).

Results - matched 8 threads (the new default)

Pipeline	ggml ms/image	reference ms/image	speedup (ref/ggml)	note
buffalo_l pipeline (detect+align+embed)	201.9	101.2	0.50x	ref FaceAnalysis also runs genderage+landmark heads
ArcFace recognizer (embed only)	44.6	13.2	0.30x	ok
YuNet+SFace pipeline (detect+align+embed)	43.7	13.8	0.32x	ok
SFace recognizer (embed only)	12.3	5.4	0.43x	ok
buffalo_l genderage (detect+analyze)	159.2	29.3	0.18x	ok
buffalo_l detect (SCRFD only)	153.9	26.0	0.17x	ok

Results - matched 1 thread (the prior default, for reference)

Pipeline	ggml ms/image	reference ms/image	speedup (ref/ggml)	note
buffalo_l pipeline (detect+align+embed)	667.9	204.4	0.31x	ref FaceAnalysis also runs genderage+landmark heads
ArcFace recognizer (embed only)	155.3	51.6	0.33x	ok
YuNet+SFace pipeline (detect+align+embed)	149.7	36.1	0.24x	ok
SFace recognizer (embed only)	47.6	18.4	0.39x	ok
buffalo_l genderage (detect+analyze)	508.7	120.1	0.24x	ok
buffalo_l detect (SCRFD only)	515.0	118.6	0.23x	ok

Pass 2: conv kernel routing (im2col -> ggml_conv_2d_direct, thread-gated)

The shared fd::conv2d (SCRFD detector + ArcFace recognizer, both dominated by 3x3 stride-1 convs) always used ggml_conv_2d (im2col + tinyBLAS mul_mat). im2col materializes a 9x patch buffer per conv - a large memory stream. ggml_conv_2d_direct has no patch buffer (far less memory traffic) but a less throughput-dense kernel. The two trade off by memory-bandwidth pressure, which is set by the thread count.

Measured crossover (buffalo_l 640x640 detect, f32, min-of-5 isolated runs, this 16-core box; the box was under heavy concurrent load so absolute numbers are noisy and reported as the least-contended minimum):

threads	im2col ms	direct ms	winner
1	594.8	768.3	im2col +29% (spare bandwidth -> GEMM throughput wins)
2	372.0	435.0	im2col +17%
4	218.5	230.6	im2col +5%
8	176.6	135.0	direct +24% (9x stream saturates bandwidth)

So the routing is now: direct for K>1 convs on CPU when n_threads >= 8, else im2col; 1x1 convs always stay im2col (pure GEMM, no 9x expansion); GPU always im2col (cuBLAS-class mul_mat beats the basic direct CUDA conv). The process-global default is min(hw,8), so an 8+-logical-core box (the common deployment target) lands in direct's win region while smaller boxes default below the crossover and keep im2col - no regression at any measured thread count. A/B via FACEDETECT_CONV=direct|im2col|auto.

Before/after at the production default (8 threads, f32, min over 5 paired reps)

Pipeline	im2col (before)	direct (after)	speedup	ref onnxruntime 8t	new ratio
SCRFD detect (640x640)	176.6	135.0	1.31x	26.0	0.19x (was 0.17x)
ArcFace recognizer (112x112)	37.1	29.2	1.27x	13.2	0.45x (was 0.30x)
buffalo_l pipeline (detect+align+embed)	185.5	150.6	1.23x	101.2	0.67x (was 0.50x)

Parity preserved (re-gated at 8 threads): scrfd raw heads max|d| ~1e-6 vs ONNX (1e-3 gate); detect boxes/landmarks <= 0.73 px (1 px gate); ArcFace stem/block + golden-landmark + isolated embed cosine = 1.000000 (>= 0.9999 gate). The direct kernel changes float accumulation order only (diffs ~1e-6), not the result.

Dead-end recorded for pass 2

• Blanket switch to ggml_conv_2d_direct for all CPU thread counts (the naive depth-anything recipe): NET NEGATIVE at low thread counts. direct regresses single-thread detect by +29% (594.8 -> 768.3 ms) and 2-thread by +17%, because with spare memory bandwidth im2col's tinyBLAS GEMM out-throughputs the direct kernel. The win only appears once threads saturate bandwidth (>= 8 here). Hence the thread-count gate, not a blanket switch. (depth-anything's DPT head convs run at full image resolution -> bandwidth-bound even single-thread, so direct won there unconditionally; SCRFD/ArcFace feature maps are smaller, so the crossover sits at a higher thread count.)

Round 2 CPU optimization (stacked, vs prior baseline 523aee1)

A second optimization round stacks two parity-safe levers on top of the pass-2 baseline (523aee1, the conv_2d_direct thread-gated routing): a custom AVX2 Winograd F(2x2,3x3) for the large 3x3 stride-1 convs, and an ArcFace BatchNorm fold to host constants. Measured contention-robust: back-to-back A/B (baseline binary vs optimized binary in the same session), median-of-N, on a loaded 16-core box. The absolute ms below are load-specific - treat the speedups, not the raw numbers, as the signal. Parity held after each lever (SCRFD boxes/landmarks <= 1 px; embedding cosine = 1.0 on the isolated and golden-landmark paths).

Pipeline	1t before -> after	1t speedup	8t before -> after	8t speedup	lever (parity)
SCRFD detect (headline)	507 -> 371 ms	1.37x	147 -> 72-80 ms	~1.7-2.05x	custom AVX2 Winograd F(2x2,3x3) for 3x3 s1 convs; SCRFD <= 1 px
buffalo_l pipeline (detect+align+embed)	658 -> 516 ms	1.28x	189 -> 95-107 ms	1.58-1.77x	Winograd detector + ArcFace BN-fold; cosine = 1.0 isolated/golden
ArcFace recognizer (embed only)	-	-	(8t)	~1.06-1.13x	BN-fold to host constants + graph-overhead reduction; cosine = 1.0

The headline: the custom AVX2 Winograd F(2x2,3x3) BEAT tinyBLAS sgemm, which refutes the doubt recorded in the prior pass (pass 2 deferred Winograd on the grounds that it "must beat llamafile's already-tuned sgemm, which even plain direct conv loses to" - this round implemented it and it won). It re-targets the 3x3 stride-1 pad-1 convs that dominate the SCRFD backbone/neck/head stems (min feature-map dim >= 80) plus the ArcFace 112x112 stem, with the transformed filter cached once per shape and tiles split across ggml threads. Detect goes 1.37x at 1 thread and ~1.7-2.05x at 8 threads; the full buffalo_l pipeline inherits 1.28x / 1.58-1.77x.

The ArcFace recognizer lever folds the inference-mode BatchNorm affine (scale = gamma/sqrt(var+eps), shift = beta - mean*scale) to host constants once per loaded model, collapsing a ~5-op live-node chain repeated ~25x per embed to a single mul + add per BN (algebraic constant folding, parity preserved). PReLU could NOT be fused without breaking the GPU path - this ggml exposes neither a per-channel PReLU nor a binary max op, so a single fused PReLU op would require a CPU-only custom op; left numerically identical (documented).

Honest framing (Round 2)

These deltas are measured vs our OWN prior ggml baseline (523aee1), NOT a fresh head-to-head vs onnxruntime this round. The wins NARROW the gap to onnxruntime substantially (SCRFD detect ~1.7-2x at 8 threads; sibling voice-detect.cpp ERes2Net ~1.3-1.6x at 8t), but on the conv-bound face / encoder models a residual gap remains intrinsic: ggml's im2col / Winograd conv on small/mid feature maps vs onnxruntime's tuned conv kernels. GEMM-bound transformer heads (e.g. the voice-detect.cpp wav2vec2 analyze heads) are the most likely to reach near-parity, especially on a gated AVX512 build. No model regressed.

Honest read

• Absolute latency improved 3.2-3.9x by defaulting to 8 threads (e.g. SCRFD detect 515.0 -> 153.9 ms, full pipeline 667.9 -> 201.9 ms). For any consumer that took the library default this is a real, parity-safe, out-of-the-box win.
• The ratio vs the reference did NOT improve - onnxruntime/OpenCV multithread at least as well (the detector reference scales ~4.5x, ggml ~3.3x), so the gap is intrinsic, not a thread-count artifact. ggml's conv (im2col + mul_mat) stays ~2-6x behind onnxruntime's dedicated, cache-blocked conv kernels on CPU.
• Realistic CPU ceiling. At matched threads, ggml will not beat onnxruntime on these conv-heavy detectors/recognizers on CPU; the value of the port remains no-Python deployment, a small libfacedetect.so, GGUF quantization, and one code path across ggml GPU backends. The thread default removes the artificial 1-thread penalty; it does not change the per-core conv-kernel gap.

Profiling attribution (per pipeline)

Time was attributed by thread-scaling (a forward that scales ~linearly with threads is compute-bound; host work does not scale) and by mode isolation (recognizer mode times the embed graph alone on a pre-resized crop).

• SCRFD detect (640x640): dominated by the conv graph compute (ggml_backend_graph_compute). 503 ms @ 1t -> ~128 ms @ 8t, plateauing at ~6-8 threads (the large im2col is memory-bandwidth bound). Host-side letterbox + anchor decode + NMS is negligible (it would not scale 4x otherwise; the anchor grids are tiny scalar loops over a few thousand candidates).
• ArcFace / SFace recognizer (112x112): compute-bound matmul/conv, 156 ms @ 1t -> ~44 ms @ 8t, also plateauing ~8 threads.
• Full pipeline / analyze: detect dominates (detect ~154 ms vs embed ~45 ms at 8t), so the detector's conv kernel is the binding constraint.

Dead-ends (measured, reverted, recorded)

• -DGGML_NATIVE=ON (AVX512 on Zen5): NET NEGATIVE for this workload. It helps the small recognizer (embed 44 -> 33 ms @ 8t) but significantly regresses the dominant bandwidth-bound SCRFD detector (detect 128 -> 166 ms @ 8t; 503 -> 766 ms @ 1t) - classic AVX512 down-clocking hurting a memory-bound im2col conv while the detector dominates the pipeline. Kept the build at AVX2/FMA + tinyBLAS (GGML_LLAMAFILE), which is what the headline table above uses.

GPU rows

Appended by scripts/gpu_verify.sh on a CUDA host (not run here - no GPU).

GPU (CUDA) - NVIDIA GB10 (Grace Blackwell, sm_121a), CUDA 13.0

Built -DFACEDETECT_GGML_CUDA=ON -DGGML_CUDA_NO_VMM=ON -DGGML_NATIVE=OFF -DCMAKE_CUDA_ARCHITECTURES=121 (ggml auto-promotes 121 -> 121a for GB10). Device forced via FACEDETECT_DEVICE=CUDA0; the CLI confirms fd::Backend using device: CUDA0 (GB10, compute 12.1, 122 GB unified VRAM). ggml-GPU and ggml-CPU measured on the SAME GB10 box (N=30, warmup excluded, default min-of-hw-and-8 threads). The reference column is the CPU onnxruntime numbers from the 8-thread CPU table above, measured on an AMD Ryzen 9 9950X3D - a DIFFERENT machine, so GPU-vs-reference is cross-machine and indicative only.

Latency (ms/image, full detect+align+embed pipeline)

Pipeline	ggml-GPU (GB10)	ggml-CPU (GB10, 8t)	reference onnxruntime (Ryzen, 8t)	GPU vs ref
buffalo_l (SCRFD+ArcFace)	19.99	246.6	101.2	5.1x
buffalo_s (det_500m+MobileFaceNet)	11.58	47.3	n/a	-
YuNet+SFace	5.89	65.97	13.8	2.3x

GPU is 4-12x faster than the same-box CPU and beats the cross-machine CPU onnxruntime reference (2.3x-5.1x). The GPU port is competitive.

GPU numerical parity (FACEDETECT_DEVICE=CUDA0, vs the committed CPU/reference goldens)

NOTE: the committed test binaries hard-pin setenv(FACEDETECT_DEVICE,"cpu",overwrite=1), which would silently override the GPU device. For this GPU run the DGX test copies were patched to overwrite=0 (an explicit env wins, default stays cpu) so the gates actually exercise CUDA. This patch is NOT in the source tree and should be wired into the harness.

• buffalo_l ArcFace recognizer:
  • golden-landmark path (decode -> norm_crop -> embed): cosine 1.000000, max|d| <= 1.2e-4 on ALL THREE fixtures -> parity HOLDS (strict 1e-3 bound met).
  • golden-crop (isolated) path: cosine 1.000000, max|d| 1.0e-4 -> HOLDS.
  • full production path (detect -> align -> embed): cosine 0.999945 (gate >= 0.9999 PASS); max|d| 1.5e-3 is REPORTED-not-gated (sub-pixel landmark drift, same as on CPU).
• buffalo_l SCRFD detector: top score 0.90054 vs ref 0.90063; max box-corner err 0.0217 px, max landmark err 0.0193 px -> parity HOLDS (1 px gate).
• buffalo_m / buffalo_s / YuNet / SFace: no committed reference golden in-tree, so GPU was NOT parity-checked vs reference this run (benched only).
• Teardown SIGABRT (benign): test_embedding prints all parity comparisons as PASS, then aborts at process exit with CUDA error: driver shutting down in ~ggml_backend_cuda_buffer_context (ggml-cuda.cu:637). This is a static-destruction-ordering artifact (the CUDA driver tears down before the ggml buffer destructor runs); the voice tests avoid it by calling vd::shutdown_backend() explicitly. It does NOT affect the parity results, which are all emitted before the abort.

Fair GPU comparison (GB10): ggml-GPU vs reference-GPU (SAME GPU)

The GPU section above compares ggml-GPU against the reference on a DIFFERENT machine's CPU, which is not a fair test. This section measures the REFERENCE on the SAME NVIDIA GB10 (Grace-Blackwell, sm_121a, CUDA 13.0).

• Reference frameworks (all confirmed resident on the GB10 GPU):
  • insightface 1.0.1 FaceAnalysis(..., providers=["CUDAExecutionProvider"], ctx_id=0) for the buffalo_l / buffalo_s end-to-end pipelines. Per-model session.get_providers() printed ['CUDAExecutionProvider', ...] for detection / recognition / genderage / 2d106 (insightface keeps only the 3d68 landmark head on CPU).
  • onnxruntime-gpu 1.24.0 (CUDAExecutionProvider) for the strict per-model ONNX forwards (SCRFD det_10g, ArcFace w600k_r50, YuNet, SFace). Wheel: onnxruntime_gpu-1.24.0-cp312-linux_aarch64 from the NVIDIA jetson-ai-lab SBSA/cu130 index (PyPI onnxruntime-gpu is x86-only). CUDA EP asserted first provider per session.
  • opencv-python 4.13.0 cv2.FaceDetectorYN/FaceRecognizerSF (the YuNet+SFace cv2 reference) cannot run on the GPU: the pip opencv-python DNN module has no CUDA backend. So the YuNet+SFace GPU reference is the YuNet + SFace ONNX forwards run through ORT CUDA EP instead (align/NMS stay on host, as they do on both sides).
• Method: warmup 20, N=100 timed, per-call median (ms). Same image (face_a.jpg, 641x800). speedup = reference-GPU / ggml-GPU (>1 means ggml-GPU is faster).

Pipeline (detect+align+embed) - ggml row vs GPU reference

Pipeline	ggml-GPU ms	reference-GPU ms	framework + provider	speedup (ref/ggml)
buffalo_l (strict, SCRFD+ArcFace forwards)	19.99	9.39 (5.64+3.75)	onnxruntime-gpu CUDA EP	0.47x (ref 2.1x faster)
buffalo_l (full product pipeline)	19.99	34.45	insightface ctx_id=0 CUDA EP	1.72x (ggml "wins" - but ref ALSO runs genderage + 2d106 + 3d68 heads)
buffalo_s (full product pipeline)	11.58	10.08	insightface ctx_id=0 CUDA EP	0.87x (ref faster, and ref runs extra heads)
YuNet+SFace (NN forwards)	5.89	2.06 (1.37+0.69)	onnxruntime-gpu CUDA EP	0.35x (ref 2.9x faster)

Per-model reference-GPU forwards (onnxruntime-gpu CUDA EP / insightface)

Model	reference-GPU ms	framework + provider
SCRFD det_10g (640x640 forward)	5.64	onnxruntime-gpu CUDA EP
ArcFace w600k_r50 (112x112 embed)	3.75	onnxruntime-gpu CUDA EP
YuNet (640x640 detect forward)	1.37	onnxruntime-gpu CUDA EP
SFace (112x112 embed)	0.69	onnxruntime-gpu CUDA EP
buffalo_l SCRFD detect (full, incl post)	6.84	insightface ctx_id=0 CUDA EP
buffalo_l genderage (detect+analyze)	8.32	insightface ctx_id=0 CUDA EP

Honest GPU verdict (face)

On the SAME GB10 GPU and a like-for-like detect+align+embed workload, the reference is faster on every pipeline: ~2.1x on buffalo_l (SCRFD+ArcFace ONNX forwards 9.39 ms vs ggml 19.99 ms) and ~2.9x on YuNet+SFace (2.06 ms vs 5.89 ms). ggml-GPU only "beats" the insightface full pipeline (34.45 ms) because insightface additionally runs the genderage + 2d106 + 3d68 landmark heads (strictly more work); against a same-work GPU reference ggml-GPU does NOT win. cuDNN's conv kernels (ORT CUDA EP) still out-run the ggml CUDA conv graph on these conv-dominated detectors/recognizers. Note: the YuNet+SFace cv2 DNN reference could NOT be GPU-benched (opencv-python has no CUDA backend); the ORT-CUDA ONNX forwards are used as the GPU reference instead. All ONNX/insightface models DO run on Blackwell sm_121a (no model was blocked).