Benchmark - Garage Door Classifier
August 6, 2026 ยท View on GitHub
This benchmark compares the frozen DINOv3 ViT-S/16 teacher against distilled MobileNetV4 students across three targets: Mac (Apple Silicon), ESP32-S3, and Orange Pi 5 Ultra (RK3588). All latency numbers are isolated ONNX inference only; they exclude preprocessing, I/O, and camera capture. AP is measured on the same 280-image date-isolated validation set.
Results
| Model | Target | Backend | AP | FPS (p50) | Params |
|---|---|---|---|---|---|
| DINOv3 ViT-S/16 + linear probe | Mac (Apple Silicon) | ONNX Runtime CPU | 0.9765 | 102 | 21.60 M |
| DINOv3 ViT-S/16 + linear probe | Mac (Apple Silicon) | ONNX Runtime CoreML / ANE | 0.9765 | 24 | 21.60 M |
| MobileNetV4 Small 0.5x (distilled) | Mac (Apple Silicon) | ONNX Runtime CPU | 0.99278 | 1639 | 0.95 M |
| MobileNetV4 Small 0.5x (distilled) | Mac (Apple Silicon) | ONNX Runtime CoreML / ANE | 0.99278 | 2564 | 0.95 M |
| MobileNetV4 Small 0.5x ESP-DL INT16 | ESP32-S3 (8 MiB PSRAM) | ESP-DL INT16 | 0.99223 | 4.2 | 0.95 M |
| MobileNetV4 Medium (distilled) | Mac (Apple Silicon) | ONNX Runtime CPU | 0.98299 | 356 | 8.40 M |
| MobileNetV4 Medium (distilled) | Mac (Apple Silicon) | ONNX Runtime CoreML / ANE | 0.98299 | 2174 | 8.40 M |
| MobileNetV4 Medium RKNN INT8 | Orange Pi 5 Ultra (RK3588) | RKNN INT8 NPU | 0.98565 | 369 | 8.40 M |
Notes
- FPS is computed from p50 latency as
1000 / p50_ms. ESP32-S3 FPS is the sustained-run throughput reported by the instrumented firmware (560 inferences / 133.13 s). - AP for the teacher is the validation-set AP of the unified production candidate (1241-image dataset, 961 train / 280 validation, 224x126 input). AP for students is the validation-set AP of the frozen H/seed42 candidate (224x128 letterbox input). Both use the same 280-image date-isolated validation split; the input resolution differs by two pixels in height due to letterbox rounding.
- The Orange Pi RKNN INT8 AP (0.98565) is marginally higher than the float Medium AP (0.98299). AP measures ranking consistency, and the quantized threshold is re-derived on hardware logits rather than reused from the float model.
- Mac CoreML is slower than CPU for the DINOv3 ViT-S/16 teacher because only 387 of 760 graph nodes are supported by CoreML; the remaining nodes fall back to CPU and incur cross-EP data copies. CNN students benefit from CoreML: Medium is 6x faster on ANE than on CPU.
- The Mac Medium student on CoreML/ANE (2174 FPS) outperforms the Orange Pi RK3588 INT8 (369 FPS) by 5.9x, reflecting the higher compute density of the Apple Neural Engine and the absence of INT8 quantization overhead on the Mac path.
- Parameter counts are total ONNX graph initializer elements. INT16 and INT8 quantization do not change the parameter count; they only reduce storage bytes.
- These are isolated inference benchmarks. Camera capture, image conversion, application scheduling, and end-to-end command latency are outside these measurements.
Method
- Mac latency: 200 runs with 10 warmup iterations, dummy input, pure
session.run. CPUExecutionProvider and CoreMLExecutionProvider tested separately. - ESP32-S3 latency: instrumented model-evaluation firmware, 280 inferences plus a 560-inference sustained run, FNV-1a hash-verified serial payloads.
- Orange Pi 5 Ultra latency: official RKNN C API runtime, 280 inferences plus a 39130-inference sustained run over 120 seconds.
- Full hardware validation methodology, observation contracts, and bootstrap harnesses are documented in
skills/skill.mdandexamples/hardware/.