Motion Detection Performance
September 18, 2026 ยท View on GitHub
Last update: 2026-09-18
Source: data/dataset_info.json
Evaluation view: HT20/HT-LTF
Dataset revision: sha256:0c93c7c0f21ed7dab74209f8e7414069aa58c9472eadd93b74c33dc0a048127c
Input revision: sha256:f4d95ae22924d090e4d488dc917304c74e927ffc4685b2c1f6ee81acb809fb10
Generated by: tools/generate_performance_report.py
Run started: 2026-09-18T02:55:11+02:00
Run duration: 16.50s
Inputs: 13 real paired datasets, 0 synthetic paired datasets, 5 long quiet datasets
This report shows how ESPectre's motion detectors perform on recorded data.
tools/generate_performance_report.py also verifies that the host-side C++ and Python implementations produce matching results when they replay the same recordings.
- Lightweight Detection: Uses feature-fusion coefficients fitted offline on training recordings and calibrates its threshold for each recording without refitting those coefficients.
- High-Accuracy Detection: Uses the ML neural implementation trained on recordings marked as motion or no motion. Published replay results use only recordings outside its training corpus.
See ALGORITHMS.md for the full detector design.
Current Host Resource Benchmark
The generator compiles the current production C++ sources once per source revision and executes the benchmark on every report run. Timings and CPU estimates therefore describe the report host, not an ESP device. Persistent memory is the detector object's sizeof plus live heap allocations after construction; allocator metadata is not included.
Nominal load: 100 packets/s and 4 inferences/s; window 100 packets.
| Detection profile | Persistent memory | Packet median / p90 | Inference median / p90 | Modeled detector CPU | Transient heap |
|---|---|---|---|---|---|
| Lightweight | 1,792 B | 0.16 / 0.22 us | 1.37 / 1.48 us | 21.02 us/s | 0 B |
| High Accuracy | 5,068 B | 0.62 / 0.73 us | 3.55 / 3.65 us | 76.05 us/s | 0 B |
Required Performance
| Metric | Target |
|---|---|
| Recall | >95% |
| FP Rate | <5% |
Recall is the percentage of motion correctly detected. FP Rate is the percentage of no-motion readings incorrectly reported as motion.
Tests Included
- C++
test_motion_detection - C++
test_long_recordings - Python
TestPerformanceMetrics - Python
test_validation_long_recordings.py
Normal Wi-Fi Signal
Effective Alarms counts how many times the detector incorrectly switches to MOTION when no motion is present. It uses the same filtering as the firmware.
These recordings contain periods without motion followed by periods with motion, captured with a normal Wi-Fi signal. Both detectors are reported only on the combined selection + holdout corpus.
Lightweight Detection
| Metric | ESP32-C3 | ESP32-C5 | ESP32-C6 | ESP32 | ESP32-S3 |
|---|---|---|---|---|---|
| Recall | 99.7% | 100.0% | 100.0% | N/A | 100.0% |
| Min Recall | 99.4% | 100.0% | 100.0% | N/A | 100.0% |
| Precision | 99.4% | 94.6% | 98.9% | N/A | 98.1% |
| FP Rate | 0.3% | 2.9% | 0.6% | N/A | 1.0% |
| Max FP Rate | 0.4% | 2.9% | 0.6% | N/A | 1.0% |
| F1-Score | 99.6% | 97.2% | 99.4% | N/A | 99.0% |
| Effective Alarms | 0 | 4 | 1 | N/A | 1 |
High-Accuracy Detection
| Metric | ESP32-C3 | ESP32-C5 | ESP32-C6 | ESP32 | ESP32-S3 |
|---|---|---|---|---|---|
| Recall | 100.0% | 100.0% | 100.0% | N/A | 100.0% |
| Min Recall | 100.0% | 100.0% | 100.0% | N/A | 100.0% |
| Precision | 100.0% | 99.7% | 100.0% | N/A | 100.0% |
| FP Rate | 0.0% | 0.1% | 0.0% | N/A | 0.0% |
| Max FP Rate | 0.0% | 0.1% | 0.0% | N/A | 0.0% |
| F1-Score | 100.0% | 99.9% | 100.0% | N/A | 100.0% |
| Effective Alarms | 0 | 0 | 0 | N/A | 0 |
Reserved Augmentation Diagnostic
Lightweight and High Accuracy are evaluated on the same reserved selection + holdout pairs after deterministic base+drift+burst-loss packet augmentation. As in production ML training, the fixed views 20260807, 20260808 contribute alternating row positions. These combined reserved results are diagnostic, not an uncontaminated promotion gate.
| Detection profile | Corpus | Recall | FP Rate | F1 |
|---|---|---|---|---|
| Lightweight | Reserved, mixed two-seed augmentation (13 pairs) | 98.8% | 0.8% | 98.6% |
| High Accuracy | Reserved, mixed two-seed augmentation (13 pairs) | 99.4% | 0.1% | 99.5% |
Weak Wi-Fi Signal
These reserved selection + holdout recordings show detector behavior when the Wi-Fi signal is weak. The High Accuracy requirements for this test are recall >90% and FP <10%; Lightweight results are included for information only.
Lightweight Detection
| Metric | ESP32-C3 | ESP32-C5 | ESP32-C6 | ESP32 | ESP32-S3 |
|---|---|---|---|---|---|
| Recall | 99.3% | 97.2% | 99.7% | N/A | 99.3% |
| Min Recall | 98.5% | 94.4% | 99.7% | N/A | 98.9% |
| Precision | 99.4% | 95.1% | 98.9% | N/A | 97.4% |
| FP Rate | 0.3% | 2.5% | 0.6% | N/A | 1.3% |
| Max FP Rate | 0.3% | 4.9% | 0.6% | N/A | 2.1% |
| F1-Score | 99.3% | 96.1% | 99.3% | N/A | 98.4% |
| Effective Alarms | 0 | 5 | 1 | N/A | 3 |
High-Accuracy Detection
| Metric | ESP32-C3 | ESP32-C5 | ESP32-C6 | ESP32 | ESP32-S3 |
|---|---|---|---|---|---|
| Recall | 99.0% | 98.6% | 99.7% | N/A | 98.7% |
| Min Recall | 98.3% | 98.0% | 99.7% | N/A | 96.3% |
| Precision | 100.0% | 100.0% | 100.0% | N/A | 99.6% |
| FP Rate | 0.0% | 0.0% | 0.0% | N/A | 0.2% |
| Max FP Rate | 0.0% | 0.0% | 0.0% | N/A | 0.6% |
| F1-Score | 99.5% | 99.3% | 99.9% | N/A | 99.1% |
| Effective Alarms | 0 | 0 | 0 | N/A | 0 |
Long Quiet Reserved Recordings
These selection + holdout recordings contain no motion. They show how often each detector reports motion during quiet periods.
Lightweight Detection
| Metric | C3 | C5 | C6 | ESP32 | S3 |
|---|---|---|---|---|---|
| Avg FP Rate | 0.13% | N/A | 3.65% | N/A | 1.05% |
| Max FP Rate | 0.13% | N/A | 4.93% | N/A | 1.97% |
| Effective Alarms | 0 | N/A | 11 | N/A | 4 |
High-Accuracy Detection
| Metric | C3 | C5 | C6 | ESP32 | S3 |
|---|---|---|---|---|---|
| Avg FP Rate | 0.00% | N/A | 0.13% | N/A | 0.02% |
| Max FP Rate | 0.00% | N/A | 0.13% | N/A | 0.04% |
| Effective Alarms | 0 | N/A | 0 | N/A | 0 |