Test Strategy

Goals

Automated and manual tests prioritize workflow contracts that can silently compromise classifier reliability: file path alignment, audit completeness, leakage-resistant splits, provenance propagation, workbench privacy boundaries, and deployment artifact identity.

Automated Tests

tests/test_core.py covers:

• Image-size parsing and embedding/probe provenance tracking
• Deterministic and class-balanced linear probe training
• Validation-only threshold selection
• Date-safe dataset splits and single-class validation rejection
• Active sampling alignment and acquisition strata selection
• Blind contact-sheet coverage and audit merging
• Append-only task-owner overrides and sign-off validation
• Source/output file alias rejection, including hardlinks
• Calibration digest propagation across training steps
• Deployment gate rejection paths for mismatched artifacts
• Successful complete-cohort export execution and explicit owner-verdict enforcement with heavyweight backbone calls stubbed
• Single-head sigmoid semantics in generated deployment prompts

tests/test_workbench.py covers:

• Named input parsing for multi-dataset workbenches
• Unique dataset-boundary sample selection
• Complete two-model disagreement filtering
• Classifier and embedding-universe identity checks
• Exact cohort digests and non-truncated queue metadata
• HTML escaping and optimized image asset generation
• Static HTTP server path allowlisting, path traversal blocking, query string handling, HEAD requests, and symlink escape prevention

tests/test_distillation.py covers:

• Teacher feature/logit/path alignment and provenance-complete probe requirements
• Leakage-stratum selection, non-date universe manifests, cross-split rejection, and missing-label rejection
• Hard, temperature-scaled Bernoulli logit, and centered-cosine embedding losses
• Unbiased sparse hard-loss weighting and invalid hyperparameter rejection
• Deterministic capped teacher selection, complete owner-label retention, stratum/logit-cell coverage, and ordered path identity
• Unlabeled validation-universe rows, singleton-batch redistribution without row loss, and modified-image hash rejection
• CLI forwarding of independent teacher-selection and student-training seeds
• student_candidate.pt naming, explicit candidate/unapproved flags, teacher-center retention, and projector exclusion from candidate state
• CLI parser and dispatch coverage for student inspection, candidate export, and checkpoint/ONNX scoring
• Inspection rejection for invalid schema, metadata role, and exact timm-version mismatch without image access
• ONNX graph validation plus source-provenance, preprocessing, threshold, checkpoint, and graph hash sidecars
• Manifest duplicate/missing-path rejection and versioned score provenance
• Direct-image checkpoint/ONNX Runtime scoring parity, plus real tagged MobileNetV4 raw-logit parity when distillation and inference extras are installed

tests/test_hardware_validation.py uses only synthetic files and a monkeypatched candidate sidecar loader. It covers CLI dispatch, deterministic path-free plans, passing and gate-failing results, exit 1 for valid gate failure, independent hardware threshold selection, artifact hash drift, exact observation coverage/order, unknown fields, bool/non-finite numbers, unsafe identifiers, stale image hashes, and malformed score/plan provenance. No ONNX Runtime, vendor SDK, target hardware, or private fixture is required.

The ESP32-S3 example's Python tests cover deterministic letterboxing, strict plan and manifest identity, model-hash binding, fragmented serial framing, protocol rejection, and atomic transcript output. The RK3588 example's Python tests cover candidate/plan provenance, static Toolkit2 input shape, conversion cleanup, normalized NHWC packs, and direct path-free TSV metadata; its dependency-free CMake test covers latency summaries. None requires a vendor SDK or physical device.

GitHub Actions executes the dependency-light synthetic suite on Python 3.11 plus the RK3588 dependency-free CMake/CTest target for pushes to master and pull requests. Optional timm/ONNX Runtime integration tests skip when those extras are absent.

Private Integration Validation

The garage case study validates the workflow through deterministic retraining and final-cohort generation on 121,467 private video frames, cached embeddings, independently mined manifests, full-universe score artifacts, and static workbenches. An earlier checkpoint passed ONNX Runtime smoke tests, but the current signed final export remains blocked on owner review. Private image files, labels, review queues, model checkpoints, and generated pages remain gitignored and are excluded from public CI fixtures.

This integration pass confirms that all components operate together as intended. However, it is not reproducible in public CI and does not establish universal accuracy or latency benchmarks.

The private distillation integration additionally verifies labeled-only and full-train-universe teacher artifacts, a five-recipe three-seed Small ablation with one fixed 12,000-row universe selection, MobileNetV4 Medium recipe transfer, single-file ONNX graphs, PyTorch/ONNX Runtime parity, ESP-PPQ INT16 conversion, and an ORT QDQ INT8 sensitivity proxy. ORT quantization is not RKNN simulator evidence, and local macOS runs cannot establish physical-device latency.

Manual Release Checks

• Execute pytest -q inside the project virtual environment.
• Run git diff --check to detect whitespace or formatting issues.
• Inspect git status and staged diffs for private paths, images, labels, API credentials, model files, and generated queues.
• Verify that the example prompt in README.md references AGENTS.md and skills/skill.md correctly.
• Inspect workbench pages on desktop and mobile viewport dimensions.
• Verify loopback serving and explicit LAN serving modes separately.
• Confirm that private JSONL, NPZ, and model checkpoint paths return 404 from the static server.
• Load a signed ONNX export in ONNX Runtime and compare predicted probabilities against the source PyTorch probe on reviewed frames.
• Confirm that generated deployment prompts specify the reviewed image resolution, label names, sigmoid semantics, and threshold.
• Confirm that every row in the final disagreement cohort has explicit task-owner review entries before creating deployment sign-offs.
• For capped distillation, verify that every recipe and student seed uses the same ordered selection digest and selection seed.
• Inspect every frozen student with inspect-student, verify checkpoint/ONNX score parity, and generate the complete teacher-versus-student disagreement cohort before requesting owner approval.
• Confirm student checkpoints and sidecars remain artifact_role: candidate and deployment_approved: false; no candidate export is an approved deployment.
• Confirm hardware plan/result files contain no paths, labels, or per-row result data; both passing and failing results remain deployment_approved: false.
• Treat evaluate-hardware exit 1 as a completed validation with failed gates, distinct from schema/integrity exceptions that write no result.
• Compare quantized AP, score calibration, and the quantized validation threshold; do not reuse the float threshold automatically.
• Run both example Python suites and the RK3588 dependency-free CMake/CTest target.

Known Test Gaps

• No public integration test executes a real DINOv3 weight download, embedding pass, and ONNX export in CI.
• Frame extraction and resolution sweeping are not exercised through external binaries in CI.
• Mobile layout responsiveness is verified via manual inspection rather than browser automation.
• Real MobileNetV4 ONNX Runtime parity is optional rather than required in the dependency-light CI job.
• RKNN and physical ESP32-S3/Orange Pi checks are not automated in public CI. Private physical validation has covered full-split AP and threshold selection plus inference latency, memory, temperature, and sustained throttling behavior on both targets.