Working Log - DINOv3 Classifier Skill

Changelog

2026-08-05 - Target-neutral hardware validation contract

• Added create-hardware-plan, which validates current ONNX candidate scores, all candidate/source/manifest provenance, exact selected path order, current image hashes, sigmoid alignment, binary coverage, safe runtime identities, and finite gates before atomically writing a deterministic path-free plan.
• Added evaluate-hardware, which accepts only closed-schema hash-bound observation JSONL, re-hashes candidate/runtime artifacts, selects a fresh hardware maximum-F1 threshold, reports source-threshold parity and linear latency percentiles, and evaluates AP, mean probability error, and p95 latency gates inclusively.
• Valid gate failures now write status: fail and return exit 1; integrity failures write no result. Plans and results remain unapproved and never include paths, labels, or per-row result outputs.
• Added dependency-light synthetic coverage without ONNX Runtime, vendor SDKs, hardware, or private fixtures.
• Added public bootstrap harnesses for ESP32-S3 / ESP-DL and RK3588 / RKNN. Both consume the path-free plan contract, emit validator-only JSONL separately from diagnostics, and keep generated models, SDKs, private paths, and raw hardware output outside git.
• The ESP harness validates canonical plan identity, manifest ordering/content, flashed model SHA-256, RGB letterbox preprocessing, serial checksums, and atomic output. The RKNN harness validates candidate/manifest provenance, emits a normalized NHWC pack plus direct TSV row metadata, passes the static candidate shape to Toolkit2, and separates measured observations from optional sustained-run telemetry.

2026-08-04 - Audited mining loop reproduced from unlabeled data

• Sealed the legacy 444-image ground-truth manifest until final external evaluation.
• Re-audited all 295 seed labels using a strong frontier vision auditor: 6 positive, 289 negative.
• Implemented class-balanced probe training and scored all 121,467 cached embeddings.
• Round 1 strong visual audit: 180 positive, 12 negative from 192 active candidates.
• Round 2 strong visual audit: 154 positive, 38 negative from 192 candidates; one uncertain image was resolved via an independent second review.
• Produced an independent pre-confirmation dataset of 679 images: 340 positive, 339 negative.
• Pre-confirmation date-isolated validation: AP 0.9967. Validation-only maximum-F1 threshold: 0.888214.
• Pre-confirmation frozen external 444 evaluation: AP 0.9920, precision 1.0000, recall 0.9500, F1 0.9744 at the untouched validation threshold.
• Round 3 convergence audit: top-positive 40/64 positive, boundary 2/64 positive, low-negative 0/32 positive.
• Merged the confirmation round through the new coverage-checked CLI: 839 images, 382 positive and 457 negative.
• Final date split: 655 train and 184 validation. Validation AP 0.9767; validation-only threshold 0.4466304779; precision 0.9143; recall 0.9697.
• Frozen external 444 evaluation after Round 3: AP 0.9970; precision 1.0000; recall 0.9700; F1 0.9848 at the untouched validation threshold.
• Exported the then-final Round 3 backbone/probe ONNX and verified ONNX Runtime probabilities on audited validation images: open 0.9870, closed 0.0133. Later owner correction and retraining superseded this artifact.
• Added python -m dinov3_classifier CLI support.
• Added blind contact sheets, stable audit IDs and sidecars, audit coverage validation, audit merging, cached-embedding training, balanced loss, full-universe scoring, acquisition-stratified active sampling, split filtering, and validation-only threshold derivation.
• Expanded test coverage from 10 to 48 unit and workflow tests, covering idempotent owner overrides, explicit owner verdicts, artifact-bound sign-off gates, exact cohort validation, successful deployment-gate traversal, deployment-prompt semantics, static workbench generation, and HTTP path allowlisting.
• Hardened data contracts to fail closed: cache paths must match exactly, manifests cannot silently drop images, active score paths must align, audit coverage is mandatory, and date splits/threshold derivation reject empty or single-class validation sets.
• Added image-size provenance metadata to embedding caches and probe checkpoints; ONNX export now rejects resolution mismatches, and sweep writes a machine-readable winner record.
• Corrected provenance documentation: the sealed 444 reference was also model-mined and visually audited, rather than human-curated. Its role was external comparison, not infallible ground truth.
• Created a mini computer vision workbench: responsive dataset-boundary and model-disagreement HTML, optimized image assets, repeatable audit overlays, machine-readable case queues, and an HTTP allowlist server.
• Full-universe artifact comparison identified 33 disagreeing predicted labels at deployment thresholds; all 33 received independent visual review.
• In that disagreement slice, the reference-444 model matched the independent visual pass on 18 cases and the mined-839 model on 15. At each model's deployed threshold, the dataset boundary page found 20/20 agreement for sealed 444 labels and 15/20 for the independent 839 boundary review; these represent visual audit outcomes, not human ground truth.
• Task-owner calibration established the business rule that any visible opening (including slightly open, half open, opening, or nearly closed) is positive (yes) because it represents a household security risk.
• The task owner reviewed 20 mined boundary cases and supplied 12 label changes: 11 no -> yes and 1 yes -> no. The apply-overrides command preserves append-only history, rejects source/output file aliasing, and binds its output manifest to an explicit owner sign-off ID.
• Deterministic seed-42, calibration-sign-off-bound retraining yielded: validation AP 0.9731, operating threshold 0.3895591199, precision 0.9236, recall 0.9779, F1 0.9500. This supersedes earlier unbound corrected checkpoints.
• Evaluated at the untouched validation threshold, frozen 444 external evaluation reached AP 0.9984, precision 1.0000, recall 0.9700, and F1 0.9848.
• Full-universe comparison following deterministic retraining surfaced 35 prediction disagreements. All 35 form a fresh pending review cohort; prior audit verdicts were deliberately not transferred by score proximity or rank.
• The preview-overrides command eliminates calibration-signing circularity by computing the corrected semantic digest before sign-off generation. Training rejects partially signed manifests. Score artifacts carry classifier and universe hashes; complete workbench queues carry comparison IDs and total disagreement counts. Deployment requires a second sign-off bound to the exact classifier checkpoint and untruncated owner-review cohort.
• A provisional ONNX model was smoke-tested prior to adding the stricter final-cohort gate, but it is not the current deterministic classifier and must not be treated as final. Current export is intentionally blocked until all 35 new disagreements receive explicit owner review.
• Added MIT licensing and GitHub Actions configuration. Export requires the operating threshold to match both the reviewed queue and deployment sign-off; single-head deployment prompts document sigmoid semantics alongside explicit positive and negative label names.
• Reframed README.md for human users, moved reusable execution steps into skills/skill.md, and isolated garage-specific evidence in docs/case-study-garage.md.
• Completed a pre-publication privacy review with Ollama GLM. It found no blocking disclosure: private artifacts remain gitignored, aggregate case-study metrics contain no identifying paths or media, and workbench queues remain outside the allowlisted static root.

2026-08-04 - Initial implementation

• Scaffolded the public skill, CLI, Python package, test suite, and documentation.
• Embedded 121,467 garage frames at 224x126 resolution in approximately four minutes using MPS acceleration.
• Executed local LM Studio labeling, discovering 23 false positives among 29 proposed positive candidates.
• Exported a combined DINOv3 backbone and probe to ONNX, verifying predictions in ONNX Runtime.
• The initial validation run incorrectly fell back to pre-existing 444 labels during training. That result did not validate the mining loop and was superseded by the independent run documented above.

Validated Best Practices

Visual audit is a capability boundary

• Local vision-model output is a proposal, not a label. Repeatedly prompting a model with systematic visual errors does not produce convergence.
• Final visual auditing requires a reliably capable vision model or human reviewer. Weaker or non-visual models may orchestrate the workflow, but must delegate visual label judgment.
• Auditing must remain blind. Displaying proposed labels or score colors anchors the reviewer to the proposal model's mistakes.
• Every contact-sheet cell requires a stable audit ID and sidecar file path mapping. Ambiguous thumbnails must be inspected in the original image; unresolved items require an independent second review.
• Strong visual capability still cannot establish business policy. The workflow must prompt the owner to calibrate edge-case semantics and inspect failure cases before declaring completion.

Mining needs three acquisition strata

• top_positive efficiently harvests rare positive examples once a seed probe exists.
• boundary isolates errors that refine the classifier decision surface.
• low_negative is necessary to estimate hidden false negatives and justify stopping. Auditing only high scores evaluates precision while obscuring recall performance.
• Applying diversity selection within each stratum prevents spending audit calls on adjacent video frames.
• Seed probes trained on rare positives require class-balanced loss. With only 6 positive and 289 negative examples, unweighted binary cross-entropy loss collapses toward predicting all negatives.

Evaluation needs a reference firewall

• A legacy labeled dataset is not a source of training seeds. It must remain unread until model weights and validation thresholds are frozen.
• Train/validation splits must be partitioned by date, session, location, or another leakage-resistant group. Random frame splits are invalid for video data.
• Operating thresholds are derived exclusively from validation predictions, never from an external reference set and never from a hardcoded 0.5 default.
• Retraining on additional labels alters score calibration. Never reuse a threshold derived from a different fitted probe without a reserved calibration split.

DINOv3 and deployment

• Preserve source aspect ratios. DINOv3 Rotary Position Embeddings (RoPE) handle arbitrary input shapes; 224x126 significantly outperformed forced-square inputs for the 16:9 garage camera view.
• Higher resolution does not automatically improve linear probe performance on small datasets. Use a validation resolution sweep.
• Cache the full embedding universe once. Probe training, scoring, and active sampling passes should not rerun the backbone.
• The combined ONNX backbone and probe loaded successfully in ONNX Runtime with input shape (batch, 3, 126, 224).

2026-08-05 - Unified garage dataset handoff

• Merged former reference_444 and owner_corrected_839 into data/unified/ (1241 images; train 961 / validation 280).
• Retrained production candidate at 224x126: validation AP 0.9765, threshold 0.2917941213, F1 0.9521.
• Exported the unified private ONNX candidate and handed the production artifacts to the private deployment workspace.
• Removed temporary mining intermediates and old score/label scratch files while retaining the unified dataset, embedding cache, and source video privately.
• Future garage accumulation continues through the private incoming queue.

2026-08-05 - CNN distillation infrastructure

• Added split-safe teacher artifact export with aligned DINO features, raw single-head logits, leakage strata, image-size provenance, and source hashes. Export requires current probe metadata proving the exact training split, paths, and labels.
• Added timm CNN student training with hard-label BCE, temperature-scaled Bernoulli logit distillation, and centered-cosine embedding distillation through a training-only projector.
• Added deterministic teacher-universe capping. Every owner-labeled row is retained; unlabeled rows are allocated by leakage-resistant stratum and teacher-logit decile; selection seed, ordered path digest, and selected cell counts are persisted separately from the student seed.
• Added per-image SHA-256 binding from embedding caches through teacher artifacts; selected images are revalidated before training. Unlabeled validation-universe rows participate in group isolation but not metrics, and final singleton batches are merged rather than dropped.
• Candidate checkpoints bind architecture, exact timm version, preprocessing, labels, validation operating point, teacher/manifest hashes, training configuration, and teacher selection. The projector is excluded from student_candidate.pt and ONNX graphs.
• Removed direct ONNX export from distill. Distilled students remain candidates until student-specific scoring, complete disagreement review, deployment sign-off, quantization validation, and a gated export command exist.
• Verified tagged MobileNetV4 Small and Medium output 1,280-dimensional pre-logit features despite stale model metadata. Dummy-forward feature probing and singleton-batch handling prevent shape and pooled-BatchNorm failures.
• Private labeled-only one-epoch smokes reached AP 0.9820 for MobileNetV4 Small 0.5x and AP 0.9703 for MobileNetV4 Medium. PyTorch/ONNX maximum absolute logit differences were below 2e-6.
• ESP-PPQ INT16 conversion produced a 2.0 MB Small artifact with 98.4% float/quant prediction agreement on a 64-row simulator subset. The Medium ORT QDQ INT8 proxy reduced 33.7 MB to 8.7 MB and preserved AP (0.9703 to 0.9699) while shifting score calibration materially. ORT is not RKNN evidence, and quantized thresholds must be derived again.
• A provenance-complete replacement teacher probe reproduced the existing unified probe exactly with zero logit difference and validation AP 0.9765. The full train-universe artifact contains 115,548 rows over four isolated train dates.
• Three-seed labeled-only ablation results were statistically close: H AP 0.9905 +/- 0.0013; L-S AP 0.9912 +/- 0.0010. A one-epoch full-universe L-U benchmark took 511.7 seconds on MPS, motivating a fixed 12,000-row capped experiment rather than silently spending 6-8 hours on the full matrix.
• The singleton-safe final matrix superseded those preliminary labeled baselines. Mean AP over seeds 17/42/73 was H 0.99109, L-S 0.99063, L-U 0.98782, E-U 0.98953, and LE-U 0.99106. All universe runs used one 12,000-row selection digest. Because H and LE-U differed by only 0.00003 AP, the simpler H recipe was frozen; canonical Small seed 42 reached AP 0.99278 and max-F1 0.96951.
• Moving H/seed42 to MobileNetV4 Medium did not improve validation quality: AP 0.98299 and max-F1 0.95152. Medium remains an Orange Pi conversion candidate for hardware comparison, not the accuracy winner.
• Physical hardware validation is complete for both frozen candidates. MobileNetV4 Small ESP-DL INT16 retained AP 0.99223 on ESP32-S3 at 4.21 inference FPS, with stable memory and die temperature over 133 seconds of measured inference. MobileNetV4 Medium RKNN INT8 reached AP 0.98565 on RK3588 at 326.08 inference FPS, with stable NPU temperature and frequency over 120 seconds. Both targets require their newly derived quantized thresholds; these are isolated inference benchmarks rather than camera-to-action latency measurements.

2026-08-05 - Student candidate lifecycle

• Renamed the distilled inference checkpoint to student_candidate.pt; no legacy filename fallback exists.
• Added explicit artifact_role: candidate and deployment_approved: false flags to candidate checkpoints and ONNX sidecars while preserving teacher artifact, manifest, selection, preprocessing, and training provenance.
• Added inspect-student for machine-readable schema, metadata, state summary, and exact timm-version validation without model construction or image access.
• Added export-student-candidate for unapproved ONNX candidates and hash-bound sidecars. The command is an evaluation artifact export, not a deployment approval path.
• Added score-student for ordered manifest scoring through either PyTorch checkpoints or ONNX Runtime. Score NPZ files include raw logits, probabilities, image hashes, ordered-path universe identity, candidate and manifest hashes, source provenance, preprocessing, split, and runtime.
• Added fail-closed tests for CLI dispatch, candidate flags, inspection, candidate export, checkpoint/ONNX scoring parity, duplicate and missing manifest paths, and sidecar/checkpoint/graph hash mismatches.
• Kept the existing reviewed DINO deployment gate unchanged. Student candidates still require complete teacher-versus-student disagreement review and explicit owner approval; no approved student deployment export exists.

Remaining Gaps

• No data augmentation or multi-crop feature averaging support.
• No single iterate command. Individual CLI subcommands cover the active loop reproducibly, but the agent orchestrates visual audit iterations.
• No end-to-end integration test running a miniature embed-to-export pipeline with a live backbone; automated tests verify data contracts without downloading weights.
• Resolution sweeping and frame extraction commands exist, but have not been executed end to end in this public repository.
• ONNX export emits a PyTorch dynamo dynamic_axes deprecation warning.
• Garage production now uses a private unified ONNX candidate; public skill export still keeps the strict owner-signoff gate for generic deployments.
• Distilled students can now be inspected, exported as unapproved ONNX candidates, and scored for review. Complete disagreement review, explicit owner approval, and an approval-bound student deployment export remain intentionally absent.

Pitfalls

Pitfall	Consequence	Prevention
Weak or non-visual final auditor	Mining silently fails and falls back to legacy labels	Delegate visual judgment to a capable vision model or human reviewer
Visible proposed labels during audit	Reviewer repeats proposal model bias	Use contact-sheet --blind
Missing audit IDs or sidecar files	Corrections cannot be mapped to source paths	Verify exact coverage using merge-audits --expected
Unweighted seed probe training	Rare positive examples are ignored	Use train --balanced
Omitting low-negative auditing	Recall remains unknown despite high AP	Reserve 20% of active batches for low_negative
Random video frame splitting	Temporal duplicates artificially inflate metrics	Split by session, date, or location
Reference set used during mining	Reproduction claims become invalid	Keep reference set sealed until model and threshold are frozen
Hardcoded 0.5 operating threshold	Deployment operating point is arbitrary	Derive operating threshold strictly on validation data
Forced square image resizing	Aspect distortion leading to lower AP	Preserve natural aspect ratio (e.g. 224x126)
LM Studio reasoning consuming output tokens	Empty label responses returned	Allocate at least 512 output tokens and use resumable writes