Product Requirements - DINOv3 Classifier Skill

Product Definition

The product is an AI-agent workflow backed by a Python CLI. It converts an unlabeled image collection and a binary visual question into a reviewed ONNX classifier built from a frozen DINOv3 backbone and a linear probe. When the reviewed teacher is too large for a target device, the workflow can also train a provenance-bound CNN student candidate for later hardware-specific validation and gated deployment.

Target User

The target user is a builder who repeatedly applies a vision-language model to a narrow visual question and wants reduced inference latency, lower recurring costs, local execution, or edge deployment. The user can supply raw images and business domain rules, but is not expected to manage DINOv3 architectures, active learning algorithms, or operating threshold calibration directly.

Problem

Training a linear head on embeddings is technically simple. The true challenges lie in discovering rare positive examples, preventing video-frame data leakage, filtering noisy candidate labels, defining ambiguous business rules, and verifying that the final deployed model matches the exact artifacts reviewed by human stakeholders.

A workflow that optimizes solely for average precision can still fail in real applications. It may learn an incorrect positive class definition, conceal false negatives behind biased sampling, or export an unverified model after retraining alters boundary decisions.

Product Principles

• Embed the full image universe once and reuse feature caches across training iterations.
• Direct visual auditing effort toward diverse, high-information samples.
• Keep proposed labels hidden during independent visual audits.
• Treat external reference datasets as sealed evaluation tests, never as data-mining sources.
• Select operating thresholds strictly on validation data, never on external reference sets.
• Give the human task owner ultimate authority over business semantics and edge cases.
• Bind corrected labels, model checkpoints, thresholds, and reviewed cohorts through cryptographic digests.
• Fail closed whenever provenance, threshold, or review evidence is incomplete or mismatched.
• Preserve every owner label when computationally capping an unlabeled distillation universe, and freeze one deterministic teacher selection across comparative runs.
• Keep physical-runtime execution outside the CLI while binding target-neutral observations to exact candidate, runtime, image, and plan identities.

Workflow

1. Setup: Install the CLI, obtain gated DINOv3 weights, and select a visual audit path.
2. Mining: Embed the universe, sample diverse seeds, audit candidate labels, train a seed probe, and iterate over positive, boundary, and low-negative strata.
3. Training and evaluation: Build a leakage-resistant split, sweep input resolutions, train deterministically, derive the operating threshold on validation data, and optionally evaluate a sealed external reference set.
4. Workbench and calibration: Display dataset boundaries and full-universe model disagreements, collect task-owner overrides, retrain, and review the resulting disagreement cohort.
5. Optional edge distillation: Export split-safe teacher embeddings and logits, compare CNN student loss recipes, and use deterministic stratum/logit sampling when a measured full-universe experiment exceeds the compute budget.
6. Optional hardware validation: Create a path-free validation plan, run the identified artifact through a separate target harness, and evaluate strict observations with a newly selected hardware threshold. Public bootstrap harnesses cover ESP32-S3 / ESP-DL and RK3588 / RKNN.
7. Deployment: Export a combined ONNX model and an optional downstream-agent prompt only when calibration and deployment sign-offs match the exact evaluated artifacts.

User Stories

• As a builder, I can provide an image path and a binary question to a repository-aware AI agent without manually running CLI steps.
• As an auditor, I can inspect blind contact sheets without seeing model proposals or score-color indicators.
• As a task owner, I can clarify edge-case business rules and update labels without editing source manifest files in place.
• As a reviewer, I can examine images nearest a decision threshold alongside prediction disagreements between two candidate models.
• As a mobile user, I can open workbench review pages over an explicitly enabled local network without exposing private manifests or absolute paths.
• As a deployer, I can verify that the exported ONNX model, operating threshold, review queue, and sign-offs reference the identical model evaluation state.
• As an edge developer, I can cap a redundant teacher universe without dropping owner labels or changing the sampled rows between student seeds.

Functional Requirements

• Frame extraction from video files and gated model weight downloading
• Full-directory and manifest-based DINOv3 CLS feature extraction
• Diversity, cluster, and acquisition-stratified active sampling
• Optional resumable candidate labeling via an LM Studio-compatible endpoint
• Blind contact sheets with stable audit IDs and exact-coverage audit merging
• Leakage-resistant dataset splitting and class-balanced deterministic probe training
• Precision-recall curves, average precision, threshold derivation, and resolution sweeps
• Static dataset boundary and model comparison pages accompanied by private review queues
• Append-only task-owner override provenance and dataset semantic digests
• Calibration and deployment sign-off validation
• Combined backbone-plus-probe ONNX export and downstream prompt generation
• Split-safe single-head teacher artifacts containing aligned DINO embeddings and raw logits
• Hard-label, Bernoulli-logit, and centered-cosine embedding distillation into timm CNN students
• Deterministic teacher-universe caps that retain all labels and stratify unlabeled rows by leakage-resistant group and teacher-logit decile
• Candidate student checkpoints that bind preprocessing, threshold, source and per-image hashes, training configuration, and ordered teacher-selection identity
• Path-free hardware validation plans, closed-schema observation JSONL, inclusive AP/parity/latency gates, and non-approving result artifacts
• Public-safe ESP32-S3 / ESP-DL and RK3588 / RKNN harness examples without bundled SDKs, models, or private data

Non-Goals

• Fine-tuning the frozen DINOv3 backbone
• Replacing human judgment in safety-critical decisions
• Real-time video streaming server infrastructure
• A universal vendor-neutral NPU compiler or bundled vendor SDK distribution
• Ungated deployment of distilled or quantized student models
• General object detection, bounding-box localization, or semantic segmentation
• Browser-based editing of source manifests
• Supporting more than two models in a single disagreement page in the initial release

Success Criteria

• A user can supply the example prompt from the README to a coding agent, enabling the agent to discover and follow the canonical skill.
• Every CLI command features reproducible inputs and outputs, eliminating the need for ad hoc data transformation scripts.
• Synthetic tests cover success and failure paths for sampling, audit coverage, provenance tracking, workbench privacy, and deployment gates.
• A private-data run demonstrates end-to-end execution without allowing sealed reference labels to influence data mining or threshold selection.
• Generated deployment prompts reflect single-head sigmoid semantics accurately.
• Static HTTP serving exposes only generated HTML files and optimized image assets.
• Private artifacts remain outside tracked repository files and the served static directory.
• Final export rejects unsigned data, stale classifiers, mismatched thresholds, incomplete cohorts, altered queues, and rows without explicit owner verdict metadata.
• Distillation tests prove label retention, deterministic selection, cell coverage, projector exclusion, checkpoint provenance, and PyTorch/ONNX Runtime parity when optional runtime dependencies are installed.
• Hardware validation tests prove deterministic path erasure, artifact re-hashing, exact observation coverage, independent threshold selection, gate-failure exit semantics, and fail-closed schema handling without hardware SDKs.

Task performance targets depend on the domain. The workflow must report measured validation and external metrics without promising universal accuracy, fixed latency reduction, or guaranteed cost savings across arbitrary tasks.