DINOv3 Classifier Skill

August 6, 2026 ยท View on GitHub

Convert a repeated binary visual question into a compact, local ONNX classifier using active learning, blind visual audits, and a frozen DINOv3 backbone.

This repository combines two core assets:

  • An AI-agent execution workflow defined in skills/skill.md.
  • A Python CLI for sampling, probe training, evaluation, visual inspection, provenance verification, and ONNX export.

It is designed for binary visual questions such as "is the loading bay occupied?" or "is the safety gate open?" when sending every image to a large vision-language model is too slow or too expensive.

Use It With An AI Agent

The most straightforward way to use this repository is to hand its URL and your task to a coding agent. For example:

Install and use the DINOv3 Classifier Skill from:
https://github.com/grapeot/dinov3-classifier-skill

Read AGENTS.md and skills/skill.md before doing any work.

My unlabeled images are in: <local path>
My visual question is: <yes/no question>
The positive class means: <business definition>

Keep all source images, labels, model weights, and review queues private and
out of git. Show me boundary cases and model disagreements for approval before
exporting the final ONNX model. Do not use any existing labeled reference set
during mining, training, threshold selection, or stopping.

The agent clones the repository, sets up the Python environment, executes the workflow, and pauses whenever task-owner decisions are required. Repository-aware CLI agents like OpenCode, Claude Code, and Codex can run this prompt directly.

What The Workflow Does

  1. Embeds the unlabeled image collection once using frozen DINOv3 features.
  2. Selects diverse seed images, then mines likely positives, boundary cases, and low-confidence negatives.
  3. Treats vision-model predictions as proposals and requires blind visual audits before probe training.
  4. Trains a deterministic linear probe and selects the operating threshold exclusively on validation data.
  5. Generates a mobile-friendly visual workbench for dataset boundaries and full-universe model disagreements.
  6. Applies task-owner corrections with append-only provenance records and artifact digests.
  7. Exports a combined DINOv3-plus-probe ONNX model only after reviewed artifacts and deployment sign-offs match.

The heavier visual model is used during dataset creation and review. The deployed classifier runs a compact linear head over a roughly 22M-parameter frozen backbone. Latency and cost savings apply strictly to repeated inference, not to the initial mining and audit phase.

Human And AI Responsibilities

The AI agent executes the workflow and inspects artifacts. A vision-capable agent may perform visual audits; otherwise, it must delegate them to a capable visual model or human reviewer. The human task owner remains responsible for defining what yes and no mean in ambiguous, partial, or safety-relevant scenarios.

The deployment gate is intentionally fail-closed. Retraining surfaces a new model disagreement cohort that must be reviewed explicitly rather than inheriting verdicts from an earlier checkpoint.

Installation

Requirements:

  • Python 3.10 or newer
  • uv
  • Approved access to gated DINOv3 weights on Hugging Face
  • ffmpeg (required only when extracting frames from video files)
  • A capable visual auditor, available either directly to the agent or via a local tool such as LM Studio
git clone https://github.com/grapeot/dinov3-classifier-skill.git
cd dinov3-classifier-skill
uv venv
source .venv/bin/activate
uv pip install -e '.[dev,inference,distillation]'
pytest -q

LM Studio provides an optional local adapter for generating inexpensive candidate labels. It is not an authority for ground-truth labels and is not required when the agent can audit candidates through another vision-capable path.

Main Capabilities

  • Frame extraction and gated weight downloads from Hugging Face
  • Full-universe CLS embedding caches with image-size provenance
  • Active sampling across diversity, cluster, and acquisition-stratified batches
  • Resumable label proposal generation
  • Blind contact sheets with stable audit IDs and exact-coverage verification
  • Leakage-resistant train/validation splits by session, date, or location
  • Class-balanced single-head linear probes and legacy two-head support
  • Precision-recall curves, average precision, validation-only threshold derivation, and resolution sweeps
  • Static workbenches for dataset boundaries and two-model disagreement cohorts
  • A local HTTP server that serves HTML and optimized images without exposing private JSONL queues
  • Owner override management, calibration sign-offs, deployment sign-offs, and artifact-bound ONNX export
  • Optional split-safe DINO-to-CNN distillation with deterministic capped teacher-universe sampling
  • Student candidate inspection, unapproved candidate ONNX export, and direct checkpoint/ONNX manifest scoring
  • Target-neutral hardware plans and strict observation evaluation without vendor SDK dependencies
  • Public bootstrap harnesses for ESP32-S3 / ESP-DL and RK3588 / RKNN

The CLI primitives are orchestrated step by step by the agent workflow. Run dinov3-classifier --help to list available subcommands. See skills/skill.md for the full execution guide.

Privacy

The repository ignores common generated artifacts and everything stored under data/ and models/. Keep all private files in those directories, verify staged content before committing, and do not bind the workbench server to 0.0.0.0 unless local network access is explicitly intended.

The static server enforces an allowlist. It serves generated HTML pages and optimized JPEG assets while returning 404 for manifests, score files, private queues, path traversal attempts, and symlink escapes. Generated pages display filenames rather than absolute file system paths.

Validation And Limits

The workflow was validated on a private garage-door dataset through deterministic retraining and generation of the final disagreement cohort. The current signed final export remains intentionally blocked on task-owner review; an earlier provisional ONNX smoke test is not the final artifact. See docs/case-study-garage.md for methodology, empirical results, and limitations.

distill writes student_candidate.pt. Use inspect-student to validate its schema, exact timm version, candidate flags, preprocessing, and source provenance without opening any manifest image. export-student-candidate creates an ONNX candidate plus a hash-bound sidecar; both retain artifact_role: candidate and deployment_approved: false. score-student reads ordered manifest rows directly and produces raw logits, probabilities, image and artifact hashes, preprocessing, and runtime provenance for workbench review.

Candidate export is not deployment approval. A student must still be scored over the complete review universe, compared against the reviewed teacher, pass complete disagreement review, and receive explicit task-owner approval. This repository does not provide an approved student deployment export.

Hardware validation boundary

create-hardware-plan validates a current ONNX score-student NPZ, sidecar, manifest selection, image contents, runtime artifact, and acceptance gates. It writes a deterministic path-free plan containing indexed image hashes, binary labels, and reference logits. A separate target-specific harness executes the identified runtime and returns the exact run-header plus observation JSONL contract documented by skills/skill.md; the CLI never accepts an arbitrary runner command. This repository includes bootstrap harnesses for ESP32-S3 / ESP-DL and RK3588 / RKNN under examples/hardware/, while vendor SDK installation and generated artifacts remain external.

evaluate-hardware --plan ... --candidate-onnx ... --runtime-artifact ... --observations ... --output ... re-hashes both artifacts, rejects incomplete or reordered observations, derives a fresh maximum-F1 threshold from hardware logits, reports fixed-source-threshold parity and linear-percentile latency, and evaluates the declared AP, mean probability error, and p95 latency gates inclusively. Integrity errors write no result. A valid passing result exits 0; a valid gate failure writes status: fail and exits 1. Plans and results always retain deployment_approved: false and contain no source paths or per-row result data.

Current limitations include:

  • No backbone fine-tuning
  • No single-command autonomous mining loop
  • No real-time video streaming server or universal vendor-neutral NPU export pipeline
  • No browser-based manifest editing
  • Workbench model comparisons are limited to two models
  • No public image test fixture that runs the full backbone end to end in continuous integration
  • Distilled students remain unapproved candidates; the complete disagreement-review and owner-approval gate has no approved student export command

Documentation

License

MIT. See LICENSE.