Physical AI Data Factory - Auto Labeling

September 1, 2026 · View on GitHub

Overview

Auto Labeling is the PAIDF workflow repository for turning raw image and video datasets into reusable annotation artifacts and training-ready outputs. It composes containerized stages for super resolution, detection and tracking, captioning, Visual QA, reasoning, 2D grounding, referring expressions, person-attribute search, and training export.

Auto Labeling is cookbook-driven: each cookbook defines the input media, stage order, model endpoints, checkpoint mounts, prompts, question banks, and output layout for a complete labeling workflow. The workflow runner compiles those cookbooks into local container execution plans and writes results into a shared DAFT scene directory containing raw/, contextual/, task/, and sidecars/ artifacts.

Use this repo when you need to:

run a sample auto-labeling workflow after staging NGC sample media
adapt a cookbook to a new dataset or annotation target
validate stage outputs and DAFT artifact contracts
package completed annotations for downstream training or review

Quick Start

Prerequisites

Python 3.12 or newer
uv
GNU Make

Container workflows also require Docker or Podman. GPU stages require a compatible GPU runtime, such as Docker with the NVIDIA Container Toolkit, plus the checkpoints and VLM/LLM endpoints selected by the cookbook.

Run a sample workflow

make sync
make run SCRIPT=workflow-runner:main ARGS='--help'

Copy the nearest tracked cookbook to a gitignored *.local.yaml, replace its media, output, model-cache, and endpoint placeholders, then inspect the container plan before execution:

cp cookbooks/video_data_augmentation/configs/pipeline_video.yaml \
  cookbooks/video_data_augmentation/configs/pipeline_video.local.yaml

CONFIG=cookbooks/video_data_augmentation/configs/pipeline_video.local.yaml

ARGS="--cookbook-file ${CONFIG} --container-dry-run" \
  make run SCRIPT=workflow-runner:main

ARGS="--cookbook-file ${CONFIG} --container-user auto" \
  make run SCRIPT=workflow-runner:main

Full walkthrough: Getting Started.

Keep API keys in the host environment. Do not place credentials in cookbook files, command history, logs, or generated evidence. Pass the required environment-variable name to stage containers, for example --container-env NVIDIA_API_KEY or --container-env GEMINI_API_KEY. Remote paths also require Multi-Storage Client configuration.

Contribute to this repository

After make sync, run the contributor checks before you change code:

make lint-check
make mypy
make test

Details: Local Development. Model-client internals for new VLM/LLM wiring: Model Client Architecture.

Architecture

PAIDF Auto Labeling Architecture Diagram

Services package and expose reusable task implementations. The workflow runner launches selected services in a fixed relative order over a shared DAFT scene.

flowchart LR
    C[Cookbook or CLI] --> W[Workflow runner]
    W --> S[Stage service]
    S --> T[Task package]
    T --> D[(Shared DAFT scene)]
    D --> S

Dependency direction is services -> tasks -> core. Services communicate through DataEntry JSONL manifests and the scene directory, not through direct service imports. Core must remain reusable, and tasks must not import services.

See Services Overview for the stage order and Artifact Contract for file ownership and reuse behavior.

Components

Capability	Product reference
Workflow orchestration	Workflow Runner
Super resolution	Super Resolution
Detection and tracking	Detection and Tracking
Captioning	Captioning
Visual QA	Visual QA
Reasoning	Reasoning
Visual Attribute Search	Visual Attribute Search
2D grounding	2D Grounding
Referring expressions	Referring Expressions
Training export	Training Export
DAFT validation	Artifact Contract

example_service and example_task are scaffolding templates, not production pipeline stages.

Input and Output Contract

Service inputs are DataEntry records:

{"id": "clip-001", "media_path": "/data/clip.mp4", "data_path": "/output/clip-001"}

media_path identifies the caller-provided media.
data_path identifies the shared DAFT scene directory.
The shared pipeline preserves the original source as sidecars/raw.<ext> and promotes transformed media through sidecars/active.<ext>.
Existing active sidecars are reused across stage containers; callers should not rewrite media_path between stages.

Manual and scripted experiments use:

<experiment-root>/
├── input.jsonl
├── source/
├── logs/
└── data/
    └── <entry-id>/
        ├── raw/
        ├── contextual/
        ├── task/
        └── sidecars/

See Experiment Output Layout and the Artifact Contract for the complete contract.

Running Services

Use the interactive selector or specify a registered script:

make run
make run SCRIPT=captioning-service:main ARGS='--help'
make run SCRIPT=example-service:main ARGS='--input-file payloads/simple.jsonl'

Scripts are discovered from [project.scripts] in package pyproject.toml files. Interactive runs offer presets from [tool.run.default-args]; explicit ARGS overrides the preset and is forwarded unchanged to the selected script.

For larger inputs, prefer --input-file over inline JSON. For development runs that must preserve the source scene, pass --dev-data-root <path-or-url> to a service built on the shared service interface. The service copies each input scene to <root>/<entry.id>, replacing an existing ID-named copy, and runs against that copy. A missing local source starts as an empty scene; the development root may be local or supported remote storage.

Build registered images interactively or by target:

make build
make build IMAGE=captioning-service:main

Image targets are registered in service pyproject.toml files. See Stage Images for the complete target list. make build discovers them from [tool.build.images], and registered build commands run from the repository root.

Repository Layout

packages/core/                    # shared DAFT, media, model clients, and storage
packages/tasks/<task-name>/       # reusable annotation behavior
services/<service-name>/          # CLI and container packaging
services/workflow_runner/         # cookbook compiler and local launcher
cookbooks/<scenario>/             # workflow configs, prompts, and question banks
docs/                             # user-guide, developer docs, and compatibility pointers
docker/                           # shared Docker assets
observability/                    # collector and dashboard assets
payloads/                         # example DataEntry manifests
scripts/                          # workspace run, build, and media-toolchain helpers
skills/                           # repository-local agent guidance

Development

make help        # list supported targets
make sync        # install all workspace packages and extras
make lint        # format and fix lint findings
make lint-check  # check formatting and lint without changes
make mypy        # run static type checks
make test        # run tests, coverage, and JUnit reporting
make check       # sync, format/fix, type-check, and test

Before submitting a merge request, run the non-mutating checks or review any formatting changes produced by make check.

Pre-commit hooks are optional. Install them with uv tool install pre-commit --with pre-commit-uv --force-reinstall, then use the repository .pre-commit-config.yaml.

Documentation

Use the Documentation Index to choose a path:

Run the product: User Guide → Getting Started
Change the code: Developer Docs → Local Development

Repository-local guidance for supported agent workflows lives under skills/.

Project Policies

Contributing explains the development and review workflow.
Security Policy provides the private vulnerability-reporting path.

For non-sensitive questions and defects, consult the documentation first, then use the repository's GitLab issue workflow. Do not report security or conduct concerns in a public issue.

Responsible Use

Auto Labeling outputs are machine-generated annotations and may be incomplete, inaccurate, or biased. Apply human review and domain-specific quality checks before using them for training, evaluation, or operational decisions.

Only process media and metadata that you are authorized to use. Protect personal and confidential information, confirm applicable dataset and model licenses, and evaluate the downstream impact of generated labels for the intended use case.

License and Contributions

The PAIDF Auto Labeling Project is licensed under the Apache 2.0 license. This project is currently not accepting contributions.

Redistribution Notice

The PAIDF Auto Labeling Project redistributes modified code from other projects. Details may be found in the following files:

Boosttrack: see packages/tasks/detection_and_tracking/src/detection_and_tracking/backends/boosttrack/UPSTREAM_LICENSE.md

SeedVR2: see packages/tasks/super_resolution/src/super_resolution/UPSTREAM_LICENSE.md

Notice

NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided "AS IS", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk.