VeriPhy

September 4, 2026 · View on GitHub

VeriPhy — Agentic Physical Reasoning for World Model Evaluation and Refinement

Paper (arXiv:2609.03153) · Project page & video demo

VeriPhy

A critic that watches a generated video and tells you which parts of the prompt it did not deliver — and shows you the measurement behind each accusation.

Give it the prompt a video was generated from and the video. It breaks the prompt into individual checkable claims, works out how each one could be checked, runs those checks with specialist perception tools (detection, counting, tracking, segmentation, depth, sound-event detection, OCR, temporal grounding), and returns the claims the video contradicts.

The point is not that a model watched the video and had an opinion. The point is that "there are five balls" was answered by a counter, "the logo says BON APPÉTIT" by an OCR engine reading the actual pixels, and "we hear the balloon pop" by a sound-event detector with a timestamp — each one a measurement you can look at.

This repository is the critic. The refinement loop shown on the project page — fold the contradicted claims back into the prompt and resample with the same generator and seed — is what the paper builds on top of these verdicts.

A note on names. The system is VeriPhy. The code predates the name and still uses the prefixes it was born with: modules and environment variables are vac_* / VAC_*, and the original experiment's launchers export EXP029_*. Both prefixes are read at runtime (see Configuration), so the rename breaks nothing — but do not expect to find the string veriphy in the source. Use VAC_ for anything new.

What it does, on one clip

prompt ─► decompose ─► one checkable claim each
                          │
                          ▼
                       PLANNER (an LLM, writing short Python)
                       check(c0, eq(count("steel balls"), 5))
                       check(c1, before(window("ball is released"), window("ball strikes")))
                       check(c2, judge("the lighting is warm"))
                          │
                          ▼
                       COMPILER (deterministic — no model judgement here)
                       lower to tool calls ─► SHARE identical ones ─► execute
                          │
                          ▼
                       ARMS (specialist tools, one HTTP server)
                       count · object_detect · localize_event · track
                       segment · depth_pair · audio_check · text_read
                          │
                          ▼
              supported / contradicted / unknown, per claim
                          │
                          ▼
              contradicted claims = the critic's accusations

Two halves, split on purpose. The LLM decides semantics — what this claim is really asserting, and what would have to be measured to falsify it. The compiler decides mechanics — which calls are identical and can be shared, and how a measurement becomes a verdict. The compiler contains no model discretion at all, so the same plan always executes the same way, and the thing being evaluated (the plan) is separable from the thing being trusted (the tools). ARCHITECTURE.md walks through why.

Quickstart

git clone https://github.com/VeriPhy-AI/VeriPhy.git && cd VeriPhy
cp .env.example .env          # then point it at your serves and your data

# 0. offline tests — no GPU, no network, nothing to set up
scripts/test.sh

# 1. bring the tools up (one server hosting every arm)
scripts/serve_all.sh 8100
curl -s localhost:8100/health | python3 -m json.tool

# 2. ask every arm a question it already knows the answer to
scripts/canary.sh

# 3. score some clips
scripts/run_pass.sh --out pass.jsonl --clips sample:30:29 --clip-par 5

# 4. turn the rows into tables + a failure-mode breakdown
scripts/make_report.sh

Each of those is one command, documented in its own header. scripts/run_pass.sh --dry-run resolves the config, the clip set and the resume state and calls no tools — the fastest way to check a deployment is wired correctly.

What you have to supply

Nothing but code is in this repository. You provide:

env varwhat it is
a chat/vision serveVAC_ENDPOINT_REASONERany OpenAI-compatible endpoint. It is the planner, the generalist judge, and the absence verifier. The project ran Qwen3-VL-30B-A3B-Instruct.
the tool serverVAC_ENDPOINT_TOOLS_COMBINEDstarted by scripts/serve_all.sh, or split per arm
clipsVAC_CLIP_CACHEa directory of <clip_id>.mp4 (optionally backed by VAC_CLIP_S3_URI)
human labelsVAC_GOLD_LABELSonly to score the critic afterwards, never as input to it

Every variable is listed with a placeholder in .env.example. Both VAC_ and the original EXP029_ prefixes are accepted — see "Configuration" below.

Results

The paper's evaluation lives in the paper (arXiv:2609.03153): on a 149-clip core carrying 304 human-written flaw records, VeriPhy accounts for 228, against 164 for a published question-decomposition evaluator and 222 for prompting the same backbone monolithically. Cite those numbers, not these.

The numbers below are this repository's own scoring run, against a different baseline (the per-claim critic in critic/baseline.py) on a different clip set. They are here because they are the ones you can reproduce from this code with scripts/make_report.sh — every one is recomputed from the raw per-clip rows by analysis/results.py, nothing is typed by hand. Comparisons are paired clip by clip on the intersection actually scored, with 95% intervals from a 10,000-sample bootstrap at a pre-registered seed.

137 clips, 289 human-labelled flaws:

systemflaws caughtrecall
this critic (plan-based, all arms)19065.7%
per-claim baseline (one claim at a time, generalist only)17359.9%

Difference +17 flaws, 95% CI [7, 27] — the interval excludes zero.

On the 107 clips where four systems were all scored: plan-based with all arms 66.4%, plan-based with earlier arms 64.2%, per-claim baseline 60.7%, an earlier planner 56.8%.

analysis/failures.py answers the more useful question — why the misses were missed: 78% were checked and cleared (a claim covered the flaw and the critic said the video was fine), 19% were flagged but unlinked (the critic did accuse it; the matching protocol did not connect the accusation to the label), and only 3% were never decomposed. Of the checked-and-cleared misses, the large majority were cleared by a yes/no judgement with no measurement attached — which is exactly where more measurement coverage would pay.

Reproduce both tables with scripts/make_report.sh.

Layout

Organised by what things are, not by when they were built.

critic/       the critic client — everything that decides WHAT to check and WHAT it means
  ir.py                  the typed plan format (entities, events, observations, predicates, claims)
  planner.py             LLM writes short Python -> AST -> typed plan
  compiler.py            lower -> share identical tool calls -> execute typed 3-valued predicates
  value_source.py        the tool client: per-arm routing, async tickets, schema checks
  value_source_vlm.py    the generalist-VLM fallback substrate (what unroutable checks use)
  absence_verifier.py    bounded "is X present?" over the whole clip, for untyped claims
  baseline.py            the per-claim baseline critic + the gold matching used to score
  decompose.py           prompt -> atomic claims (doubled-emission proof, chunked)
  trajectory_predicates.py  the typed physical measurements taken over tracks
  clip_pass.py           run_clip: the whole pipeline for ONE clip. Start reading here.
  transport.py           chat transport to the reasoner
  infra.py               the one rule: a NET: result is not evidence

arms/         the perception tools — everything that MEASURES
  server.py              HTTP front; one lazily-started worker subprocess per arm
  workers/               worker_object, _count, _action, _track, _segment, _depth, _audio, _text
  windowing/             ask a VLM WHEN to look, then read densely INSIDE that window
  ocr/text_identity.py   recognition-head OCR + deterministic string comparison
  specialists/           the frozen output-normalisation contract for the model adapters
  launchers/             stage venvs + weights and serve, on a real GPU box

ops/          config.py (env + endpoints + data paths) · canary.py (known-answer probing)
              mirror.py (push the code to a remote box, hash-verified)
runners/      critic_pass.py — THE scoring runner, parameterised
analysis/     results.py (tables) · failures.py (why the misses were missed)
tests/        four offline suites, ~320 assertions, no GPU and no network
docs/         TOOL_SERVER_CONTRACT.md — the frozen wire protocol
scripts/      the one-command entry points
vac_paths.py  the sys.path bootstrap (read its docstring before moving any file)

Why flat modules and a sys.path bootstrap

Each arm runs in its own virtualenv, as its own process — their stacks genuinely conflict (the counting arm pins an old torch against a compiled detection stack; the temporal arm needs a newer transformers than the object arm; OCR is CPU-only). None of those venvs can pip install this repo. So modules find each other through sys.path, scoped per group by vac_paths.add(...): an arm asks for "arms" and never sees ops/config.py, so a generically-named module here can never shadow one of theirs. The full reasoning is in vac_paths.py.

Configuration

One module owns it: ops/config.py.

python3 ops/config.py --show      # the endpoint registry, the data paths, the map it builds
python3 ops/config.py --verify    # live: does every op land on a server that HOSTS that op?
python3 ops/config.py --selftest  # offline unit tests

Three rules are enforced there, each one a real outage this project had:

  1. Empty string means unset. Launchers pass optional knobs as "${VAR:-}", so an unset knob arrives as "" — and os.environ.get(name, default) returns "" because the key exists. float("") once killed the tool server at import; an empty OCR-engine variable once left no engine selected, so every text check abstained with an empty read while /health cheerfully reported "loaded". Read knobs with envstr / envnum / envflag, never a bare os.environ.get.
  2. Never hand-write an endpoint map. Every URL lives in one registry; tool_endpoint_map() builds the map string and refuses to point an op at a server that does not declare it; verify_routing() re-checks it live against each server's /health before a run starts. A missing entry does not fail loudly on its own — it routes an op to a server that does not host it, and the run then retries that clip forever instead of scoring it.
  3. One runner. runners/critic_pass.py is the only pass driver, and it calls critic/clip_pass.run_clip verbatim rather than copying it. Four near-identical runners is how a fix lands in one and silently misses the others.

Both env prefixes work. VAC_* is this repo's; EXP029_* is what the original experiment's launchers export. Both are read (VAC_ wins) and apply_env() mirrors whichever you set into the other, so old launchers and new code agree. Use VAC_ for anything new.

The canary — the most important operational habit here

scripts/canary.sh              # probe once; exit 1 if any arm is BROKEN
scripts/canary.sh --watch 900  # re-probe every 15 minutes, alarm on state change

Every serious failure this system hit was silent: the arm answered, /health said "loaded", and the measurement was empty or fabricated. An OCR engine imported fine and then threw on every single image (184 out of 184). A temporal-grounding arm returned a confident window for events that never happen. An empty environment variable left no OCR engine selected at all. A liveness check that only asks "did you reply?" cannot see any of these.

So each arm gets a known-answer canary: a clip whose correct answer is already known, and an assertion on the content of the measurement — the counter must return a count and say it derived it from per-frame modes; the sound detector must actually hear the laughter and return a timestamped window; the OCR engine must name its engine and come back with non-empty text on a clip that definitely has text on screen. An arm that fails its canary is reported BROKEN, never as a measurement. runners/critic_pass.py runs this as a gate before spending hours scoring, and refuses to start if any arm fails.

The canary clip ids are configurable (VAC_CANARY_*), but if you change them you must re-read the assertions in ops/canary.py — they assert on what those specific clips contain.

Ground rules the code enforces

These are load-bearing. Changing them changes what the numbers mean.

  • Three-valued, always. supported / contradicted / unknown. "Unknown" is a real answer for insufficient or ambiguous evidence — nothing is silently treated as satisfied.
  • Looked-and-absent is a measurement, not an abstain. A detector that searched and found nothing is honest evidence of absence and is scored as contradicted. It is never quietly converted to unknown.
  • Infrastructure failure is never evidence. A crashed or unreachable tool produces a result tagged NET:; the whole clip is retried, and if it still fails the clip is left unwritten so a later resume picks it up. It is never scored. (critic/infra.py)
  • Windowing saves frames OUTSIDE the window, never inside it. A VLM may be asked when to look; inside the window the read is dense — stride 1, every frame. If anything goes wrong (no endpoint, unparseable reply, degenerate window) it falls back to whole-clip dense. Never to a sample.
  • A forced stride is recorded, never silent. If a serve's per-request image cap forces fewer frames, the stride and the frames actually sent are reported in the result.
  • OCR reads pixels; it does not guess words. The pixel read is a recognition head, never a generative model — a generative reader's language prior "repairs" BovPRERJ= back into BON APPÉTIT and hides the exact flaw being hunted. The LLM only decides what string to look for; a deterministic Unicode-aware comparison decides whether it matched.
  • Gold labels are touched only after prediction. They are used to score accusations and for nothing else. The critic never sees them.
  • The wire contract is frozen. docs/TOOL_SERVER_CONTRACT.md is the same whether one server hosts all arms or eight servers host one each. A per-arm server is that same code with a filter, never a fork.

Provenance

This is a repackaging of the critic built as experiment EXP-029 in a larger research repository, lifted out of its experiment-numbered directories and reorganised by what things are. The code is carried across as-is apart from import paths, configuration and data locations; comments still reference the original ledger entries (E64, F26, F29, …) where they explain why a decision was made, because that reasoning is worth keeping. Deployment-specific hostnames, buckets and credentials were removed and replaced with environment variables. That history is also why the code says vac_ and EXP029_ while the system is called VeriPhy.

Citation

@article{xu2026veriphy,
  title   = {VeriPhy: Agentic Physical Reasoning for World Model
             Evaluation and Refinement},
  author  = {Xu, Wenzhuo and Zhu, Yuchen and Ge, Chongjian and Shen, Xuan
             and Shi, Jing and Kuen, Jason and Chen, Yongxin and Tao, Molei
             and McComb, Christopher and Grande Guti\'errez, Noelia
             and Gu, Jiuxiang},
  journal = {arXiv preprint arXiv:2609.03153},
  year    = {2026}
}