Jev pick-and-place study

A small, reproducible MuJoCo pilot exploring whether Jev can choose the actions for a pick-and-place task, and how that compares with Claude Haiku and simple rules. The aim is to support an honest research discussion and build on it later.

Watch the demo · Results table · Full results · Evaluation traces

Initial results

Ten unseen starting seeds, each tested normally and with a target change after lifting: 20 episodes per controller. Same observations, small actions, motion controller, success test, and 100-decision limit.

Controller	Ordinary	Changed target	Median decision	API cost per success*
Reactive rules	10/10	10/10	<0.01 ms	$0
Jev 1.13.0	10/10	10/10	146 ms	$0.0035
Claude Haiku 4.5	4/10	1/10	732 ms	$0.4174

*Includes API spend on failed evaluation episodes; excludes development and local compute. Evaluation API spend: Jev $0.0695; Claude $2.0870. Prices are estimates from returned token usage. More timing/cost detail is in table.csv.

The main finding: Jev chose exactly the same action sequence as the rules controller in all 20 episodes. It reliably executed the supplied policy, but this fixed task showed no task-performance advantage from adding AI.

What this does and does not show

The model chooses one bounded command every 0.2 simulated seconds. Shared code handles geometry and executes movements of 5-20 mm. Inputs include exact positions, contact/alignment flags, and explicit task rules. There is no vision or training. Success requires a grasped lift, transfer, release, retraction, and stable placement.

This is one engineered interface and prompt setting. Haiku used structured action-only output with thinking disabled; this is not a general model ranking. The shared controller already follows the current target, so the target-change result is not evidence of sophisticated AI replanning. Simulation pauses during API calls; completion estimates add simulated duration and measured decision time. They are not hardware cycle times. Twenty trials are exploratory evidence.

The demo uses smoother motion increments and actual 500 Hz physics states exported at 50 fps. Playback speed and omitted API waits are labelled. The yellow disk is a flat tabletop target marker. Protocol and raw traces are included.

Run or replay

Tested on Windows with Python 3.11. Install uv, then:

uv venv --python 3.11
uv pip install --python .venv\Scripts\python.exe -r requirements.txt
Copy-Item .env.example .env  # Add your own API keys locally; never commit this file.
.venv\Scripts\python.exe compare.py --method rules --split eval --start 1001 --count 10

Replace rules with jev or claude for paid runs. Caps are $1 Jev / $5 Claude; keep the local spending ledgers and traces to preserve caps and resume progress. Credentials and generated working files are Git-ignored. To replay the provided video without API calls or keys:

.venv\Scripts\python.exe -m zipfile -e results\evaluation_traces.zip outputs\comparison
.venv\Scripts\python.exe demo.py render outputs\comparison\eval_jev_ordinary_1001.jsonl

Code: demo.py (simulation/video), compare.py (controllers/evaluation). Sources: TypeSafe API, Jev pricing, Claude models, Fetch environment.