OpenJev benchmark figures

Paper draft (PDF) · LaTeX source and build instructions

These figures visualize existing development evidence. No new model training or gameplay runs were performed to produce them. Each experiment has a separate scope; they do not form a single leaderboard.

Chess

All twelve published candidate-policy fits were evaluated on the same 4,096 externally sourced positions after excluding previously encountered roots and legal successors, including mirrors. Mean move agreement is 26.03% for direct, 26.36% for action only, 26.47% for exact delta and 26.79% for full afterstate. These small descriptive differences do not change the earlier failed continuation criteria. Transfer results and evidence.

Exact native candidate consequences give a small agreement gain: 33.15% ordinary and 27.10% shifted for delta versus 31.75% and 26.81% for direct scoring. All four required bounded-loss comparisons and both required game-score thresholds fail. Delta's full CPU decision averages 3.415 ms versus 0.617 ms for direct. All twelve fits, 286 scored games and two unfinished games are retained. Candidate results, weights and first scheduled replay.

The larger policy reduces bounded engine-score loss by 22.99% on ordinary positions and 9.26% on shifted positions, below the required 20% on both. Raw shifted centipawn loss worsens. Its 18 wins, 64 draws, 12 losses and two unfinished games give a 52.08-54.17% point bound, below the 60% continuation threshold. Parameters and compute are unequal. Capacity results, all six weights and replay.

Nine original chess students trained on the same 4,096 Stockfish-labeled positions. Across three fits, teacher agreement on 1,024 development positions was 14.36% for the circuit, 14.19% for rewired and 14.52% for GRU. The circuit's 0.16-point gain missed the required 3-point improvement. All fits, weights and protocol.

The separate arena compares the fixed seed-17 students with ChessFly, ChessLFM, Qwen and Astra through Codex. It reports 24 public first-move puzzles and an 18-game color-paired schedule. These models differ in training and compute; the panel is not an Elo estimate or a matched architecture comparison.

Recurrent world models

The latest fixed-score memory comparison passed 25/25 checks. Persistent GRU lowered native control cost by 31.86%/25.99% versus cached-observation GRU and 38.22%/35.36% versus current-packet GRU on six-step/ten-step gaps. Every paired fit improved. The secondary cached-MLP reductions were 18.15%/12.96%. All twelve saved fits were evaluated without retraining. All 51 rows, fixed first-case GIF and audit.

All planned rows now use CEM256 and geometry scoring. The three supplied-physics references still beat every learned fit on every panel, while taking more measured controller time. Equal candidate budgets do not match information or total compute. Independent audit replayed 78,933,504 executed/candidate/selected native transitions, plus 310,464 public-observer transitions, with zero discrepancy. The result supports these trained memory policies in one development environment, not biological wiring or a new architecture. A stronger two-observation/action-history baseline is next.

The preceding zero-fit scoring intervention passed 25/25 checks. Approximate geometry reduced persistent-GRU native control cost by 35.7%/28.7% on six-step/ten-step sensing gaps. With the same geometry score, GRU beat cached MLP by 22.6%/15.7%, improving every paired fit. This supports changing the score while retaining the learned transition model; it does not establish architecture or biological novelty. All 51 rows, audit and fixed-case replay.

Geometry added 52.8%/53.4% to GRU whole-controller time on the two gap panels. These shared-host measurements include trace work and do not match total compute. The separate auditor replayed 329,088 native transitions with zero discrepancy, without new learned-model calls. Physics references use a single 64-sequence bank, while learned controllers use CEM256. The earlier failed cache comparison below remains unchanged.

The earlier 15-fit cache comparison failed its continuation rule, passing 15/28 checks. Cached MLP lowered mean control cost by 6.32%/5.57% versus persistent GRU on six-step/ten-step gaps and improved every paired fit. It also beat the packet MLP by 5.04%/5.22%; cached GRU instead worsened relative to its current-only control. Persistent recurrence predicted angles and rewards better on the shared prediction cohort but controlled worse than cached MLP. These are development results in one environment, not cache equivalence or an architectural novelty claim. All results, GIF and reproducible evidence.

Cached MLP used about 38% more whole-row time than persistent GRU on the gap panels. All 15 fits and 60 rows are retained. Total execution took 1,234.34 seconds, followed by a 36.72-second audit of 235,200 native transitions with zero discrepancy. Timings include validation, copies and traces where charged to controller rows; training and global hashing remain separate. Equal updates and near-equal parameters do not match compute.

The completed 12-fit Reacher memory comparison passed 25/25 checks. Persistent GRU recurrence lowered mean control cost by 9.27%/9.00% versus a GRU trained without cross-decision history on six-step/ten-step gaps. Its longer-gap advantage over a three-packet GRU was 6.51%. Every paired GRU comparison improved on both gap panels, although one fit contributes much of the current-only mean gain. The MLP was much closer, with only 1.06%/1.72% lower cost for persistent recurrence in the family means and a paired fit that beats persistent recurrence. Supplied-physics references remain better. This qualifies useful public-history memory under the fixed recipe, not biological wiring, velocity inference or a new architecture. All fits, criteria, prediction errors and replay.

All four families share the training corpus and 1,152 updates per fit. They do not match compute. Three-packet training took 2.89 times persistent-GRU fitting time. The complete execution took 1,083.97 seconds, plus 30.97 seconds for a saved-output audit that replayed 206,400 native transitions with zero discrepancy. Plot timings amortize the whole controller row, including setup, native steps and traces; fitting and final global hashing are separate. Shared-host measurements do not establish isolated latency or Rust/Python speed differences.

The completed Reacher search comparison reused all six saved recurrent models on fresh cases. For the residual family, adaptive cross-entropy search lowered native control cost by 25.48% ordinarily and 25.26% with longer sensor gaps, compared with random search using the same 256 candidate evaluations per decision. Every paired residual fit improved; the primary 8/8 and adaptive-search competence 15/15 checks passed. Supplied-physics references remain better. This establishes a planning improvement, not a memory, connectome or JEPA advantage. The earlier reward-head study's failed continuation rule remains unchanged. Full results, all fits, diagnostic exceptions and reproducible traces.

Candidate-scoring budgets match; FLOPs are not claimed equal. Decision times include loading saved proposals, scoring, recurrent updates and trace storage, amortized over batched cases, and exclude row setup and native stepping. These shared-host measurements are throughput figures, not isolated single-agent latency.

Autonomous follow-up: twelve fresh fits completed 12.58 million training interactions. Same-size success averaged 17.19% for either store versus 34.38% for GRU; eight fits timed out on every intact evaluation episode. No cue swap changed a scored branch choice. Selective writing failed its continuation rule, despite the supervised retention result below.

The initial seven-action supervised check failed across four memory architectures. A separate factorial diagnostic found that training over the two relevant candidates solves the four training examples, while ordinary recurrence remains unreliable on longer routes. With the corrected objective, both associative stores reach 100% at every length and fall to 50% under store-only resets. Selective writes tie global writes and fail the superiority gate. These are four-context supervised diagnostics, not gameplay or unseen-task scores. Protocols, all fits and interpretation.

An initialization audit found nonzero actor and critic gradients before any reward in twelve initial PPO rollouts. Zeroing the value head removes those gradients on the same data. This is a measured initialization effect, not evidence that the change improves learning.

The completed zero-critic follow-up reused all original controls and trained twelve fresh fits. Same-size success rose to 50.52% for the adapter and both stores, but GRU fell from 34.38% to 17.19%. No cue swap changed the recorded behavior, and resetting the selective store changed no outcomes. Both continuation rules failed. This is an initialization effect on previously scored development seeds, not acquired memory or independent confirmation.

The follow-up made the cue visible at the start: 12 fits, 6,291,456 training interactions and 17,664 evaluation episodes. Recurrent and predictive PPO both reached 29.2% same-size success versus 52.1% for current-only PPO. No scored branch choice changed under a cue swap, and state clearing changed no outcomes. Results, learning curves, transfer and retention probe.

Twelve fits completed 6,291,456 training interactions and 8,448 evaluation episodes on an adapted MiniGrid Memory task. Full prediction reached 30.2% same-size success versus 51.6% for recurrent PPO and failed the predeclared continuation rule. Clearing recurrent state left all scored episode outcomes unchanged; a replay audit found that no policy acquired an initially unseen cue. Protocol, all fits, learning curves and actual policy GIF.

Text student controls

Astra through Codex: four fresh teacher batches labeled 128 training examples, then three MiniLM students reached 62.12% domain-routing accuracy versus 9.47% untrained and 60.23% gold-supervised. BoolQ remained at 51.19% and failed the combined continuation rule. All three fits are shown on the same 172 previously scored development examples; this is hard-label distillation, not an independent confirmation or architecture result.

Bound-rule follow-up: 12 learned fits and three fixed controls on previously unused development worlds. A 225-parameter operator with a handwritten parser and explicit logical operations reaches 100% shift macro accuracy. Sixteen recurrent steps also solve all 240 constructed counterfactual pairs. The fixed solver matches 100%; novelty and a runtime advantage remain unproven. The frozen selector chose six steps on a tie and failed its longer-chain challenge gate.

Full-context follow-up: three fine-tuned MiniLM fits reached 69.81% raw shift accuracy but only 49.36% after equal weighting of label/negation groups. A training-fitted word-only control scores 87.89% raw and 50% balanced. The shortcut audit is post-hoc; no reasoning or novelty gain is established.

RuleTaker memory study: 18 fits on 19,809 training questions. Three reads reached 56.95% on depth 3-5 development questions versus 64.21% for the question-only control. The continuation rule failed. These are selected, verified development subsets, not the full official benchmark.

Corrective-recurrence follow-up: 18 fits produced a small development gain, 95.35% versus 95.10%, below the predeclared improvement threshold. Confirmation stayed closed. This is not an established novel-method benefit.

Learned-head follow-up: 12 matched-parameter fits raised development accuracy to 95.10% for the linear metric control. Dense and sparse recurrence did not improve it. The independent confirmation set remains unscored.

Associative architecture follow-up: a full-training-bank prototype reached 91.6% in-scope accuracy on development data and outperformed the tested recurrent retrieval. Confirmation was not opened. This is a different task and data budget from the small student pilot below.

Gold-label training raised CLINC domain-routing accuracy from 9.5% to 60.2%, averaged across three fits, with substantial fit-to-fit variation. BoolQ remained near chance. These are balanced, filtered subsets, not full benchmark scores. This original Responses API teacher arm remains unrun; the separate Codex result appears above. Protocol, all fits and limitations.

JEPA rewards and RL variants

The actual pretrained V-JEPA 2 reward arm continuation criteria not met. All nine methods received 8,192 additional interactions per fit, across three corresponding warm-start policies. Dots show means; crosses show individual fits; paired intervals resample both training and evaluation seeds. Local training time includes video processing and updates but excludes shared historical PPO and external JEPA pretraining. Full report and evidence.

SRPO adaptation pilot

Latent self-reference failed its continuation criteria: lower mean Center kills than binary rewards, raw similarity and unchanged PPO, with wide exploratory intervals. Line outcomes matched across all methods. Costs include rollouts, updates, encoding and I/O, and exclude shared historical PPO training. Equal trajectory-group budgets do not imply equal interactions or compute. Full SRPO report. The paper PDF above covers the preceding studies through PPO/DQN.

Direct reinforcement learning

PPO with history passed the frozen combat gate against event-memory rules on 96 new seeds per scenario, retaining all three training fits. It also improved Center kills against current-only PPO and the simple controls in exploratory secondary comparisons. Line actions match always-fire. Dots are means; crosses show each training fit's mean kills, not confidence intervals. Full report and intervals.

The training curves show trailing 20-episode means, including exploration, and do not substitute for held-out evaluation. The old command-window metric can miss delayed hits under alternate fire/wait schedules. The new study uses kills and duration as its primary outcomes.

Prospective improvement studies

All three studies and failed continuation gates. These ran 6,408 new episodes; the figures below were generated from preserved results. Each confirmation uses 96 paired seeds in both scenarios. Bars are exploratory 99.375% bootstrap intervals, adjusted within the primary comparison family of each study.

The combined event-memory rule reduces redundant commands with identical paired kills, but fails the broader comparisons against historical models.

The selected fitted model improves utility over rules, but does not establish kill noninferiority in Line or clear the historical-bank comparisons. All component comparisons.

Original gameplay

Twenty matched seeds per policy, two-tic actions, Defend the Center. Every dot is an episode; red marks are means. The 5,253-parameter imitation model approximately matches its rule teacher. This uses full steering policies, unlike the Bayesian firing-only comparison. Episode data · Vector figure.

Bayesian decision utility

The 440-episode valid development study includes training, audit and evaluation episodes. Learned methods use five fits and ten paired evaluation seeds per scenario. Existing baseline episodes are reused across fits, not counted as independent copies. Seven-tic actions and rule steering are shared. Intervals are exploratory, unadjusted paired crossed-bootstrap intervals. Protocol and limits · Analysis.

Probability prediction on a common audit

Lines pair MAP and Bayesian predictions from each of five fitted models. Diamonds are means, not confidence intervals. Each scenario uses the same ten audit episodes for all fits, with dependent events within each episode. A lower Brier score indicates better overall probability prediction, not calibration alone. The shifted audit's descriptive improvement did not establish better control utility. Vector figure.

Rust versus Python/NumPy scoring

Eleven timed batches per implementation, after three warmups, on identical float64 inputs: 32 rows, 151,936 vocabulary entries and six candidates. Points are batch milliseconds divided by 32; red marks show the median and interquartile range. Rust ran first, other Docker services were active, and only one environment was measured. NumPy uses native numerical kernels. These are implementation-specific timings, not a general language comparison or model-inference speedup. Raw timings and environment · Vector figure.

Regenerate

From the repository root:

uv run --no-sync --with matplotlib==3.11.2 python research/plot_benchmarks.py

This creates PNG, SVG and PDF figures, a LaTeX results table, and a source-hash manifest. It does not rerun experiments or overwrite their measurements. The earlier utility comparison has its own research/plot_bayesian_doom.py renderer.

Board-aware chess follow-up

Twelve final fits improve on the material heuristic, but future prediction does not beat the matched spatial controls. Full results and models · Frozen protocol.

Chess refinement

Targeted mating accuracy improved, while ordinary decisions regressed. More untrained recurrent steps also hurt. All results and scope.

Input-anchored chess recurrence

Input anchoring with varied-depth training preserves agreement at sixteen steps, but neither move-quality nor extra-computation continuation criteria pass. All results, checkpoints and scope.