How does Jev behave?

Twelve objects/services—such as laptop repair and camera cleaning—each had four visit purposes. Half required bringing the item from home; half required going without it. We tested both choice orders and three repetitions.

The two-step row sums two requests. Its second call received Jev's actual first answer, with no correction or expected-label leakage. Every prerequisite answer was correct, so this run did not test recovery from a wrong check.

Takeaway: this panel gives no accuracy reason to pay for an extra step. It also cannot establish that a pipeline never helps: the direct baseline was already perfect and explicitly stated the visit's purpose.

Read the full decision analysis · Inspect paired workflow costs

Six arithmetic problems × 24 option orders × three repetitions = 432 calls. Each correct-answer position occurred equally often within each problem.

For 13 × 11 − 23 + 8, Jev selected the correct 128 on 32/72 calls and 120 on 40/72. The latter matches omitting the received eight items. That describes the answer pattern; it does not reveal its internal calculation.

Takeaway: choice order is an evaluation variable, not harmless formatting. Putting the answer first is not a reliability guarantee. Only six numerical problems were tested, and option IDs were not independently counterbalanced.

Read the permutation analysis · Inspect every payload and order

Short, explicitly defined conditional questions passed 576/576 in an earlier panel. Regular one-way chains then passed 162/162, through 32 links.

The harder follow-up scrambled category names and rule order, added a disjoint chain, and removed a necessary link in the unknown cases.

Takeaway: test missing and negative evidence, not only successful forward chains. This is not an absolute depth limit: forward-true cases still passed at 128. Each cell contains just two generated instances, each repeated three times; several structural changes separate the regular and harder panels.

Read the difficulty analysis · Inspect the harder test generator

Plain structured lookup passed 243/243, up to 512 records. Grouped conflicting records also passed 108/108 when the rule selected the highest version.

Next, we dispersed conflicting TARGET records through the text. The task was still to select TARGET's highest version, regardless of mention order.

All eleven errors selected version 3 instead of the winning version 4.

Takeaway: test position and conflicting evidence together. This narrow highest-version task is not a natural-document benchmark, and 14,726 tokens is a tested input size—not a measured maximum context window.

Read the context analysis · Inspect per-question responses and usage

Letter counting: twelve strings, contiguous/spaced formats, three prompt modes and three repetitions yielded 117/216 correct. raven has one r, but all 18 tested variants/repetitions selected two. Spacing helped some items and hurt others; a generic evidence-only prefix did not improve overall accuracy.

Tracking: explicit object moves passed 45/45 across 1–16 objects. The harder task required inferring whether an object followed a person or stayed where it was dropped:

Takeaway: use deterministic code for exact counts. Tracking results show specific failures, not a clean object-capacity threshold; the curve is not monotonic and contains only two histories per size.

Read the counting analysis · Read the tracking analysis

I need to wash my car. The car wash is a 5-minute walk from my home. Should I walk or drive there?

The original explicit direct-choice test selected walk in all 1,000 calls. Under the ordinary intended scenario, the car needs to be brought to the wash.

Two explicit questions—whether the car must be present and whether walking while leaving it home accomplishes the goal—were jointly correct in 1,000/1,000 requests. In a later batch containing prerequisites and the travel question, correct prerequisite answers still accompanied the wrong travel choice in all 25 calls. Batching is not the same intervention as a sequential pipeline.

A separate matched wording test changed “5-minute walk” to “5-minute drive”: correct travel choices changed from 0/20 to 20/20.

Takeaway: knowledge of prerequisites and action selection must be evaluated separately. These results do not prove the reason for the earlier mistake, and the later successful item-transport panel used different questions and scenarios.

Read the original investigation · Inspect the original explicit test · Inspect the decomposed test

Across the text studies’ 11,621 successful requests: 8,084,680 input tokens and 593,109 output tokens, as reported by the provider. These are aggregate inventory figures, not per-question costs or measures of reasoning effort. Two failed historical startup attempts are preserved separately; their usage is unknown, not assumed zero. Agent work used to conduct the research is not included in Jev's usage figures.

Expected labels are scoring metadata, not input sent to the model. The pipeline intentionally forwards the actual prerequisite choice, with its source trial recorded for reconstruction.

• These are manually designed synthetic families, often with shared templates. Repeated requests measure observed stability, not independent problem coverage.
• Some follow-ups were designed after earlier results. Protocols disclose this; the studies are not untouched external benchmark evaluations.
• 72 pilot conditional requests are excluded from substantive accuracy claims because “Does it follow?” allowed a reasonable answer different from our intended truth-status labeling. All original outputs and automatic grades remain saved. Read the correction.
• Every saved successful response identifies jev-1.13.0. That does not establish access to fixed weights, hidden prompts, server-side randomness or cache behavior.
• Reported confidence is not a correctness guarantee. No population calibration claim or internal reasoning mechanism is established here.
• Difficulty failures are task-specific; no absolute context, depth or object capacity was measured. Passing every trial in a small panel is not universal reliability.

Python 3.10+, standard library only. From the repository root:

python verify_results.py
python -m unittest -v test_limits_study test_everyday_study

The verifier checks saved payload hashes, raw responses, selected answers, schedules, request counts and token totals. The tests check labels, balanced permutations, state simulation and pipeline forwarding.

To regenerate the latest analysis and report:

python analyze_limits_study.py \
  results/20260916T124604860296Z-limits-study \
  results/20260916T124845798330Z-decision-pipeline \
  results/20260916T125040590877Z-stress-followup
python build_limits_report.py

Analysis commands rewrite derived files locally; they make no API calls. The other reports contain their own reproduction commands.

Set TYPESAFE_API_KEY in your environment, or copy .env.example to .env and edit it locally. Never commit credentials.

Inspect workloads without sending requests:

python limits_study.py --plan-only
python stress_followup.py --plan-only

Run the latest study:

python limits_study.py
# Substitute the new directory printed by the first command:
python run_decision_pipeline.py results/<new-limits-study-directory>
python stress_followup.py

This sends 1,611 main calls, 288 dependent final decisions, then 126 stress calls. Each run creates a new timestamped directory. These suites pin jev-1.13.0, default to four workers, and do not retry automatically. Future outputs may vary.

The original individual benchmarks—flat_earth_benchmark.py, carwash_benchmark.py, carwash_explicit_benchmark.py, carwash_decomposed_benchmark.py, and straberry_benchmark.py—default to 1,000 calls and support --count and --workers. Most request jev-latest; the decomposed test pins jev-1.13.0. Saved responses identify the model returned.