Jev Pong

Pong where the ball moves one step per model decision. Slow model, slow ball.

Play it: jev-pong.ably.dev, arrow keys or W/S, drag on a phone. The how page shows the real state and the real question every model gets.

Why I built this

I'm Matt, co-founder of Ably. Jev landed on Vercel AI Gateway on September 16, 2026, and two days later this was playable, because the model is a string id and one AI SDK call.

I wanted to see one thing for myself: whether a model that answers in a couple of hundred milliseconds changes what an agent can be. Everything we've built with AI so far waits. Request and response waits for the answer. Streaming and background agents wait more gracefully. A model that decides inside human reaction time can be a participant in what you're doing right now, and a game is the least forgiving place to test that, because the ball doesn't wait.

This is a latency demo, not an intelligence demo. The chat models answered the same question correctly 95 to 100 percent of the time in our runs. They're slow because they answer with language, and this question doesn't need a paragraph. Jev is a typed-decision model, state in and a typed choice out, so it doesn't generate any. That's the whole difference, and it's enough to make the ball move.

The other half of the demo is how the agent takes part. It isn't behind a request. It's a member of the same Ably channel as the player: present, reading the paddle input, publishing the ball, watchable by anyone with the link. That's the pattern we think live agents need, and it's why the code is open.

The numbers

Four lanes play the same game with a different model on the right paddle. Same serve, same rules, same question, same Vercel AI Gateway key. One decision moves the ball one segment, eight segments cross the court, and nothing is skipped or sped up. A lane's speed is that model's decision latency and nothing else.

Recorded on Vercel (iad1) through the Gateway on September 17, 2026, 45 seconds per lane. Every figure on the site is read from public/replay-stats.json at build time; nothing is typed in by hand.

Model	How it answers	Decisions per second	Average	p95	Decisions in the first 12 s
Jev (TypeSafe AI)	experimental_evaluate, a typed choice	4.4	227 ms	400 ms	47
Gemini 3.8 Flash	structured output	0.32	3.2 s	7.4 s	3
Claude Haiku 4.5	structured output	0.40	2.5 s	8.4 s	2
GPT-5.6 Sol	structured output	0.28	3.5 s	10.4 s	2

Jev is reachable through Vercel AI Gateway as typesafe-ai/jev with AI SDK 7.0.105 or later; see the Vercel changelog, the evaluate docs, and TypeSafe's launch post. The chat models are asked the identical question through the same Gateway with structured output at temperature 0, reasoning off, and no retries for anyone.

Why Pong: in the Atari-GPT benchmark, Pong was the game chat models handled worst, below random, because deciding takes them longer than the game gives them.

Since then: other models, same question

The four lanes were chosen to be simple: one typed-decision model against the three chat models most people recognise. After launch, people asked about the newest frontier models, about Qwen, and about the smallest chat models, so on September 19, 2026 the same question went to them too: 30 real game states, one call per decision, sequential, reasoning switched to the lowest setting each provider accepts (Astra refuses "minimal" and takes low, medium or high; Fable 5.1 refuses "disabled" and takes adaptive with an effort). It was measured from a Vercel function in iad1, next to the Gateway, which is where the lanes were recorded, so Jev's median here is the same round trip the front page shows. The data is public/model-comparison.json, with every counted latency in it, and the how page renders it.

Model	Setting	Correct	Median	p95	Slower than Jev
Jev	evaluate	28/29	222 ms	288 ms	1.0x
Ministral 3B	default	26/30	396 ms	588 ms	1.8x
Qwen 3 235B	thinking off	28/30	608 ms	919 ms	2.7x
Gemini 3.5 Flash-Lite	thinking off	30/30	660 ms	799 ms	3.0x
Claude Haiku 4.5	thinking off	30/30	791 ms	1885 ms	3.6x
GPT-5.4 Nano	default	29/30	799 ms	1070 ms	3.6x
Qwen 3.8 Max	thinking off	30/30	834 ms	1222 ms	3.8x
Qwen 3.8 27B	thinking off	30/30	950 ms	2928 ms	4.3x
Qwen 3.8 Flash	thinking off	30/30	951 ms	2358 ms	4.3x
GPT-5.6 Sol	default	30/30	1153 ms	1954 ms	5.2x
GPT-6 Astra-fast	reasoning effort low	30/30	1355 ms	3158 ms	6.1x
GPT-6 Astra	reasoning effort low	30/30	1542 ms	2113 ms	6.9x
GPT-6 Astra	reasoning effort high	30/30	1548 ms	2763 ms	7.0x
Claude Fable 5.1	adaptive thinking, effort low	30/30	4530 ms	6887 ms	20.4x

Astra at medium effort landed with low and high, all within 200 ms of each other on a one-token answer, and Fable at high effort came in at 3.6 s with an 11 s p95, so the setting is not what makes them slow. Qwen 3.8 thinks by default (85 output tokens and 3 to 11 s for a one-word answer); enableThinking: false is the switch, and the Qwen rows use it. Nothing gets near Jev's time without starting to get the question wrong: Ministral 3B is the closest at 1.8x and answered 26 of 30; the fastest model with every answer right is Gemini 3.5 Flash-Lite at 3.0x. Jev missed one, and got no answer to one call (a 503 from the Gateway), which is not counted. Nemotron Nano 9B returned no usable object on any call through the Gateway and is left out. Not smarter, faster. POST /api/compare runs one lane from the Vercel function (admin token required, like /api/record) and scripts/compare-models.ts drives it lane by lane.

What counts as one decision

The state is a JSON object of numbers and nothing else, about 125 bytes. The engine builds the real one from a served game:

{"court":{"w":160,"h":100},"ball":{"x":80,"y":50,"vx":19,"vy":4.4},"paddle":{"y":50,"h":20},"interceptY":67.4,"dir":"toward"}

The question is DECISION_INSTRUCTIONS in lib/decide/prompt.ts: it states the court semantics and asks for the right paddle to cover interceptY, treating the paddle as already covering it when within five units. There are three answers, up, down, and stay, described by MOVE_CRITERIA in the same file. Every lane reads those two constants, so nothing is worded twice.

The number beside a lane is the round trip of that one call, timed on the server around the model call and nothing else.

How a game runs

Every participant in a game is a member of one Ably channel, pong:game:{id}, including the agent.

• The browser gets a short-lived token from /api/ably-token, enters presence as a player, subscribes to state, and publishes input. An input is a level, not an event: up, down, or stay, repeated every 150 ms while held, and the worker forgets it after 400 ms, so a closed laptop can never pin a paddle against the wall.
• The agent is a Node process started by POST /api/game and kept alive inside the same Vercel function invocation with after(), for up to the function's maxDuration. It joins the channel with the API key, enters presence as agent, runs the physics, asks the model for a move once per ball step, and publishes a state snapshot after every decision, every paddle movement, and at least once a second.
• Spectators attach to the same channel with rewind and enter presence as spectators. The viewer count is presence. A second human is one more player on the channel, which is why human versus human needs no extra code.
• A referee ends the game when a score reaches five, when a player has been gone for 15 seconds, when nobody took a side within a minute, when a demo has been unwatched for two minutes, or 15 seconds before the platform deadline. The final frame says why, and the agent leaves presence.

In a mode a human plays, a ball step never completes faster than 260 ms, even when Jev answers sooner. The published latency is still the real model latency. Demo mode is unpaced.

What the browser predicts

The court is drawn from the wire, one step behind the worker, with two exceptions that are worth reading because they are where "it feels wrong" lived.

The player's own paddle is predicted locally the moment a key goes down, at the worker's own speed, and reconciled against the wire only after a frame acknowledges the player's latest input (Snapshot.inputSeq). Reconciling earlier drags the paddle back toward a position that is a round trip old. The ball's step is timed by the worker's clock, not by when its frames arrived, and a frame that lands early continues the walk from where the ball is drawn rather than jumping. lib/ui/paddle.ts and lib/render/live.ts are both pure and both tested.

Read the code in this order

1. lib/game/types.ts: the contract. The court, the mechanic (one decision is one tick), and DecisionState, the 125 bytes a model is given.
2. lib/game/engine.ts: the whole game as pure functions. One segment of ball per decision, the paddle planes the ball turns at, and the scripted left paddle that never misses.
3. lib/decide/prompt.ts: the one question every model is asked, and its three answers.
4. lib/decide/jev.ts and lib/decide/llm.ts: the two ways the question is put, experimental_evaluate for Jev and structured output for the chat models. Both time the model call and nothing else.
5. lib/worker/game-worker.ts: the agent as a member of the channel. The decision loop that advances the ball, the continuously moving human paddle, the input acknowledgement, and what goes on the wire.
6. lib/worker/referee.ts: when a game stops, as a pure state machine with no clock of its own.
7. app/api/game/route.ts: one HTTP request, one function, one game.
8. lib/ably/hooks.ts: everything a browser does. Subscribe to snapshots, be present, publish input. No game logic.

Run it

pnpm install
cp .env.example .env.local   # then fill in the two keys
pnpm dev

Variable	Used by	Notes
AI_GATEWAY_API_KEY	/api/game (worker), /api/record, /api/decide	Vercel AI Gateway key. On Vercel, OIDC works instead.
ABLY_API_KEY	/api/ably-token, /api/game	Server only. Mints browser tokens and connects the agent. No API key reaches a browser.
MAX_LIVE_GAMES	/api/game	Optional cap on concurrent agent workers (default 20).
ADMIN_TOKEN	/arena, demo games, /api/record	Gate for anything that spends credit with nobody playing. Pass as x-admin-token or ?token=.

When the Gateway budget is exhausted the API returns out_of_credits and the lane shows it. No meter, no retries.

pnpm test        # engine determinism, the sampler and paddle prediction, referee rules, worker lifecycle, token route
pnpm typecheck
pnpm lint

To re-record the numbers, deploy and POST /api/record with the admin token, so the latencies are measured next to the Gateway rather than on your broadband. The response is the replay and its stats; drop them into public/. pnpm render:clip turns the replay into the clips in public/media: --layout wide for this page, --layout social for a 4:5 phone cut of all four lanes, and --layout duel --lanes jev,haiku,gpt for the three-lane cut below.

Deploying to Vercel needs the two keys as environment variables, vercel.json (already here, so Vercel builds it as Next.js), and a plan whose functions can run for the length of a game. Web Analytics is on: cookieless page views and four custom events (game started, game over, watch opened, share copied), listed in lib/ui/analytics.ts. Nothing a person types is sent.

Stack and credits

Next.js 16 (App Router) on Vercel. ai 7 with the Gateway provider. ably 2 with ably/react. Canvas courts, Tailwind 4, no game engine library.

Jev is by TypeSafe AI, reachable through Vercel AI Gateway. Realtime agent transport by Ably. An Ably Labs demo, Apache 2.0.