LemonSim

LemonSim is the deterministic simulation arena in the Lemon stack. It runs LLMs inside explicit game and benchmark worlds, records what happened, and produces artifacts that can be replayed, scored, and verified.

Shape

• LemonSim.Kernel owns the event-sourced core: state, events, runner flow, updater/action/projector behaviours, decision adaptation, persistence, and pubsub helpers.
• LemonSim.LLM owns model-facing execution: ToolLoopDecider, tool policies, provider/model setup, throttling, live-run helpers, and transcripts.
• LemonSim.Bench owns benchmark artifacts: manifests, scorecard behaviours, registry-driven scorecard recompute verification, hash verification, and shared run-bundle helpers.
• LemonSim.Examples.* owns the worlds. The examples are 92.3% of the current Elixir source under apps/lemon_sim/lib, because the interesting work is in scenario rules, tools, projections, and scoring.

ToolLoopDecider is the fairness boundary for heterogeneous models. A scenario offers a constrained tool/action surface, the model chooses through that surface, and the updater turns accepted actions into authoritative events.

Scenarios

There are 19 examples in this app:

• TicTacToe is the smallest self-play scenario.
• Werewolf is the social-deduction showcase with hidden roles, transcripts, and video replay tooling. It is one of five scenarios in the persistent multi-game league system (LemonSim.Bench.League + per-scenario adapters, alongside SpaceStation, StockMarket, Survivor, and Poker): randomized model-to-seat match planning, per-game records, and per-model/per-role standings with Bradley-Terry ratings. The lemon_sim_ui arenas run them around the clock.
• VendingBench is the operator/physical-worker vending business benchmark, with deterministic CI, paper, V2, offline Arena, and live multi-agent Arena modes.
• TcgShop is a single-operator local game store benchmark with inventory, suppliers, customers, events, online orders, accounting, and scorecards.
• Poker is a multi-hand no-limit hold'em session with per-seat model assignment; it is the fifth league/arena scenario (ranked mode on final chip stacks).
• Diplomacy, Skirmish, Survivor, Pandemic, Auction, Courtroom, SpaceStation, StockMarket, SupplyChain, DungeonCrawl, MurderMystery, Legislature, IntelNetwork, and StartupIncubator cover other state machines and decision surfaces.

Run

Small local smoke run:

mix lemon.sim.tic_tac_toe --offline-strategy random --seed 42 --no-persist --max-turns 10

Deterministic benchmark run with artifacts, also keyless:

mix lemon.sim.vending_bench --preset ci --offline-strategy baseline --sim-id vb_ci_baseline
mix lemon.sim.verify apps/lemon_sim/priv/game_logs/vending_bench/vb_ci_baseline
mix lemon.sim.score apps/lemon_sim/priv/game_logs/vending_bench/vb_ci_baseline

TCG Shop deterministic lanes:

mix lemon.sim.tcg_shop --preset ci --offline-strategy baseline --sim-id tcg_ci_baseline
mix lemon.sim.tcg_shop --preset ci --offline-strategy pressure --sim-id tcg_ci_pressure
mix lemon.sim.tcg_shop --preset stress --offline-strategy overextended --sim-id tcg_stress_bad_operator

Live model runs require configured providers and credentials:

mix lemon.sim.tic_tac_toe --no-persist --max-turns 10
mix lemon.sim.werewolf --player-count 6 --no-persist --max-turns 50
mix lemon.sim.werewolf --player-count 5 --models anthropic:claude-sonnet-4-20250514,openai:gpt-4.1,google_gemini_cli:gemini-2.5-flash,openai-codex:gpt-5.1-codex-mini,kimi:k2p5
mix run apps/lemon_sim/priv/scripts/werewolf_5model.exs

A single mix lemon.sim.replay <scenario> <log> task renders JSONL transcripts for every scenario. For example, a Werewolf transcript produced by mix lemon.sim.werewolf --models ... --transcript-path path.jsonl:

mix lemon.sim.replay werewolf apps/lemon_sim/priv/game_logs/werewolf_4model.jsonl
mix lemon.sim.replay skirmish priv/game_logs/abc123.jsonl --output replay.mp4 --fps 4

A single positional argument is treated as a skirmish log for backwards compatibility. Run mix lemon.sim.replay --help for the full scenario list. The replay task requires rsvg-convert and ffmpeg on PATH. (VendingBench uses mix lemon.sim.vending_bench_replay, which builds a static HTML browser.)

Artifacts

Benchmark runs write bundles under apps/lemon_sim/priv/game_logs/... unless --artifact-dir is provided. Vending Bench and TCG Shop write rich bundles with scenario-specific replay/transcript files. Poker, Stock Market, and Pandemic write verified scorecard bundles with manifest.json, hashes.json, final_world.json, events.jsonl, actions.jsonl, scorecard.json, and usage.json.

Use:

mix lemon.sim.verify path/to/run
mix lemon.sim.score path/to/run

Determinism

The kernel (kernel/runner.ex) is a pure reducer: a run is a fold of an event sequence through apply_event, so the same seed produces a byte-identical run. This is verified, not just asserted — test/lemon_sim/determinism_test.exs runs the offline VendingBench strategy (a scripted decider, no LLM) twice with one seed and asserts both the in-memory final world state and every artifact's sha256 in hashes.json are identical. It is fast (~0.2s) and fully offline, so it belongs in the CI lane. The deterministic_artifacts?: true option pins the two knobs that are nondeterministic by design (a unique run id and the wall-clock timestamp); everything else falls out of the reducer.

verify checks the manifest, hash schema, required benchmark files, manifest integrity hashes, and hashed file contents. For scenarios registered in LemonSim.Bench.Scorecard.Registry, it also recomputes scorecard.json from final_world.json and compares canonical JSON. Registered verified scenarios: courtroom, diplomacy, intel_network, legislature, murder_mystery, pandemic, poker, space_station, startup_incubator, stock_market, supply_chain, survivor, tcg_shop, vending_bench, vending_bench_arena, and werewolf. Unregistered scenarios still get manifest/hash verification and skip scorecard recompute. score verifies first, then prints the scorecard.

Regenerate local demo bundles with:

scripts/generate_sim_demo_bundles.sh [out_dir]

The generator writes deterministic VendingBench baseline and pressure bundles, a CI-scale VendingBench Arena bundle, a small VendingBench suite, and ratings output. It runs mix lemon.sim.verify and mix lemon.sim.score over every bundle and prints a final metric summary. The default output directory is tmp/sim_demo_bundles.

Suites and Leaderboards

Suites run one scenario across competitors and seeds, verify each run bundle, aggregate the registered primary metric, and rank competitors with tokens and cost alongside score. Keyless deterministic suites use offline strategies:

mix lemon.sim.suite --scenario vending_bench --preset ci --seeds 11,22,33 --offline baseline,pressure --out /tmp/vending-suite

Competitor specs accept exactly one execution mode:

Shape	Scenario support	Cost behavior
{"id":"baseline","offline_strategy":"baseline"}	Deterministic adapters	$0.0000
{"id":"model","model":"provider:model-id"}	Live adapters	Provider pricing or unknown
{"id":"my-agent","external_cmd":"python3 my_agent.py"}	vending_bench	Unknown

Suite run adapters are currently available for vending_bench, tcg_shop, and vending_bench_arena. Other registered scorecards can be recompute-verified from artifact bundles but do not yet have suite runners.

The keyless CI smoke lane is:

scripts/ci_sim_bench.sh

It runs deterministic offline suites for VendingBench, VendingBench Arena, and TCG Shop under $RUNNER_TEMP or tmp/, verifies and scores every produced run bundle, writes cross-suite ratings, and checks VendingBench byte-reproducible artifact output with a double-run diff -r gate.

That writes /tmp/vending-suite/suite.json, one verified bundle per competitor/seed under /tmp/vending-suite/runs/, and /tmp/vending-suite/leaderboard.md. The leaderboard shape is:

# LemonSim Suite Leaderboard

Scenario: `vending_bench`
Preset: `ci`
Seeds: 3
Metric: `score_modes.v1_net_worth` (maximize)

All included runs are manifest hash and scorecard verified.

| Rank | Competitor | Mean score_modes.v1_net_worth (maximize) | Per-seed values | Tokens | Cost |
|---:|---|---:|---|---:|---:|
| 1 | baseline | ... ± ... (n=3) | 11: ..., 22: ..., 33: ... | 0 | \$0.0000 |
| 2 | my-agent | ... ± ... (n=3) | 11: ..., 22: ..., 33: ... | 0 | unknown |

Each ranking entry in suite.json includes additive stats with n, mean, sample std, min, and max for the primary metric across included seeds.

Live competitors can be added with repeatable --model MODEL_ID options when provider credentials are configured. VendingBench external agents can be added with repeatable --external-cmd COMMAND options. Re-render an existing suite without rerunning scenarios:

mix lemon.sim.leaderboard /tmp/vending-suite
mix lemon.sim.leaderboard /tmp/vending-suite --recompute

--recompute re-verifies the run bundles and rebuilds rankings. Any failed or tampered run stays visible in suite.json and the failure section of leaderboard.md, but it is excluded from rankings.

Write A Scenario

Start with LemonSim.Examples.TicTacToe; it is the smallest complete example. The standard anatomy is:

• top-level scenario module, such as apps/lemon_sim/lib/lemon_sim/examples/tic_tac_toe.ex
• action_space.ex for legal model/user actions
• updater.ex for authoritative state transitions
• events.ex for emitted facts
• optional performance.ex, game_log.ex, frame_renderer.ex, or video_generator.ex when the scenario needs scoring or replay media

Keep scenario-specific rules in LemonSim.Examples.*. Shared runner, model-loop, artifact, and verification logic belongs in Kernel, LLM, or Bench.

Spectator UI

apps/lemon_sim_ui is the LiveView spectator surface. It can watch registered simulation runs, render scenario-specific boards, and serve replay-oriented views for long or public runs.