JevArena

JevArena is an open-source arena for testing Jev against another judge. Ask a question, choose possible answers and an opponent, then vote before the models, speed, and cost are revealed.

Open JevArena · Try examples · Browse community questions · Submit a question

How it works

1. Ask — enter a question and its possible answers, or start from an example.
2. Choose — connect OpenRouter and select any supported model to face Jev.
3. Judge — compare the two hidden decisions, then vote to reveal identity, latency, cost, and available confidence data.
4. Contribute — opt out before submitting if you do not want your question, result, and vote sent for community screening.

Every live battle includes Jev. A vote records preference, not verified correctness. Community questions are AI-screened before publication and remain unverified until independently reviewed.

Keys and privacy

JevArena does not save your model API key. The key stays in the current tab's memory and disappears on refresh. OpenRouter calls go directly from the browser to OpenRouter. Use a dedicated key with a small provider-side limit, or run locally.

Questions and answers are sent to the selected providers only when you start a run. Public contribution is a separate, visible choice. Do not submit secrets, private conversations, or sensitive personal information. Read the Privacy Policy and Security Policy.

Run the website

cd web
npm ci
npm run dev

npm run typecheck
npm test
npm run build
CI=true npm run test:e2e

Public live comparisons use OpenRouter. Vercel AI Gateway is available for local use and remains disabled on the hosted site pending its production relay gate. Browser tests use synthetic responses and never spend model credits.

Run JevJudge-Bench

The Python package is the reproducible research harness behind the project. It transforms pinned JudgeBench, RM-Bench, and RewardBench 2 data into one diagnostic pairwise protocol; its scores are not the upstream leaderboards.

uv sync
uv run jevjudge fetch
uv run jevjudge prepare --groups-per-domain 2 --partition test --out data/pilot.jsonl
uv run jevjudge run --data data/pilot.jsonl --config configs/mock.json --out runs/smoke
uv run jevjudge report runs/smoke

Mock output is marked MOCK / PIPELINE TEST. Read PROTOCOL.md before making model claims and RESEARCH_PLAN.md before paid runs. Actual runs require an explicit call and cost limit; the runner has no hidden retries.

Contribute

• Submit a community question
• Report a reproducible case
• Report a bug
• Join Discussions

See CONTRIBUTING.md for tests, evidence states, and licensing. Code is Apache-2.0. Original case contributions use CC BY 4.0; third-party material keeps its original license and attribution.

Maintained by Richard Tang. JevArena is independent and is not affiliated with Arena, TypeSafe, or the model providers.

Maintainer notes

Provider behavior, release gates, data handling, and canary evidence live in docs/ instead of the product introduction. The project-scoped Supabase MCP in .mcp.json is read-only; production changes remain migration-led.