Jev Paper Judge

Upload a paper as a PDF and get feedback in seconds: a structured judgment of two things.

1. Understandability. Can a reader follow it on a first pass? Sentence clarity, defined terms and symbols, signposting, logical flow.
2. Completeness. Does it give readers everything they need? Problem, motivation, contributions, background, reproducible method, experimental setup, interpreted results, limitations, plus mechanical checks such as undefined acronyms and unreferenced figures.

It measures presentation, not correctness. A wrong paper can still be clear and complete.

Why a snap judgment

Whether a paper is easy to understand is not something a reader reasons out. It is a reaction. Within a paragraph, someone in the field, or a bit outside it, already knows whether the terms are explained, whether the sentences land, whether the point comes through, and whether they could keep reading without looking things up. Deciding that takes no chain of thought. It takes a calibrated gut feeling from a reader who has seen many papers.

That is what a System One model is for. Jev does not write an essay about the paper. It answers small, literal questions ("are the symbols in this passage defined when they first appear?") with a probability, in one pass, in well under a second. Three things follow:

• Speed. Every passage is judged in parallel, so a 16-page paper comes back in two to three seconds. The loop is fast enough to edit a paragraph, re-upload, and watch the score move.
• Composability. A probability can be weighted, thresholded, and compared across passages. The scores are arithmetic over the answers, and the arithmetic is visible in src/rubric.ts.
• Honesty about doubt. A calibrated model can say it is unsure. Answers with low confidence are marked in the report instead of being smoothed over.

A generative model asked to "review this paper" would be slower, more expensive, less consistent between runs, and would spend its effort on the one thing this tool deliberately leaves alone: whether the work is right.

How it works

The browser does the extraction; the Worker does the judging. One upload is one request to the Worker.

browser                              Cloudflare Worker                          TypeSafe
───────────────────────────────────  ─────────────────────────────────────────  ─────────────
PDF ─pdf.js─▶ text ─▶ sections ─▶
  passages ──── POST /api/judge ──▶  validate, rate-limit
                                     ├─ paper-level state + 13 questions ────▶  POST /v1/systemone
                                     ├─ passage 1 + 13 questions + 1 per term ▶
                                     │  passage 1 as paragraphs + 2 per ¶ ────▶
                                     ├─ passage 2 …          (in parallel)      ◀── probabilities
                                     └─ checks, confidence routing, profiles, calibration
report ◀────────────────────────────  JSON

The rubric, the fan-out, and the API key all live on the server. The client never sees which questions are asked, and the endpoint cannot be used as a generic proxy: it accepts a document, not questions.

Jev is a "System One" model: it answers typed questions (yes/no, choice, ordered score) with calibrated probabilities instead of generating text. That makes it a good fit for a rubric, and the docs are explicit about what it is bad at (counting, arithmetic, large unrelated context, non-literal phrasing). The rubric follows those constraints:

• One judgment per question. Score levels describe situations ("symbols are used without saying what they stand for"), never degrees ("poor"/"good").
• Passages, not the whole paper. The browser splits the body at headings into passages of roughly a thousand words. The server judges each on its own with the same fixed question set, and the paper-level state (title, abstract, introduction, conclusion, section outline) gets its own set.
• Terms are checked one at a time. Code finds the technical terms a paper uses (acronyms, capitalized names, coined hyphenated compounds), works out in which passage each first appears, and adds one question per term to that passage: does this passage say what "X" means? The answers become the "terms explained at first use" dimension and a checklist item that names the unexplained terms. A single question ("are terms explained?") was tried first and turned out to measure citation density instead.
• Feedback points at paragraphs. A second, parallel request per passage carries the same text split into keyed paragraphs (paragraphs.p1, p2, …; the model cannot count, so keys replace "the third paragraph") and asks, per paragraph, whether it opens with its point and whether it says how it follows from the one before, plus which paragraph is hardest. Many questions in one request add no latency (the speculative fan-out pattern); the paragraphs get their own request because sending them alongside the passage shifted the passage-level answers.
• Findings are routed by confidence. Scores use every answer, because averaging many answers cancels noise. A finding that tells the author to change a specific term or paragraph is higher-stakes, so it is shown under "Fix now" only when the model's confidence is at least 0.6, under "Worth checking" between 0.3 and 0.6, and otherwise kept in the raw answers. Counted facts (long sentences, unreferenced figures) always go to "Fix now". When more than 40% of score-bearing answers are unsure, the whole report is marked provisional.
• One judgment, three reader profiles. The atomic answers are combined under three weightings (readers outside the field, readers in the field, specialists), each with its own calibration table; the page lets you switch after judging. Profiles differ in weights, in which kinds of term must be explained, and in checklist emphasis; see PROFILES in src/rubric.ts.
• Counting stays in code. Acronym definitions, figure references, whether a figure mention says anything beyond "see Figure 3", sentence and paragraph lengths, citation and symbol density, and section coverage are deterministic checks in src/checks.ts.
• Equations and tables are tolerated, then flagged. PDF extraction leaves display equations, table rows, and the labels inside figures in the text. The passage-level questions turn out to read around them (filtering them changed nothing on a math-heavy paper), so the passage is sent as extracted; but terms and paragraphs are taken from the prose only, so a figure label is never reported as an unexplained term. A passage that is less than 70% prose is marked equation-heavy, and its "hard to follow" finding is demoted to "worth checking", because the judge never sees the mathematics the argument lives in.
• Weights are visible. Understandability is a word-weighted mean of per-passage clarity dimensions; completeness is a weighted checklist. Both live in src/rubric.ts and are easy to tune.
• Scores are relative to accepted papers. The raw means cluster tightly, so each headline score is reported as a percentile among 340 accepted NeurIPS 2024 and ICML 2024 papers and mapped onto a display scale where the typical accepted paper is 65 (see Calibration).
• Uncertainty is shown. Answers where the model's own confidence was low are marked with ? in the report, following the confidence-gating pattern from the docs.
• Appendices are supporting material. Understandability is scored on the main body only. Appendix passages (at most four, prose only) can raise a completeness item, for example when the experimental setup lives there, but never lower one. Figure-reference and acronym checks score the main text; the same checks over the appendix are reported for information.

A 16-page paper costs about 19 upstream requests and 45k input tokens, roughly $0.002 at the current Jev price, and finishes in two to three seconds.

What the judge can and cannot see

Every scored question was tested by degrading real passages in the way that question is meant to catch, then re-judging (60 NeurIPS 2024 and ICML 2024 papers, 649 passages, 30 passages per manipulation, with unchanged and randomly-shortened passages as controls). Jev is deterministic, so an unchanged passage scores identically; a real defect has to move the score to count.

Question	Manipulation	Drop (0–100)	Kept?
Sentence clarity	join every pair of sentences with ", and"	−22, 30 of 30 passages	yes
Logical flow	shuffle the paragraphs / the sentences	−48 / −30	yes
Purpose signposted	delete the opening sentence	−17	yes
Topic sentences	move each paragraph's first sentence to its end	−16	yes
Terms explained (per term)	delete the acronym's spelled-out expansion	−34 on that term, −1 on the others	yes
Concrete examples	delete the example sentences	−18	yes
Abstract self-contained	keep only the first sentence	−80	yes
Title descriptive	keep only the acronym before the colon	−59	yes
Conclusion answers / limitations / future work	cut the conclusion to two sentences	−30 / −48 / −61	yes
Terms defined (one question per passage)	delete the definitions	−3	dropped: tracked citation density (ρ 0.54), not explanations
Design rationale	delete the "because" sentences	−3	dropped
Figures explained	replace mentions with "Figure 3 shows the results"	−6	moved to code
Notation defined	delete the "where x denotes" sentences	−6	dropped: extracted math is too noisy to judge
Paragraph opens with its point (per paragraph)	move one paragraph's first sentence to its end	−28 on that paragraph, −1 on the others; but only −3 on the mean when every paragraph is buried	used for findings only, on a clear no: the answer is relative to the neighbouring paragraphs, so it locates an odd one out rather than measuring an absolute property; 32% of paragraphs in accepted papers "fail" it
Paragraph connects to the previous one	shuffle the paragraphs	−20 per paragraph	used only to say where a weak passage's flow breaks; 46% of paragraphs in accepted papers "fail" it
Hardest paragraph (choice)	merge the sentences of one paragraph	picks it 37% of the time (20% before the change)	mentioned as the model's pointer only when its confidence is at least 0.75
Can a reader in the field follow this	fake acronyms / merged sentences / shuffled paragraphs	−18 / −2 / −5	dropped: reacts only to jargon, which the per-term check already covers

The per-paragraph questions were kept for localization rather than scoring: they respond to a real defect in the right paragraph and nowhere else, but they judge each paragraph against its neighbours (burying every paragraph's point barely moves them, while the passage-level question still drops) and they fail a large share of paragraphs in accepted papers, so a lone "no" is not evidence of a problem. The routing rules above turn them into findings only when the answer is a clear, confident no, and for links only inside a passage the passage-level flow question already marked as weak.

Two things the judge does not see, by design. It cannot tell whether the claims are true, only whether they are presented so a reader can follow them. And it does not track acceptance tier: on 340 NeurIPS and ICML 2024 papers, oral, spotlight, and poster papers scored the same on understandability (51.2, 50.8, 51.2), which is what one would expect from a presentation-only judgment of papers that all passed the same bar.

Calibration

Raw scores are means of probabilities and barely spread: on the reference papers, raw understandability runs from 46 to 65 with a standard deviation of about 4. The number is reproducible (re-judging the same paper moves it by at most a point), but "56" tells an author nothing. So the report places every score among accepted papers instead:

1. scripts/bench/run.mjs judges a directory of PDFs with the production pipeline and keeps each full report.
2. scripts/bench/reference.mjs turns those reports into src/reference.ts: percentile tables, per reader profile, for the two headline scores, each clarity dimension (paper-level and passage-level), and each checklist item.
3. At judging time, src/calibrate.ts looks up the raw score's percentile and maps it through the anchors in CALIBRATION (src/rubric.ts): percentile 5 → 40, 25 → 55, 50 → 65, 75 → 75, 95 → 88. Below the worst reference paper the display keeps falling one point per raw point.

On that scale the verdict bands mean something concrete: "Clear" (70 and up) is the top 40 percent of accepted papers, "Hard to follow" (below 50) is the bottom 15 percent. Passages are flagged as hard to follow when their clarity is below the 20th percentile of reference passages, and each passage's weakest dimension is the one furthest below its peers, not the lowest raw number. The page draws the comparison instead of describing it: each score card shows the reference distribution with the paper's position marked, "Where it stands" shows every dimension's p10–p90 span, median, and the paper's dot, and the passages section opens with each passage placed by percentile. The quantile tables come with the report (reference.tables), so the JSON download is self-contained.

The reference set was sampled with a fixed seed from the conference programmes (60 papers per decision tier, PDFs from the open proceedings sites); acceptance tier did not affect the scores (raw understandability 57.2 / 56.7 / 57.1 for oral / spotlight / poster), so the set is used as one pool. Paper length does not affect the raw score either (Spearman −0.05 against log word count), so no length adjustment is applied. Rebuild it after any change to the rubric, since the tables describe the rubric that produced them:

node scripts/bench/run.mjs path/to/pdfs bench/reports     # about \$0.002 and 2 s per paper
node scripts/bench/reference.mjs bench/reports --label "accepted NeurIPS 2024 and ICML 2024 papers"

Layout

public/            static frontend, served as-is (no build step)
  index.html       page shell, SEO tags (absolute URLs are filled in by the Worker)
  app.js           file input → extraction → POST /api/judge → report DOM
  charts.js        ECharts views: score distributions, dimension ranges, passage overview
  pdf-text.js      pdf.js items → lines → sections → passages → document (heuristic, LaTeX-tuned)
  og.png, favicon.*, icon-*.png, apple-touch-icon.png, site.webmanifest, robots.txt
  vendor/          pdf.js and ECharts builds, copied from node_modules on install (gitignored)
brand/             icon.svg and og.html, the sources for the images above
src/
  index.ts         the Worker: /api/judge, the landing page, static assets
  evaluate.ts      request validation, term assignment, fan-out to Jev, report assembly
  rubric.ts        every question asked, weights, checklist, calibration anchors   ← tune here
  checks.ts        deterministic checks (terms, acronyms, figures, sentences, density)
  calibrate.ts     raw score → percentile among reference papers → display score
  reference.ts     percentile tables, generated by scripts/bench/reference.mjs
scripts/judge.mjs  run the same pipeline from the terminal
scripts/bench/     run.mjs judges a directory of PDFs; reference.mjs rebuilds src/reference.ts
scripts/lib/       TypeSafe client shared by the scripts
scripts/render-brand.mjs  re-render icons and the OG image from brand/ (`pnpm brand`)

API

POST /api/judge takes the document the browser extracted and returns the report:

{
  "title": "…",
  "abstract": "…",            // or null
  "introduction": "…",        // full text of the introduction section, or null
  "conclusion": "…",          // or null
  "numPages": 16, "words": 7860,
  "referencesFound": true, "outlineUsable": true, "skipped": 0,
  "sections": [{ "heading": "1 Introduction", "kind": "body", "words": 540 }, …],
  "passages": [{ "id": "s1", "heading": "1 Introduction", "kind": "body", "text": "…", "words": 540, "pageStart": 1, "pageEnd": 2 }, …]
}

An optional "profile": "general" | "field" | "expert" chooses which reader profile the top-level scores use (default field); every profile is returned under profiles regardless.

Limits (LIMITS in src/evaluate.ts): 32 passages of up to 12,000 characters each; body passages are kept before appendix ones. The rate limit is 12 papers per minute per IP.

Setup

pnpm install                       # also copies pdf.js into public/vendor
cp .dev.vars.example .dev.vars     # then paste your TypeSafe API key
pnpm dev                           # http://localhost:8787

Get a key at https://console.typesafe.ai/settings/keys.

Deploy

pnpm exec wrangler secret put TYPESAFE_API_KEY
pnpm deploy

wrangler.jsonc also declares the rate limit and a TYPESAFE_MODEL var that selects which Jev release answers. Set it to a versioned ID such as jev-1.13.0 if you tune thresholds and want them to stay put when jev-latest moves.

Branding and SEO

index.html carries the title, description, canonical link, Open Graph and Twitter tags, a web manifest, and JSON-LD structured data. Their absolute URLs are written as __ORIGIN__ and the Worker replaces that with the origin the page is served from, so the same build works on *.workers.dev, a custom domain, and localhost.

The icons and the 1200×630 Open Graph image are committed. To change them, edit brand/icon.svg or brand/og.html and run:

pnpm brand      # renders with your installed Google Chrome via playwright-core

Judge from the terminal

Useful when tuning the rubric: it runs the identical extraction and judging code (Node runs the .ts files directly) and prints the full report.

pnpm judge paper.pdf                       # text report
pnpm judge paper.pdf --json report.json    # plus the full report as JSON

Options: --model jev-1.13.0, --max-chunks 28, --concurrency 6.

Tuning the rubric

Everything a reviewer would want to argue about is in src/rubric.ts:

• PAPER_QUESTIONS and SECTION_QUESTIONS: the questions. Keep them literal; Jev answers what is written, not what is meant.
• CLARITY_DIMENSIONS: which passage answers feed understandability, and their weights. terms_explained is not a question but the mean of the per-term answers; TERMS caps how many terms one passage is asked about.
• CHECKLIST: completeness items, their weights, which paper kinds they apply to, and where their evidence comes from (paper-level answer, passages with a given role, or a code check).
• THRESHOLDS and VERDICTS: cut-offs for present/partial/missing, the uncertainty band, and the headline labels.
• PROFILES: the reader profiles, each a set of clarity weights, term kinds, and checklist multipliers.
• THRESHOLDS.confident / borderline and the paragraph-finding rules: what it takes for a finding to be shown as "fix now" or "worth checking".
• CALIBRATION: the percentile → display-score anchors. Changing questions or weights also means rebuilding src/reference.ts (see Calibration).

After changing anything, run pnpm judge on a couple of papers you know well and check the passages table: the per-dimension numbers show exactly which question moved. Before adding a question to the score, degrade a few passages in the way it should catch and confirm that its answer drops; the table above is the bar.

Limits

• PDFs need a text layer. Scanned papers are not OCR'd.
• Heading detection is heuristic and tuned for LaTeX-style papers. When it finds fewer than two headings it falls back to page-sized passages, and the paper-level questions use the first and last passages as introduction and conclusion. Short sections are merged into a neighbour to reach a judgeable size, but never forward across a top-level section boundary: a small "3.4" folds back into 3.3 rather than into "4 Experiments".
• Tables and equations come out of the PDF as noisy text. They are left in, because removing them reliably is harder than tolerating them.
• The Worker keeps nothing. Reports exist only in the browser tab; use "Download report JSON" to keep one.