jev-headline-bench

Can Jev pick the winner of a real headline A/B test? Built on jev-go.

Yes. 64.5% on 10,984 randomized experiments, rising to 74.7% when the two headlines genuinely performed differently.

The label here is not anyone's opinion about good writing. Upworthy ran these tests on its own readers: two headlines for the same article, the same image, a randomly split audience, clicks counted. The archive is 32,487 of them.

Jev sees the two headlines and nothing else. No article, no image, no date, no click counts.

Run on 2026-09-16 against jev-1.13.0. 21,968 requests, about 8 minutes.

The result

	exploratory	confirmatory
pairs	2,295	10,984
accuracy	63.9%	64.5% [63.6, 65.4]

The archive ships with an exploratory and a confirmatory split. Every design decision was made on the exploratory set; the confirmatory set was run once, with nothing changed. It replicated within 0.6 points.

Accuracy tracks how real the difference was

This is the control that makes the result mean something.

true difference between the headlines	accuracy	median lift
none (z < 1)	53.1% [49.6, 56.6]	+13%
weak	57.4%	+45%
clear	65.3%	+103%
decisive (z ≥ 4)	74.7% [72.8, 76.4]	+192%

The top row is a placebo that comes free with the data. Those pairs performed the same, so there is nothing to predict, and accuracy sits at chance. If the model had scored well there too, it would be picking up an artefact of how the pairs were built rather than anything about clicks. It climbs monotonically from chance to 75% as the real gap widens.

It beats every rule of thumb

Editors have opinions about headlines. On 10,984 experiments, most of them are wrong.

rule	how often the headline following it won	n
Jev	64.5%	10,984
has a number	58.5%	1,969
longer headline	54.2%	10,615
has a quotation	51.9%	3,891
says "you" or "your"	50.3%	3,033
HAS A SHOUTED WORD	41.6%	860
has a question mark	35.5%	2,624

Headlines with a question mark lost roughly two times out of three. Writing "Want A Garden Full Of Butterflies? Go Native." instead of "She Made One Simple Change And Suddenly, Her Garden Was Full Of Butterflies" cost real clicks, 1,692 times over. Addressing the reader as "you" was worth nothing at all.

Its confidence means something

how sure it was	how often it was right	n
barely leaning (0.50-0.60)	51.7%	2,932
0.60-0.70	60.8%	2,785
0.70-0.80	67.9%	2,391
confident (0.80+)	78.5%	2,876

When it says it does not know, it does not know. That is the part a number gives you and a paragraph does not: you can throw away the bottom band and keep an edge worth having.

Controls

Position bias, measured and removed. Jev prefers whichever headline is shown second: across the sample it gives the first slot a probability of 0.428, not 0.500. Every pair is therefore scored twice, once in each order, and the two are averaged. TestCombineCancelsPositionBias proves the arithmetic: a model that always picked slot B scores exactly chance after averaging, not a win.

Anyone running this without the swap would report a number contaminated by a 7-point positional preference.

The placebo stratum, above: at chance where there is nothing to predict.

Image held constant. A test only qualifies if every variant in it used the same eyecatcher_id. Otherwise readers were choosing between pictures as well as words. That filter cuts 32,487 tests down to 13,279 clean headline-only ones.

Contamination. Upworthy's headlines were public, but the click counts were not until the archive was released, and memorising 150,000 of them to reconstruct which of two variants won is not a plausible mechanism. The exploratory and confirmatory splits were released at the same time and score the same, which is the check available here.

What it got right, and wrong

Confidently right:

won  (3.14% CTR)  A 50 Year Old Ad For Temps That's As Unbelievably Sexist As It Is
                  Wildly Revealing About Today
lost (0.04% CTR)  This Is Actually Real: An Ad For Temp Workers From The 60's
                  Jev gave the winner 0.93. Real lift: +8838%.

Confidently wrong:

won  (1.77% CTR)  It's Not A Sexy Issue, But It Is A Ridiculously Important One That
                  You Should Know More About
lost (0.09% CTR)  Strawberries Are Sexy. Farmworker Rights Are Not. But Both Are
                  Pretty Ridiculously Important
                  Jev gave the winner 0.07. Real lift: +1941%.

The second pair is the honest reminder. Both headlines say nearly the same thing in nearly the same words, one got twenty times the clicks, and the model backed the wrong one hard.

Reproducing

export TYPESAFE_API_KEY=...
uv run --with pandas --with numpy python data/prep_upworthy.py exploratory
uv run --with pandas --with numpy python data/prep_upworthy.py confirmatory

go run ./cmd/jev-headlines -run -split exploratory    # design here
go run ./cmd/jev-headlines -run -split confirmatory   # then this, once
go test ./...

cmd/jev-headlines       runner and report
internal/headlines      pairs, battery, both orders, scoring
data/prep_upworthy.py   pair construction, the file to argue with
results/                per-pair output for both splits

Source data: the Upworthy Research Archive, CC BY 4.0, via Matias, Munger et al. Not redistributed here; prep_upworthy.py pulls it.

Caveats

Upworthy in 2014-15 is one publisher, one audience, one era of internet writing. The question-mark result is about these readers seeing these headlines, not a law of language.

Pairs are the best and worst variant inside each test, chosen to maximise the real difference. That makes the task easier than picking between two arbitrary headlines, and it means the z statistic on a selected pair overstates its own significance. Nothing here quotes that p-value: z is used only to sort pairs into strata, and the strata behave exactly as they should.

Accuracy is measured against which headline actually got more clicks, so a pair where both were equally good is unpredictable by construction. That is why the placebo row exists rather than being hidden inside the average.

Single model version, single run.

License

MIT