Eval report: filter vs. no-filter

September 20, 2026 · View on GitHub

webctl 0.0.013, 2026-09-20. Scores in this report are on the 0–3 rubric-index scale used at the time; the product now reports the same judgment scaled to 0–10 (multiply by 3.33; the 1.8 cut is today's 6). The numbers come from eval runs made while the suite was being built (then stored in a SQLite file, since replaced by JSON run files under ~/webctl/evals/); regenerate current tables with webctl eval report --compare.

Summary

Jev filtering removes almost all of the noise in a web search's top-k and keeps the results an expert would want, at a cost of about half a second per query.

On the reference run (29 cases, local SearXNG aggregating Google and others, version 0.0.013):

raw top-kafter filter
results delivered439162
characters delivered76,95529,011
hand-labelled junk domains delivered40
pages the source-quality audit flags as SEO/affiliate/content-farm1571
results on the domains where the best answers live (Reddit, HN, GitHub, Stack Overflow)1414
expected themes covered55/5751/57
cases passing22/29

On the best provider (Exa's neural index, version 0.0.007, run 4) the same shape holds with more to work with: 424 raw results and 113 labelled junk became 215 results and 7 junk, with every one of 57 expected themes still covered and 23 of 24 expected-domain hits retained.

The filter's median cost is 0.44s of wall clock per query (Jev per-result scoring, 8 concurrent requests) on top of the provider's own latency. A filtered search end to end is 1.5 to 3.5 seconds.

What was evaluated

29 cases in evals/cases/, 24 of them new for this report:

groupcaseswhat makes them hard
Reddit is where the best answers livequiet keyboard switches, espresso grinders, SF-to-NYC neighborhoods, Kindle Scribeopinion queries; affiliate "best of" listicles outrank the threads
Hacker NewsSQLite in production, monolith returns, Tailscale vs WireGuardAI-written tech blogs restate the discussion
GitHubRust TUI library, pgvector, uv, LLM routeraggregator and tutorial sites around the repos
SEO-dominated topicsVPN at home, protein intake, desk back pain, closing a credit card40–70% of the top 20 are affiliate or content-farm pages
Ambiguous termspython (snake), mercury (planet), java (island), go (board game), crane (bird), jaguar (cat)the other meaning, plus the usual farms
Technical researchDPO vs RLHF, Postgres bloat, JWT refresh rotation, Kubernetes OOMKilled, mechanistic interpretability, the Transformer papermostly clean top-k; tests that the filter does not over-prune

Each case runs one search and then three stages on the same results:

  • nofilter: the raw top-k, as it would reach a context window without Jev.
  • filter: Jev scores each result 0–3 on topic and source quality; results at 1.8 or above are kept (the product default).
  • scrape (9 cases): the kept pages are fetched, split into ~2,000-character chunks, and only the chunks Jev says are worth quoting are kept.

For each stage the runner records what would be delivered (results, characters, hits on hand-labelled junk domains, hits on the domains where the best answers live) and asks Jev, in one batch request, whether the delivered text covers the case's expected themes. Two independent noise measures are used: junk_domains, hand-labelled per case from observed raw results (21 cases, 113 labelled hits on Exa), and a separate Jev audit that flags SEO, affiliate, and content-farm pages with a differently worded yes/no question. Thresholds and prompts were tuned with the audit and pass/fail; the junk labels are the ground truth the report leans on.

A run can re-judge an earlier run's provider results (--reuse-searches), so two filter versions can be compared on identical inputs. Filter-stage timing includes only Jev.

Filter vs. no-filter, per case

Reference run 13 (0.0.013, SearXNG). "flagged" is the audit; "junk" is the hand labels; "expected-domain" counts results on the case's expected_domains.

casetagsraw resultskeptraw charskept charsjunk raw→keptflagged raw→keptexpected-domain raw→keptthemes rawthemes keptfilter mspass
amb-crane-migrationambiguous,noise148343919900→06→00→02/22/2482✓
amb-go-board-gameambiguous,noise13331997441→03→00→02/22/2384✓
amb-java-islandambiguous,noise,travel15141365841→012→00→02/22/2508✓
amb-mercury-planetambiguous,noise1514365735270→01→00→02/22/2471✗
amb-python-snakeambiguous,noise156353115080→06→00→02/22/2446✓
ambiguous-jaguar15330095511→03→00→02/22/2381✓
attention-paper538004800→00→00→02/22/2255✓
github-llm-routergithub,code,noise15524248070→05→01→12/22/2514✓
github-pgvectorgithub,code,noise159224913260→01→01→12/21/2457✗
github-rust-tuigithub,code158230511810→04→02→22/22/2482✓
github-uvgithub,code15323654640→05→01→12/22/2395✓
go-generics15423886550→05→00→02/22/2420✓
hn-monolith-returnhackernews,engineering,noise14222063230→010→00→02/22/2365✗
hn-sqlite-productionhackernews,engineering15524338160→03→01→12/21/2389✗
hn-tailscale-wireguardhackernews,engineering15224283230→08→02→22/22/2474✓
mech-interp2015316423560→02→00→02/22/2546✓
noul-research-only147221311180→00→00→01/11/1417✓
reddit-espresso-grinderreddit,opinion,noise15324124880→07→01→12/22/2423✓
reddit-kindle-scribereddit,opinion,noise157237810950→03→01→12/22/2449✓
reddit-nyc-neighborhoodreddit,opinion,noise15223893200→012→01→11/20/2380✗
reddit-quiet-switchesreddit,opinion,noise15224043260→013→01→12/22/2508✓
seo-back-pain-deskseo,noise,health209322114520→08→00→02/22/2583✓
seo-close-credit-cardseo,noise,finance1612257719270→03→10→02/22/2416✓
seo-protein-intakeseo,noise,health189288014520→05→00→02/22/2517✓
seo-vpn-home-wifiseo,noise,security20132091600→011→00→01/21/2482✗
tech-dpo-vs-rlhfresearch,ml15423836320→06→00→02/21/2404✗
tech-jwt-refresh-rotationengineering,security,noise15324064720→04→00→02/22/2466✓
tech-k8s-oomkilledengineering,kubernetes15523528000→08→02→22/22/2411✓
tech-postgres-bloatengineering,databases157239811341→03→00→02/22/2409✓

What the table shows, by group:

  • Reddit and HN cases. Every expected-domain hit survives the filter (14 of 14 on this run; 23 of 24 on Exa). On the keyboard-switch query the filter dropped 13 audit-flagged listicles and kept the two community threads. The two failing Reddit/HN cases (reddit-nyc-neighborhood, hn-sqlite-production) fail on theme coverage, not on what was kept: Google-style snippets for a Reddit thread are one or two lines, and the judge sees only that text.
  • SEO cases. The filter delivers 1 audit-flagged page out of 27 flagged in the raw top-20s. seo-vpn-home-wifi keeps a single result because the honest answer to that query (a Reddit thread) was the only non-marketing page in the top 20; the case's second theme is then uncovered. The remaining pass, with 9 to 12 kept out of 16 to 20, and the kept set on the credit-card query is the bureaus, the CFPB, and the banks.
  • Ambiguous terms. Modern engines rarely return the other meaning for a phrased query, so the noise here is farms, not the other sense. The filter keeps 1 to 8 of 15 and covers every theme; amb-mercury-planet fails only because 14 NASA, ESA, and museum pages were all legitimately kept against a bound of 10 (raised to 15 after this run).
  • Research and code. Where the whole top-k is papers or repositories the filter keeps most of it (mech-interp 15 of 20, rust-tui 8 of 15); github-pgvector and tech-dpo-vs-rlhf fail on one theme each from snippet-only judging.

Scrape-to-chunk

Nine cases fetch the kept pages and keep only chunks Jev judges worth quoting. The measures are page fetch success, how much text the chunk filter removes, whether the kept text still covers the themes, and, as a recall check, whether the text that was thrown away covered any theme on its own.

runproviderpages ok / failedchunks kept / totalchars raw → keptthemes covered by kept textthemes still present in dropped text
4 (0.0.007)Exa44 / 11350 / 466780,638 → 608,24618/18not measured
13 (0.0.013)SearXNG34 / 10249 / 369627,344 → 444,38317/1810/18

The chunk filter removes 22 to 30 percent of fetched text and the kept text covers the themes as well as the whole page did. The dropped text still "covers" themes in 10 of 18 judgments because long pages repeat themselves; that is redundancy removed, not signal lost. Chunk probabilities are well separated on a Reddit thread (navigation 0.08, weak comment 0.44, substantive comments 0.81 to 0.94), so the 0.5 cut is not a sensitive knob.

Fetch failures are bot walls and PDFs: DOI resolvers and publishers answer 403, and academic PDFs are not parsed. Since 0.0.003 a page that cannot be fetched falls back to the provider's own excerpt, so those results still contribute text (the seo-protein-intake scrape stage passed with 13 of 16 fetches failing). Reddit specifically is handled: the JavaScript challenge is solved and the comment tree, which ships inside a <template> element, is read.

Scrape vs. no scrape

Two runs on 2026-09-20 with the scrape stage forced on all 29 cases, comparing what the filter stage delivered (title, URL, and the provider's snippet or excerpt) against the scraped, chunk-filtered pages of the same kept results. Jev judged theme coverage on both.

providerthemes covered: filter → scrapechars to context: filter → scrape (raw pages)Jev input tokens: filter → scrapeavg stage timepages fetched / failed
Parallel (2.8K-char excerpts per result)55/57 → 51/57 (better on 0 cases, worse on 4)86,796 → 1,303,936 (1,971,553)572K → 1,454K0.46s → 1.9s124 / 31 (20%)
SearXNG (170-char Google snippets)51/57 → 54/57 (better on 4, worse on 2)31,068 → 1,262,669 (2,068,892)463K → 1,533K0.54s → 1.6s140 / 22 (14%)

Reading: when the provider already returns excerpts, scraping adds nothing Jev can detect and costs about 15× the context, 2.5× the Jev tokens, and 4× the time. When the provider returns snippets only, scraping recovers coverage the snippets could not show (reddit-nyc-neighborhood went from 0 to 2 themes). The chunk filter removes 34 to 39 percent of fetched text. Fetch failures are bot walls (arXiv, Britannica, Stack Overflow, Forbes, IEEE, NASA answer 403), PDFs, and pages with no readable text (Facebook); a failed fetch falls back to the provider excerpt. Caveat: the theme judge reads at most 2,500 characters per result in both stages, so a long page whose relevant passage sits past that cap can judge worse than its own snippet.

Speed

stageavg ms per case (run 13)avg ms per case (run 4, Exa)
provider search1,529 (SearXNG, cold)611
audit + no-filter judge146164
filter (Jev scoring + judge)443600
scrape + chunk filter + judge (9 cases)1,4581,512
whole case, wall clock2,569 avg, 5,051 max1,844 avg, 4,437 max

The full 29-case suite runs in 40 seconds at two cases in parallel. A filtered search from the CLI is 1.5 to 3.5 seconds; adding --scrape --filter-chunks for 7 pages adds about 1.5 seconds because fetches and chunk requests run concurrently.

Jev cost: filtering 439 results took 456K input tokens in per-result mode (one request per result, each carrying the rubric). Batch mode (all results in one request) cut that by about 18 percent per case on a 15-case comparison with a comparable pass rate, but is not the default: the eval judge already hit Jev's request-size limit at 20 results, so batches would need splitting.

Changes made during this work, with evidence

Every change bumped internal/version.Version; each run below is a full 29-case pass unless noted.

versionchangeeffect
0.0.002Keyless Exa and Parallel via their hosted MCP servers; budgeted fallback chain (12s per attempt, 30s total)Searches work with no keys after DuckDuckGo started refusing this IP
0.0.003Scrape falls back to the provider's excerpt when a page cannot be fetchedWalled and PDF results keep contributing text
0.0.004Rubric v2: score topic and source quality together; default threshold 1.0 → 2.0On Exa, with the eval still cutting at 1.0, the new rubric kept 400 of 450 and delivered 91 audit-flagged pages; at the product's 2.0 cut (run 2) it kept 227 of 445 and delivered 1
0.0.006Rubric v3: a community thread on an opinion or experience question is a best-available sourceReddit threads moved from 1.7–1.96 (just under the cut) to 2.3–2.6; reddit-nyc-neighborhood went from dropping all six threads to keeping four; 24/29 pass
0.0.007Collapse results with the same URL or titleattention-paper: ten copies of the same paper became three results
0.0.008Snippets start at the first prose line, not page chromeExa's degraded keyword mode had been feeding "Skip to main content… Subscribe…" to the judge
0.0.009Chain tops up a short answer (under half the requested count) from the next provider, fused by reciprocal rankThrottled tiers answer with 10 truncated results instead of an error
0.0.010Rubric v4: level 1 is reserved for SEO, affiliate, farm, listing, and pitch pages; level 2 names reference works and established publicationsBritannica, LiveScience, agency pages had been scoring 1.8–1.97 and being cut on general-reference queries
0.0.011You.com keyless backend; a provider answering 429 or 402 is skipped for 90s per process; 3s dial timeoutA dead DuckDuckGo had cost 12s per search; a throttled tier now costs one request per run
0.0.013Default threshold 2.0 → 1.8Between 1.75 and 2.0 sat Wikipedia, museums, zoos, Reddit threads, Auth0 docs and no labelled junk; 151 kept became 184 on the same inputs with junk still 0

Threshold sweep on the v3-rubric Exa inputs (run 3), count bounds and junk labels only:

thresholdcount-bound passeskeptjunk kept / labelledexpected-domain kept / present
1.924/292299/11323/24
2.025/292157/11323/24
2.226/291884/11323/24
2.426/291742/11322/24

The same sweep on v4-rubric You.com inputs (run 10) kept junk at 0/16 from 1.7 through 2.0 while kept results went from 199 to 151, which is what moved the default to 1.8.

Caveats

  • The provider decides what the filter has to work with. Exa's neural index returns papers and repositories for research queries and long excerpts (2–8K characters per result); SearXNG and You.com return keyword results with 150–500 character snippets. Jev scores on title, URL, and snippet, and the theme judge reads the same text, so short snippets both lower scores and undercount coverage. The scrape stage is the remedy when it matters.
  • Free tiers throttle. Exa's and Parallel's keyless endpoints rate-limited after a few dozen searches and stayed limited for over an hour, and You.com's free profile answered 402 after about 70; DuckDuckGo refused TCP connections from this address all day. Runs 8, 9, and 11 are partial or single-provider for that reason and are not in the headline. The local SearXNG documented in the README has no such limit and is what the reference run used.
  • Results drift between runs. Engines return different pages for the same query hours apart, so per-case pass/fail moves by one or two cases run to run. --reuse-searches makes any two runs of the same inputs comparable; the version history above is not one input set.
  • Junk labels are judgment calls. Four hosts first labelled junk on the SQLite case (raxxo.shop, ultrathink.art, prodsens.live, 0x.run) were removed after reading their snippets: first-hand engineering posts with specific numbers, whatever the domain looks like. The remaining borderline survivors (a realtor's guide with actual price data, a keyboard guide claiming six months of hands-on use) are counted against the filter.
  • Case bounds were corrected during the work where a whole top-k was legitimately top-tier (12 meta-analyses for "what the studies say"). Those corrections are in the commit history; none loosened a junk or expected-domain check.

Reproduce

go build -o webctl ./cmd/webctl
docker run -d --name searxng -p 8899:8080 -v "$PWD/docs/searxng/settings.yml:/etc/searxng/settings.yml:ro" searxng/searxng:latest
SEARXNG_URL=http://localhost:8899 WEBCTL_PROVIDER=searxng ./webctl eval
./webctl eval report --compare          # tables per version, latest run each
./webctl eval --verbose reddit-quiet-switches   # every judged score, keep/drop