webctl
September 20, 2026 · View on GitHub
Smart web search CLI for agents, backed by Jev. Saves a lot of tokens.
webctl search "final score san francisco giants september 19th baseball" \
...or, more likely:
webctl "San Francisco giants MLB score recent games" \
--goal "The user asked to know the final score of last night's (September 19th) San Francisco Giants game in Major League Baseball"
I originally built this to give my Pi/Kimi K3 chat stack something akin to what Claude/Codex already have out of the box, but now I have Claude/Codex using it too... and it's pretty freaking dope. It really saves a lot of the context window. So now it's free for all!
How it works:
By default, webctl will try 3 web search backends. Each result set is passed (with the original query and goal) to Jev for scoring; the high-scoring subset is then deduped by some fancy math (plus Jev). This means your Claude (or whatever) doesn't have to read as much junk, which saves you $$ (sorry, Anthropic!).
Agents should use --scrape --filter-chunks instead of fetching pages themselves most of the time. webctl fetches each kept page, parses out the textual content, divides it into chunks, and sends batches of those chunks (plus the original query and goal) to Jev for scoring. Only the relevant chunks come back, so that is all that lands in the agent's context. For some workloads (think long PDFs, Reddit/StackOverflow comment threads, developer docs, entire Wikipedia articles, etc), this can save an enormous number of chat tokens compared to reading the page.
Feel free to submit a PR if I missed something! And if I miss the PR, hit me up @dorkitude and I'll get to it ASAP.
Quick start
- Install: Homebrew, apt, npm, or go install.
- Get a Jev key from typesafe.ai.
- Run:
webctl setup # asks for your Jev key
webctl search "mechanistic interpretability 2026" --goal "Recent papers on sparse autoencoders and circuit analysis"
Searching does not strictly require a key: with none configured, webctl calls the keyless Exa, Parallel, Keenable, You.com, and Firecrawl endpoints directly, then DuckDuckGo. Those throttle by IP after a few dozen searches a day, and webctl backs off with a cooldown ladder, so expect thin results without a key.
The author prefers Brave Search: 5,000 free searches a month, requires an API key. It is the first choice in webctl setup; once any search key is configured, only your keyed providers are used. (docs/providers.md)
Schematics
Filter search engine results to save tokens:
You: "latest advances in mechanistic interpretability 2025"
│
▼
┌──────────────┐
│ Search API │ (brave / ketch / parallel / ddg / etc)
│ 25 results │
└───────┬──────┘
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
┌───────────────────────────────────────────────────────┐
│ Jev │ typed relevance scoring
│ "is this on-topic, from a good source?" │
└───────┬───────┬───────┬───────┬───────┬───────┬───────┘
│ │ │ │ │ │ ✂️ the rest dropped
▼ ▼ ▼ ▼ ▼ ▼
┌────────────────────────────────────────────────────┐
│ 10–15 results (relevant) │ ✅ kept
└────────────────────────────────────────────────────┘
Filter chunks of scraped webpages to save even more tokens (--scrape --filter-chunks):
┌──────────────┐
│ Scraped page │ (2000 chars per chunk, judged with 20% overlap)
│ 25 chunks │
└───────┬──────┘
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
┌───────────────────────────────────────────────────────┐
│ Jev │ one batch request per page
│ "is this chunk relevant to the query?" │
└───────┬───────┬───────┬───────┬───────────────────────┘
│ │ │ │ ✂️ irrelevant chunks dropped
▼ ▼ ▼ ▼
┌──────────────────────────────────┐
│ 3–6 chunks (relevant) │ ✅ reassembled as the page text
└──────────────────────────────────┘
Deduplicate results:
┌──────────────┐
│ 3 engines │
│ 45 results │
└───────┬──────┘
│ pass 1: same normalized URL or title → collapsed ✂️
▼
┌──────────────┐
│ 25 results │ ──► Jev scores them (diagram 1)
└───────┬──────┘
│ pass 2: MinHash LSH over each excerpt
▼
┌───────────────────────────────────────────────────────┐
│ candidate pairs (a few) │ A~B C~D E~F
└───────┬───────────────┬───────────────┬───────────────┘
▼ ▼ ▼
┌───────────────────────────────────────────────────────┐
│ Jev │ one batch request
│ "are these two the same content?" │
└───────┬───────────────┬───────────────────────────────┘
│ yes │ yes ✂️ no: both stay
▼ ▼
┌───────────────────────────────────────────────────────┐
│ best-scored copy kept, engine tags merged │ ✅ others listed as duplicates
└───────────────────────────────────────────────────────┘
Respect rate limits with automatic cooldowns:
search ──► exa ──► HTTP 429 (rate limited) or 402 (quota spent)
│
▼
┌───────────────────────────────────────────────────────┐
│ ~/webctl/cooldown.json │ shared by every process
│ exa: strike 1, skip until +15m │
└───────────────────────────────────────────────────────┘
│
▼
next search ──► exa (skipped) ──► parallel ──► youcom ──► ddg ──► searxng
strike 1 2 3 4 5 6
window 15m ──► 1h ──► 4h ──► 12h ──► 24h ──► 72h ──► parked: one probe per 24h
▲
└── a 402 starts here
any success ──► strikes reset to 0
Installation
Prebuilt binaries for macOS and Linux (amd64 and arm64) are attached to every release.
Homebrew (macOS and Linux)
brew install dorkitude/webctl/webctl
apt (Debian and Ubuntu)
curl -fsSL https://dorkitude.github.io/webctl-apt/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/webctl.gpg
echo "deb [signed-by=/usr/share/keyrings/webctl.gpg] https://dorkitude.github.io/webctl-apt stable main" | sudo tee /etc/apt/sources.list.d/webctl.list
sudo apt update && sudo apt install webctl
npm
npm install -g webctl
go install
go install github.com/dorkitude/webctl/cmd/webctl@latest
Usage
Search
One pipeline: search providers → dedupe → Jev scores → threshold → optional scrape → print. docs/search.md
webctl search "q" --goal "what you actually need"
webctl search "q" -n 40 # results to request per provider (default 20)
Providers
Up to three providers per search, rankings fused by reciprocal rank: your SearXNG or Degoog if set, then the providers you set a key for (Brave first); keyless endpoints and DuckDuckGo only when no key exists. docs/providers.md
webctl search "q" -p exa # exactly one provider
webctl search "q" --sources 1 # first provider that answers
webctl search "q" --multi # every available provider
Filtering
Jev scores each result 0–10; the default cut is 6 ("useful" or better). docs/filtering.md
webctl search "q" --min-score 8.5 # stricter
webctl search "q" --min-results 5 # never fewer than 5 (marked backfilled)
webctl search "q" --noul "Is this a peer-reviewed paper?" # yes/no question instead of a score
webctl search "q" --rubric "off-topic,related,on-point" # custom scale
webctl search "q" --no-filter # skip Jev (works without a key)
Scraping
Fetch page text for each kept result; --filter-chunks keeps only the chunks Jev says are relevant (--chunk-chars, default 2000; judged with 20% overlap, output without it). Agents: prefer this over reading pages yourself most of the time, since only the relevant chunks reach your context. docs/scraping.md
webctl search "q" --scrape
webctl search "q" --scrape --filter-chunks
webctl search "q" --scrape --max-chars 20000 # default 50000 per page
webctl search "q" --scrape --filter-chunks --chunk-chars 1000 # default 2000
Dedupe
Exact duplicates (same normalized URL or title) collapse before scoring; near-duplicates are proposed by MinHash and confirmed by Jev after. docs/dedupe.md
webctl search "q" --no-dedupe # skip the near-duplicate pass
Cooldowns
A provider that answers 429 or 402 is skipped for a growing window (15m → 72h), shared by every process on the machine. docs/cooldowns.md
webctl cooldown # who is parked, strike, window
webctl cooldown clear exa # retry now
Output
webctl search "q" --json | jq '.[].url' # JSON array
webctl search "q" --urls-only # one URL per line
webctl search "q" --verbose # confidence, probabilities, dropped results
Config and keys
Flag → WEBCTL_* env → ~/webctl/config.yaml → default. Keys live in ~/secrets/keys.json. docs/config.md
webctl config show # every setting, its value, and where it came from
webctl config set min_score 2.2
webctl keys list|set|unset|validate
SearXNG
A local SearXNG has no quota. docs/searxng.md
docker run -d --name searxng -p 8899:8080 \
-v "$PWD/docs/searxng/settings.yml:/etc/searxng/settings.yml:ro" searxng/searxng:latest
webctl keys set searxng --value http://localhost:8899
Evals
Runs the cases in evals/cases/ through the real pipeline; results in docs/EVAL_REPORT.md. docs/evals.md
webctl eval
webctl eval report --cases
Providers
Chain order: your searxng or degoog if set, then the providers you set a key for. The keyless endpoints and ddg are used only when no key is set. Up to three are queried per search and fused. Full table with limits and cost: webctl docs providers.
| keyless | keyed |
|---|---|
exa, parallel, keenable, youcom, firecrawl (throttled by IP), ddg, searxng and degoog (your instances), ketch (if installed, via -p) | exa, parallel, sonar, youcom, brave, tavily, firecrawl, keenable, serpbase, serply |
webctl keys set brave # a key puts the provider in the chain
webctl search "query" -p tavily # exactly one provider
License
MIT