crawlora-deadweb

Most "dead links" aren't dead — they're alive and blocking your bot. A reachability scan of the top 10M domains found only ~14% are genuinely dead, not the usual ~27.6% — because most "dead link" checkers conflate a domain that's gone with one that's alive but refusing automated clients.

crawlora-deadweb is the open classifier behind that finding — a small, dependency-free CLI (and Go library) that probes a domain and labels it alive / redirect / blocked / dead, with the reason: gone (no DNS, nothing listening) vs. alive but refusing automated clients (403 / 429 / anti-bot).

It is a classifier, not an unblocker. It does a DNS lookup, a TCP connect, and one honest GET /, reads the response, and labels it. It never logs in, submits a form, solves a challenge, or tries to defeat anything.

Classification runs locally and open, from the public response. For the measured browser-fingerprint arm — re-probing a blocked domain with a real Chrome TLS/JA3 client across the proxied fleet to see which "blocked" sites are actually reachable — add --browser, which calls Crawlora's hosted engine.

This powers, and is the open companion to, the Dead-Web Index — a reachability census of the top 10 million domains that found ~14% genuinely dead, not the usual 27.6% (most "dead" is anti-bot blocking or a served error).

What the labels mean

• alive — a usable HTTP response (2xx, or a 4xx/5xx the server answered — a response isn't death).
• redirect — ended on an unresolved redirect.
• blocked — the host is up but won't serve us: anti-bot / auth / rate-limit, or it accepts a TCP connection but won't complete HTTP (tarpit / strict TLS).
• dead — no DNS resolution, a refused/reset connection, or nothing listening. Genuinely gone.

Install

# from source (Go 1.23+)
go install github.com/Crawlora-org/crawlora-deadweb@latest

# or clone + build
git clone https://github.com/Crawlora-org/crawlora-deadweb
cd crawlora-deadweb && go build -o crawlora-deadweb .

Prebuilt Linux / macOS / Windows binaries are published via GitHub Releases.

Usage

crawlora-deadweb [flags] <domain> [domain...]

$ crawlora-deadweb grooveshark.com reuters.com
grooveshark.com
outcome dead
reason  dns_failed — genuinely unreachable

reuters.com
outcome blocked
reason  forbidden (403) — alive but refusing this client
        (run with --browser for the measured browser-fingerprint arm)

--json emits NDJSON (one compact object per line) — pipe straight into jq -c or a data pipeline.

Batch / pipelines. Pass many domains as args, or pipe a list on stdin (one per line; blank lines and #-comments ignored). Domains are probed in parallel (--concurrency, default 8):

cat domains.txt | crawlora-deadweb --json --concurrency 50 > results.ndjson
printf 'grooveshark.com\nexample.com\n' | crawlora-deadweb

Each JSON record matches the open dataset schema: domain, tld, outcome, reason, first_status, final_status, final_url, scheme, hops, parked.

The browser arm (optional, hosted)

The local probe is a polite HTTP request from your IP, so "blocked" is an upper bound — a vendor refusing a datacenter client ≠ the site being unreachable. For the measured tier — what actually gets through with a real browser fingerprint and the proxied fleet — add --browser:

export CRAWLORA_API_KEY=...   # get one at https://crawlora.net?utm_source=github&utm_medium=referral&utm_campaign=crawlora-deadweb
crawlora-deadweb --browser reuters.com
# outcome blocked  reason forbidden (403) ← local
# ── measured (Crawlora API) ──
# browser arm: reachable ✓           ← live measurement

Flags

Flag	Default	Description
--json	false	Output NDJSON (one object per line).
--browser	false	Also fetch the measured browser-fingerprint arm from the Crawlora API (needs a key).
--timeout	20s	Per-attempt timeout for the local probe.
--retries	1	Retries on transient (timeout / reset) failures.
--concurrency	8	Domains probed in parallel (batch mode).
--api-key	$CRAWLORA_API_KEY	Crawlora API key (for --browser).
--api-base	$CRAWLORA_API_BASE or https://api.crawlora.net/api/v1	API base URL.
--user-agent	Chrome UA	User-Agent for the local probe.
--version		Print version and exit.

How classification works

A DNS lookup across 8 public resolvers (IPv4-first, so an IPv4-only vantage doesn't dead-label a dual-stack host), then HTTPS-first with HTTP fallback, redirects followed (max 10), and one retry on transient failures. When HTTP doesn't answer, a raw TCP-connect check on :443/:80 separates "up but unresponsive" (blocked) from "gone" (dead). The honest rule: a received status is never dead — 404/410/5xx → alive (the server answered), 403/429/anti-bot (incl. the non-standard 999) → blocked, only no-DNS / refused / nothing-listening → dead.

It's a lower bound on "alive": homepage-level, from your vantage, passive. Deep pages and residential vantages reach more — that's what --browser measures.

Use as a library

import "github.com/Crawlora-org/crawlora-deadweb/classify"

res := classify.Inspect(ctx, "example.com", classify.Options{})
fmt.Println(res.Outcome, res.Reason) // alive ok

Common Crawl: a longitudinal dead-web panel (cc)

The classifier above is a snapshot. The cc subcommands build a longitudinal view — when domains went dark — from Common Crawl's robotstxt subset: every monthly crawl since ~2018 records an HTTP response for /robots.txt on tens of millions of hosts (~40M registered domains/crawl), so a domain that was present for years and then disappears is a death/abandonment signal — over a window no single live scan can reproduce.

The catch: disappearance over-counts death (a vanished domain may be alive-but-blocking the crawler, or merely dropped from the crawl's seed set). So cc ends with a calibration step that live-probes a sample (reusing the classifier above) and reports the share that is truly dead, with confidence intervals — turning "X% disappeared" into a defensible "Y% dead".

# 1. per-crawl reachability panels (reads CC parquet over HTTPS; needs the duckdb CLI on PATH)
crawlora-deadweb cc extract --from 2018 --to 2026 --out panel/
# 2. merge into a labeled longitudinal panel: alive_present / dead_candidate / dark_ambiguous / intermittent
crawlora-deadweb cc merge --panel 'panel/*.parquet' --out labeled.parquet
# 3. analyze (label mix, last-seen-state split, coverage)
crawlora-deadweb cc analyze --labeled labeled.parquet
# 4. draw a calibration sample, live-probe it, and calibrate
crawlora-deadweb cc sample --labeled labeled.parquet -n 20000 > sample.csv
cut -d, -f1 sample.csv | crawlora-deadweb --json --concurrency 50 > probe.ndjson
crawlora-deadweb cc calibrate --sample sample.csv --probe probe.ndjson

$ crawlora-deadweb cc calibrate --sample sample.csv --probe probe.ndjson
OVERALL (share of disappearances that are truly dead):
  overall   n=20000   dead 47.2% [46.5–47.9]   blocked 6.1%   alive 46.7%

Requirements & responsible use. cc extract/merge/sample/analyze need the DuckDB CLI on PATH (a single MIT binary) and only download from data.commoncrawl.org (a CDN — not a scan). The probe step (step 4) connects to many distinct hosts, which can look like a port scan to your provider — run it from controlled egress or via --browser (the Crawlora API's proxied fleet), and keep --concurrency reasonable. Run crawlora-deadweb cc for the full command/flag list.

Scope & ethics

Each probe is an unauthenticated GET / to a public homepage (plus DNS/TCP checks) — the same request any browser makes. No authentication is bypassed, no login-only or robots.txt-disallowed paths are fetched, and no page content is republished. It's a classifier, not a bypass — respect each site's Terms of Service and robots.txt, and only probe what you're authorized to.

License

MIT. Built by Crawlora. See the open Dead-Web Index and dataset.