contextweaver

July 12, 2026 · View on GitHub

Auto-generated by make scorecard. Do not edit by hand. Source: benchmarks/results/latest.json (produced by make benchmark).

  • Harness version: 1.2
  • Seed: 42
  • Rank cutoff k: 5
  • Token estimator: CharDivFourEstimator (deterministic, no model dependency)
  • Answer-phase budget: 6000 tokens

All numbers below are reproducible deterministically by running make benchmark && make scorecard from a clean checkout. Hardware and Python version affect latency only; recall, drops, dedup, and token counts are environment-independent.

Routing accuracy & latency

Gold dataset: 200 hand-curated queries (benchmarks/routing_gold.json) covering all 8 catalog namespaces. Each query is repeated three times for latency percentile stability.

catalog_sizequeriesprecision@5recall@5MRRp50 (ms)p95 (ms)p99 (ms)
501310.11910.56490.49780.4200.5140.650
832000.08000.38250.32420.6910.8411.205
10002000.03100.14750.145637.08640.61047.855

Per-backend × per-size matrix

Generated by make benchmark-matrix (issue #208). The matrix lets you compare the bundled tfidf baseline against the optional bm25 ([bm25] extra), fuzzy ([retrieval] extra), and embedding backends (embedding_hashing stdlib baseline, embedding_st [embeddings] extra — #266) across three catalog sizes. Skipped backends are recorded explicitly.

backendcatalog_sizequeriesprecision@5recall@5MRRp50 (ms)p95 (ms)p99 (ms)status
bm251002000.08000.38250.33995.3947.5918.399ok
bm255002000.04700.22500.216528.70736.88141.865ok
bm2510002000.03300.15750.152571.63889.514112.387ok
embedding_hashing1002000.10500.51750.43600.9076.9537.218ok
embedding_hashing5002000.05600.27000.267410.07039.18043.171ok
embedding_hashing10002000.04200.20000.193137.09494.277102.979ok
embedding_st100———————skipped: missing sentence-transformers
embedding_st500———————skipped: missing sentence-transformers
embedding_st1000———————skipped: missing sentence-transformers
fuzzy100———————skipped: missing rapidfuzz
fuzzy500———————skipped: missing rapidfuzz
fuzzy1000———————skipped: missing rapidfuzz
tfidf1002000.08000.38250.32200.9491.0821.193ok
tfidf5002000.04900.23250.23149.65210.10510.803ok
tfidf10002000.03100.14750.145636.41939.12042.986ok

Mixed-namespace catalog (500 tools, head + long tail)

Generated by python benchmarks/benchmark.py --matrix --mixed-shapes (issue #277). The mixed-shape catalog has one head namespace (analytics_xl, 200 items), two mid-weight namespaces, four small operational namespaces, and a 100-namespace long tail (one item each) — a deliberately asymmetric distribution that contrasts with the uniform 8-namespace pool used by the headline matrix. Numbers here are intentionally lower: gold queries only cover the natural 8 namespaces, so the synthetic head-and-tail tools act as targeted noise items.

backendcatalog_sizequeriesprecision@5recall@5MRRp50 (ms)p95 (ms)p99 (ms)status
bm255002000.02600.12000.106520.25624.96326.822ok
embedding_hashing5002000.03400.15500.137610.68040.38042.665ok
embedding_st500———————skipped: missing sentence-transformers
fuzzy500———————skipped: missing rapidfuzz
tfidf5002000.02600.12000.10109.91210.62613.045ok

Per-namespace recall@k

Breakdown of recall@k by gold-set namespace (issue #209). Useful for identifying which tool namespaces benefit most from a given scoring backend.

backendcatalog_sizenamespacequeriesrecall@5
bm25100admin250.9200
bm25100analytics250.7200
bm25100billing250.1200
bm25100comms250.4000
bm25100crm250.1200
bm25100docs250.2800
bm25100infra250.0800
bm25100search250.4200
bm25500admin250.4600
bm25500analytics250.2400
bm25500billing250.5800
bm25500comms250.3600
bm25500crm250.0400
bm25500docs250.0800
bm25500infra250.0400
bm25500search250.0000
bm251000admin250.5800
bm251000analytics250.4000
bm251000billing250.2400
bm251000comms250.0000
bm251000crm250.0400
bm251000docs250.0000
bm251000infra250.0000
bm251000search250.0000
embedding_hashing100admin250.4000
embedding_hashing100analytics250.5200
embedding_hashing100billing250.3200
embedding_hashing100comms250.4400
embedding_hashing100crm250.6400
embedding_hashing100docs250.5600
embedding_hashing100infra250.6800
embedding_hashing100search250.5800
embedding_hashing500admin250.5000
embedding_hashing500analytics250.4400
embedding_hashing500billing250.5800
embedding_hashing500comms250.3200
embedding_hashing500crm250.0400
embedding_hashing500docs250.1600
embedding_hashing500infra250.0400
embedding_hashing500search250.0800
embedding_hashing1000admin250.5800
embedding_hashing1000analytics250.5200
embedding_hashing1000billing250.3800
embedding_hashing1000comms250.0000
embedding_hashing1000crm250.0400
embedding_hashing1000docs250.0000
embedding_hashing1000infra250.0000
embedding_hashing1000search250.0800
tfidf100admin250.9200
tfidf100analytics250.7200
tfidf100billing250.1200
tfidf100comms250.4000
tfidf100crm250.1200
tfidf100docs250.2800
tfidf100infra250.0800
tfidf100search250.4200
tfidf500admin250.4600
tfidf500analytics250.2000
tfidf500billing250.5400
tfidf500comms250.3600
tfidf500crm250.0400
tfidf500docs250.0800
tfidf500infra250.0400
tfidf500search250.1400
tfidf1000admin250.5800
tfidf1000analytics250.3600
tfidf1000billing250.2000
tfidf1000comms250.0000
tfidf1000crm250.0400
tfidf1000docs250.0000
tfidf1000infra250.0000
tfidf1000search250.0000

Reading the routing tables:

  • precision@5 is bounded by 1 / k when each query has a single expected tool, so the headline accuracy signal is recall@k and MRR.
  • Recall degrades predictably as the catalog grows — noise items compete with true matches; the routing-only experience for catalogs larger than ~200 items benefits from one of the optional retrieval backends (bm25, fuzzy) configured via Router(scorer_backend=...).
  • p50/p95 latencies stay under a millisecond at catalog_size ≤ 83. At catalog_size 1000 the p99 climbs into the tens of milliseconds because the beam search has to evaluate substantially more children per step; this is the regime where switching to a retriever-first shortlist (the Retriever protocol on the EngineRegistry) is the expected next step.

Context pipeline scenarios

Reference event logs under benchmarks/scenarios/ are pushed through ContextManager.build_sync(phase=Phase.answer). The firewall intercepts every tool_result; large results become artifacts and the prompt sees their summaries instead.

scenarioeventsincludeddroppeddeduptokensbudgetutil %artifactscompaction
large_catalog6060001480600024.7%151.00x
long_conversation8282002500600041.7%211.41x
mixed_payload15150048860008.1%41.64x
short_conversation18180048760008.1%41.00x
stress_conversation14713611465906000109.8%323.29x
tiny_payload20200025660004.3%51.00x

Reading the table:

  • dropped > 0 means select_and_pack had to evict candidates to stay under the answer-phase budget — the stress_conversation scenario is sized to force this so the budget-driven selection stage shows up in benchmark output (#181).
  • dedup > 0 proves the Jaccard near-duplicate stage actively removed redundant context.
  • compaction > 1.0× is the average ratio of raw artifact bytes to injected summary bytes — that's the firewall's load-bearing job.
  • compaction == 1.00× rows (tiny_payload, short_conversation, large_catalog) are the firewall correctly no-op'ing on tiny inputs — every tool_result still passes through the envelope, but the summary equals the raw text so no token cost is saved or added (#271). The headline reduction figures further down reflect the firewall's value when payloads are actually large; small-payload rows guard against the impression that the firewall always wins.

vs. naïve concat baseline

Token cost and coverage proxy against the no-op baseline of concatenating all tool schemas + the full conversation history. The naïve token count is measured with tiktoken.cl100k_base; the coverage proxy is items_included / event_count (deterministic, no LLM judge — see scripts/baseline_naive.py).

scenarionaive_tokenscw_tokenstoken reductioncoverage proxy
large_catalog2767148046.51%100.00%
long_conversation4365250042.73%100.00%
mixed_payload227748878.57%100.00%
short_conversation194648774.97%100.00%
stress_conversation17482659062.30%92.52%
tiny_payload170425684.98%100.00%

Methodology

  • Deterministic seeds. All catalog generation, scenario loading, and beam-search tie-breaking is seeded; identical inputs always produce identical outputs (make benchmark is a no-op on a fresh re-run for routing accuracy + context metrics; only latency varies with hardware).
  • No LLM calls by default. The harness is pure-Python, stdlib + minimal core deps. The token estimator is CharDivFourEstimator so the headline numbers do not depend on tiktoken's cached encoding state. The optional end-to-end real-model section (#269) is off by default and never runs in CI.
  • No network access by default. The benchmark is safe to run in air-gapped CI environments. The end-to-end real-model capture (#269) and the tiktoken parity check (#268) require network or a cached encoding respectively, and degrade to a skipped row when unavailable.
  • Hardware variance. Latency numbers are measured on the runner that produced latest.json. Treat them as ordering, not absolutes: the relative cost between catalog sizes is portable; the absolute microsecond count is not. See the Hardware reference rig section below for the canonical reference machine (#267).

Hardware reference rig

Reference rig (numbers in this scorecard are calibrated for):

  • GitHub Actions ubuntu-latest (2-core x86_64)
  • System: Linux
  • Machine: x86_64
  • CPU logical cores: 2
  • Python: 3.10+

Absolute latency on other hardware will differ; the relative cost between catalog sizes and backends is portable.

Measured on (host that produced the current latest.json):

  • System: Linux
  • Machine: x86_64
  • Processor: x86_64
  • CPU logical cores: 4
  • Python: CPython 3.11.15
  • Platform: Linux-6.18.5-x86_64-with-glibc2.39

Token-estimator parity check

Quantifies the drift between CharDivFourEstimator (used everywhere above) and tiktoken.cl100k_base on the gold-query corpus. This is measurement of the estimator, not a routing metric — when the two diverge, the headline token figures above are still self- consistent, but readers comparing the numbers against an OpenAI tokenizer expectation should apply the drift below (#268).

Token-estimator parity check is unavailable on this run (status: skipped: ProxyError); the headline CharDivFourEstimator numbers elsewhere in this scorecard are still valid — they do not depend on tiktoken at runtime.

Optional end-to-end real-model capture

Off by default. When enabled (--with-real-model flag plus CW_BENCH_LLM_PROVIDER + CW_BENCH_LLM_API_KEY env vars) the harness sends ≤5 gold queries through an OpenAI-compatible chat endpoint and records prompt/completion tokens, USD cost, and round-trip latency. Pure end-to-end, no fan-out, no retries; the section exists so deployment-cost readers can sanity-check the harness's offline token estimates against real provider usage (#269).

Status: skipped: offline by default.

Set CW_BENCH_LLM_PROVIDER + CW_BENCH_LLM_API_KEY and pass --with-real-model to enable. See benchmarks/README.md.

See benchmarks/README.md for the full harness reference and the per-scenario notes.

Regenerating

make benchmark   # writes benchmarks/results/latest.json
make scorecard   # writes benchmarks/scorecard.md from latest.json
git diff --quiet benchmarks/scorecard.md   # passes on clean re-run

Per-backend matrices are generated via make benchmark-matrix (#208) and now cover tfidf, bm25, fuzzy, embedding_hashing, and embedding_st (#266). Weekly scheduled regeneration runs out of .github/workflows/benchmark-scorecard-weekly.yml (#207).

Optional capture flags:

  • --mixed-shapes — emit the head-heavy + long-tail catalog matrix (#277).
  • --no-tiktoken-parity — disable the estimator parity check (#268).
  • --with-real-model — run the end-to-end real-model capture (#269).