Token-estimation calibration

Divergence of the dependency-free heuristic (HeuristicEstimator, heuristic/v2) and the tiktoken default from real tokenizers, across fixed corpus shapes. Generated by benchmarks/token_calibration.py (issue #493).

• Generated: 2026-06-11
• tiktoken model: cl100k_base
• ⚠️ tiktoken encoding unavailable when generated (offline / no warmed TIKTOKEN_CACHE_DIR). The tiktoken and relative-error columns read n/a; regenerate in an environment with a warmed cache to populate them.
• Provider tokenizers (Anthropic / Gemini): not run (opt-in via CW_TOKEN_CALIBRATION_PROVIDERS + a registered counter; never run in CI).

Shape	Chars	Heuristic	tiktoken	Heuristic vs tiktoken
prose_en	132	32	n/a	n/a
prose_cjk	96	88	n/a	n/a
json	144	36	n/a	n/a
code	123	30	n/a	n/a
logs	147	36	n/a	n/a

Reading the table: a large negative Heuristic vs tiktoken error on prose_cjk is the failure mode issue #525 fixes — the script-aware heuristic keeps CJK within roughly ±30% of tiktoken instead of under-counting ~4×. For Latin prose, JSON, code, and logs the heuristic tracks len // 4, which is adequate for budget headroom decisions.