GraphFlow Token-Savings Benchmark

Auto-generated by npm run benchmark (benchmarks/run-token-benchmark.ts). Last run: 2026-08-04T11:27:56.422Z

如何复现（How to reproduce）

npm install && npm run benchmark   # Node ≥ 20，离线运行，无需 API key

• 环境：Node ≥ 20（本表由 v22.17.1 / win32 x64 生成）；无网络、无 API key。
• 结果文件：本文件 benchmarks/RESULTS.md（人类可读）+ benchmarks/.cache/token-bench-results.json（机器可读，含 commit hash 与运行日期）。
• 输入数据、判定标准与第三方复现清单见 docs/benchmark-standards.md。

Summary

GraphFlow compresses the context an LLM agent needs for a query into a small summary + anchor package, instead of pushing whole source files into the prompt. Across 8 representative queries against this repository:

• Baseline total input tokens: 274,434
• GraphFlow total input tokens: 4,928
• Overall token savings: 98.2%

The headline GraphFlow tokens column is independently re-tokenized with gpt-tokenizer (summary + anchors), not GraphFlow's self-reported estimate. GraphFlow's own internal estimate is shown separately for transparency and is typically lower.

Results table

Query	Baseline tokens	GraphFlow tokens (independent)	GraphFlow self-estimate	Savings %
orchestrator	35,886 (10 files)	570	325	98.4%
context compression	34,099 (10 files)	672	416	98.0%
model routing	23,400 (10 files)	584	362	97.5%
graph index	41,383 (10 files)	631	359	98.5%
token savings	35,518 (10 files)	740	461	97.9%
semantic enrichment	22,640 (10 files)	524	263	97.7%
preview context	31,944 (10 files)	544	278	98.3%
skill flywheel	49,564 (10 files)	663	383	98.7%
Total	274,434	4,928	2,847	98.2%

Run environment

• Node: v22.17.1
• Platform: win32 x64
• Tokenizer: gpt-tokenizer (gpt-4o encoding) — identical for both sides.
• Graph nodes in store: 4,076
• Files (re)indexed this run: 117 (0 = incremental, graph already current)
• Benchmark wall-clock: 35.2s

Methodology & honest caveats

Baseline (no graph) — emulates a traditional coding agent with no context layer. For each query we grep the src/ tree (extensions: .ts, .tsx), rank files by the number of query-term matches, take the top 10 files, and count the tokens of their full contents. This represents the common "dump the relevant files into the prompt" strategy.

• This is a simulation, not a capture of a specific real agent. The number of files an agent reads varies; we cap it at 10 per query to keep the baseline bounded and reproducible. A higher cap would only increase the reported savings, so this cap is a conservative choice.
• We scan only src/ (not tests, build output, or generated graph artifacts), which is the code an agent would realistically read for these queries.
• We count full-file contents because agents without precise navigation tend to read whole files rather than exact line ranges.

GraphFlow (compressed) — calls the exported previewContext function. The headline number is obtained by independently re-tokenizing the exact text an agent receives (the summary channel + anchor pointer lines) with gpt-tokenizer. We deliberately do not trust GraphFlow's self-reported tokenEstimate for the headline (that estimate is shown in its own column and is typically lower, i.e. our reported savings are the conservative side).

• Anchors are lightweight pointers (id/type/layer). An agent expands only the anchors it needs via expand_anchor, so the initial context cost is what we measure here. Even if every anchor were later expanded, the upfront budget is what determines whether the task fits in a small context window.
• Semantic (LLM) compression and network embeddings are disabled for this run so it is offline and deterministic; only zero-cost graph-structure compression is active. Enabling semantic compression would typically improve savings further.

Reproduce

npm install
npm run benchmark

No API key is required. The script builds its own isolated graph under benchmarks/.cache/ and replaces its own section in this file on every run (sections appended by the other benchmark scripts are preserved). The machine-readable numbers land in benchmarks/.cache/token-bench-results.json (per-query results, totals, inputs, environment, commit hash and run date).

Skill-Flywheel End-to-End A/B Benchmark — Results (P1-2)

Appended by npm run benchmark:ab (benchmarks/run-skill-ab.ts). Last run: 2026-08-04T11:46:40.564Z Structured JSON: benchmarks/.cache/skill-ab-results.json

Summary

26 retrieval-golden tasks (10 "indirect": the golden module name is morphologically distinct from the query words, e.g. "orchestrate" vs orchestrator), run end-to-end on an in-memory graph seeded with the golden target, distractors and a decoy. Arm A additionally simulates 27 historical tasks through the real learning paths.

Metric	Arm A (flywheel ON)	Arm B (flywheel OFF)
Success proxy (golden target within Top-5)	100.0% (26/26)	61.5% (16/26)
Success via package Top-5 only	61.5%	61.5%
Tasks rescued by flywheel (B miss → A hit)	10	—
Tasks hurt by flywheel (B hit → A miss)	0	—
Hint injection rate	100.0%	0%
Episode recall rate	100.0%	0%
Mean prompt-token overhead / task	55.1	0
Total prompt-token overhead	1432	0
Mean package tokens / task	639	46.2
Decoy contamination (Top-5)	3.8%	3.8%
Decoy contamination (injection)	0.0%	0%
Mean wall-clock / task	2.5 ms	2.0 ms
Total wall-clock	0.2 s	—

Per-task detail

Task	Golden module	Direct	Top-5 (B)	Success (A)	Pkg Top-5 (A)	Injection hit (A)	Hints	Episodes	Overhead tokens
orchestrate task routing	orchestrator	no	miss	hit	no	yes	2	3	90
dag execution engine	dag-engine	yes	hit	hit	yes	yes	2	2	61
triage task classification simple complex	triage	yes	hit	hit	yes	yes	3	1	45
model router provider selection	model-router	yes	hit	hit	yes	yes	3	3	93
provider health fallback chain	provider-health	yes	hit	hit	yes	yes	3	3	85
graph compression pagerank centrality	graph-compression	yes	hit	hit	yes	yes	3	3	91
context slicer layered package	context-slicer	yes	hit	hit	yes	yes	2	2	60
skill flywheel hints scoring	skill-flywheel	yes	hit	hit	yes	yes	3	1	34
episodic memory similar episodes	episodic-memory	yes	hit	hit	yes	yes	1	1	30
embedding cosine similarity vector	embeddings	no	miss	hit	no	yes	2	1	30
file watcher incremental index on save	filewatcher	no	miss	hit	no	yes	3	2	69
sqlite graph storage fts5	sqlite-client	yes	hit	hit	yes	yes	2	3	89
repo map module overview	repomap	no	miss	hit	no	yes	1	1	25
token savings statistics	tokensavings	no	miss	hit	no	yes	2	2	56
mcp server tool definitions	tool-definitions	yes	hit	hit	yes	yes	2	1	34
cli output json formatting	formatcliresult	no	miss	hit	no	yes	2	1	34
agent delegation work items bridge	workitem	no	miss	hit	no	yes	3	1	32
six hats insight planning	brainstormer	no	miss	hit	no	yes	2	1	31
hnsw approximate nearest neighbor index	hnsw	yes	hit	hit	yes	yes	2	2	68
adaptive token budget estimation	estimatecontextbudget	no	miss	hit	no	yes	3	3	89
artifact export import graph snapshot	artifact-manager	yes	hit	hit	yes	yes	3	3	89
nightly learning trainer	nightly-trainer	yes	hit	hit	yes	yes	1	1	29
reflect episodes extract lessons	reflector	no	miss	hit	no	yes	3	1	34
dag checkpoint recovery taskrun	dag-checkpoint	yes	hit	hit	yes	yes	3	2	67
cancellation timeout controller	cancellation	yes	hit	hit	yes	yes	2	1	27
language indexers tree sitter wasm	language-indexers	yes	hit	hit	yes	yes	2	1	40

Methodology & honest caveats

• Task set is duplicated from tests/retrieval-golden.test.ts GOLDEN_SET (that file is owned by another agent and is not modified). Each task's golden node id (file:src/golden/<module>.ts) contains an expectAny alternative verbatim.
• Success proxy: Top-K = the first 5 ranked anchors of the compressed context package (the retrieval channel); success there means the golden node id is within those 5 anchors (precise id membership — avoids substring false positives from distractor/decoy nodes). For Arm A, the injected hints + episode summaries form an additional channel an agent reads in full; hints/episodes reference modules by name, so injection success is the expectAny substring check. Arm A success = package Top-5 hit or injection hit. The package-only hit rate is reported separately for a like-for-like retrieval comparison.
• Arm B success is deliberately imperfect: for the 10 indirect tasks the golden file shares zero tokens with the query (realistic: module names are morphologically different from task wording), so pure retrieval cannot find it — prior episodic experience is the only bridge.
• Both arms run through the real retrieval and learning paths (buildEnhancedContextPackage, applySkillLearning, recordEpisode, suggestSkillHints, findSimilarEpisodes, summarizeEpisodeForPrompt) on isolated in-memory graphs — no mocks, no network, no API key.
• Token counts use gpt-tokenizer (gpt-4o encoding), identical to the token benchmark. Hashing is the project's DJB2a (FNV-class) — fully deterministic within a run; episode ids embed Date.now() so ids differ across runs, but ranking depends on tokens/scores, not ids.
• This measures a mechanical success proxy, not LLM task completion. It validates that the flywheel's injected context and graph nodes move the needle on finding the expected target, and quantifies the exact token and wall-clock cost.

Episodic-Memory End-to-End A/B Benchmark — Results (P3)

Appended by npm run benchmark:memory (benchmarks/run-memory-ab.ts). Last run: 2026-08-04T11:27:08.785Z Structured JSON: benchmarks/.cache/memory-ab-results.json

Summary

62 tasks: 26 retrieval-golden queries (duplicated from tests/retrieval-golden.test.ts, same list as the P1-2 skill benchmark) plus 36 HARD-domain tasks (12 cross-module, 12 disambiguation, 12 indirect), run end-to-end on an in-memory graph seeded with the golden target, distractors and a decoy. Hard tasks are constructed so the golden node shares zero query tokens (node ids are not searchable text), so the OFF arm cannot rank them — episodic memory is the only bridge. Arm A additionally simulates 63 historical tasks through the real learning paths (applySkillLearning + recordEpisode).

Metric	Arm A (memory ON)	Arm B (memory OFF)
Success proxy (golden target within Top-5)	100.0% (62/62)	56.5% (35/62)
Success via package Top-5 only	46.8%	56.5%
Tasks rescued by memory (B miss → A hit)	27	—
Tasks hurt by memory (B hit → A miss)	0	—
Hint injection rate	95.2%	0%
Episode recall rate	100.0%	0%
Distinct memories that rescued tasks	26	—
Mean prompt-token overhead / task	70.9	0
Total prompt-token overhead	4397	0
Mean package tokens / task	500.1	67.8
Decoy contamination (Top-5)	3.2%	3.2%
Decoy contamination (injection)	4.8%	0%
Mean wall-clock / task	4.5 ms	1.6 ms
Total wall-clock	0.6 s	—

Memory-contribution summary (top-3 contributing memories)

Episode id	Episode task	Tasks rescued	Rescued queries	Mean similarity
episode:10s9w7a	token savings numbers audited in tokensavings	2	token savings statistics, numbers behind the token savings	0.481
episode:1arcr65	mcp restart stuck vscode-extension panel	1	editor panel stuck after mcp restart	0.671
episode:11q0ki	similar past problems matched by embeddings	1	find similar past problems	0.600

Attribution chain (rescued tasks: which memory carried the rescue)

Task	Golden module	Carry channel	Top episode id	Top episode task	Similarity
orchestrate task routing	orchestrator	episode	episode:wkys9z	fixed orchestrator routing deadlock	0.267
embedding cosine similarity vector	embeddings	episode	episode:cmcjy0	embeddings cosine similarity provider	0.433
file watcher incremental index on save	filewatcher	both	episode:6q1rsq	filewatcher incremental index invalidation	0.386
sqlite graph storage fts5	sqlite-client	both	episode:1yskm3p	sqlite-client fts5 migration	0.267
repo map module overview	repomap	episode	episode:18ggy5r	repomap overview generation	0.267
token savings statistics	tokensavings	both	episode:10s9w7a	token savings numbers audited in tokensavings	0.433
cli output json formatting	formatcliresult	episode	episode:1yi5b5c	formatcliresult json output handling	0.433
agent delegation work items bridge	workitem	episode	episode:1eu8wz0	workitem bridge delegation	0.433
six hats insight planning	brainstormer	episode	episode:1266q47	brainstormer insight planning	0.500
adaptive token budget estimation	estimatecontextbudget	episode	episode:mj7beu	estimatecontextbudget token sizing	0.267
reflect episodes extract lessons	reflector	episode	episode:gnni0q	reflector episode lessons	0.267
provider adapter crash kills routing	model-router	episode	episode:1js4gvz	provider adapter crash killed model-router dispatch	0.475
cli flags now ignored after settings refactor	config-loader	episode	episode:1xzimbo	settings refactor broke cli flag loading in config-loader	0.373
editor panel stuck after mcp restart	vscode-extension	episode	episode:1arcr65	mcp restart stuck vscode-extension panel	0.671
persistent storage for graph data	sqlite-client	both	episode:150tyoi	graph data persisted through sqlite-client store	0.322
upstream probe fails then what	provider-health	both	episode:9sh6e9	upstream probe failures tripped provider-health	0.350
simple versus complex task split	triage	both	episode:1utbkfu	simple tasks bypassed triage queue	0.211
learn from what went wrong	reflector	episode	episode:8bp9bm	what went wrong captured by reflector	0.529
stop a hung run safely	cancellation	both	episode:wfucm1	hung run cancelled by cancellation controller	0.386
numbers behind the token savings	tokensavings	episode	episode:10s9w7a	token savings numbers audited in tokensavings	0.529
find similar past problems	embeddings	both	episode:11q0ki	similar past problems matched by embeddings	0.600
fast nearest neighbor search	hnsw	both	episode:nbt8jp	nearest neighbor search too slow in hnsw	0.529
protocol messages over the wire	atp-schema	both	episode:19tm79n	protocol message framing changed in atp-schema	0.211
describe the work before planning	task-profile	episode	episode:1w559i5	describe work shape with task-profile	0.350
pick between conflicting options	decision-engine	both	episode:e10lwi	conflicting options resolved by decision-engine	0.433
peek at the graph state	snapshot-view	episode	episode:1el7lr7	snapshot-view graph state dump	0.433
export the learning dataset	exporter	both	episode:1fudjbu	artifact-manager export compression	0.267

Per-task detail

Task	Golden module	Kind	Top-5 (B)	Success (A)	Pkg Top-5 (A)	Carry (A)	Hints	Episodes	Overhead tokens
orchestrate task routing	orchestrator	golden	miss	hit	no	episode	2	2	55
dag execution engine	dag-engine	golden	hit	hit	yes	none	2	3	80
triage task classification simple complex	triage	golden	hit	hit	yes	none	3	3	91
model router provider selection	model-router	golden	hit	hit	yes	none	3	3	89
provider health fallback chain	provider-health	golden	hit	hit	yes	none	3	3	85
graph compression pagerank centrality	graph-compression	golden	hit	hit	yes	none	3	3	85
context slicer layered package	context-slicer	golden	hit	hit	yes	none	2	2	57
skill flywheel hints scoring	skill-flywheel	golden	hit	hit	yes	none	3	3	79
episodic memory similar episodes	episodic-memory	golden	hit	hit	yes	none	3	3	78
embedding cosine similarity vector	embeddings	golden	miss	hit	no	episode	2	2	60
file watcher incremental index on save	filewatcher	golden	miss	hit	no	both	3	3	92
sqlite graph storage fts5	sqlite-client	golden	miss	hit	no	both	3	3	88
repo map module overview	repomap	golden	miss	hit	no	episode	0	1	20
token savings statistics	tokensavings	golden	miss	hit	no	both	3	3	79
mcp server tool definitions	tool-definitions	golden	hit	hit	yes	none	3	3	100
cli output json formatting	formatcliresult	golden	miss	hit	no	episode	1	2	60
agent delegation work items bridge	workitem	golden	miss	hit	no	episode	3	3	79
six hats insight planning	brainstormer	golden	miss	hit	no	episode	0	1	20
hnsw approximate nearest neighbor index	hnsw	golden	hit	hit	yes	none	3	3	89
adaptive token budget estimation	estimatecontextbudget	golden	miss	hit	no	episode	2	3	77
artifact export import graph snapshot	artifact-manager	golden	hit	hit	yes	none	3	3	83
nightly learning trainer	nightly-trainer	golden	hit	hit	yes	none	2	2	54
reflect episodes extract lessons	reflector	golden	miss	hit	no	episode	3	2	57
dag checkpoint recovery taskrun	dag-checkpoint	golden	hit	hit	yes	none	0	2	47
cancellation timeout controller	cancellation	golden	hit	hit	yes	none	3	2	53
language indexers tree sitter wasm	language-indexers	golden	hit	hit	yes	none	3	3	96
provider adapter crash kills routing	model-router	cross-module	miss	hit	no	episode	3	3	79
adding an embedding provider changed graph scores	graph-compression	cross-module	hit	hit	no	none	3	3	78
cli flags now ignored after settings refactor	config-loader	cross-module	miss	hit	no	episode	3	1	32
indexer reindex wipes mcp session state	mcp-server	cross-module	hit	hit	no	none	3	3	85
new language grammar breaks file watching	file-watcher	cross-module	hit	hit	yes	none	3	3	84
sqlite migration loses episodic history	episodic-memory	cross-module	hit	hit	yes	none	3	3	84
tool schema update confuses agent profiles	tool-definitions	cross-module	hit	hit	no	none	3	3	75
nightly training corrupts workspace config	workspace-root	cross-module	hit	hit	no	none	3	3	78
editor panel stuck after mcp restart	vscode-extension	cross-module	miss	hit	no	episode	3	3	88
query expansion returns chinese results	query-expand	cross-module	hit	hit	yes	none	3	1	35
feedback loop inflates skill scores	skill-flywheel	cross-module	hit	hit	no	none	3	3	85
atomic skill writes lock the whole store	skill-store	cross-module	hit	hit	yes	none	3	3	72
tree walker skips hidden directories	file-indexer-walker	disambiguation	hit	hit	yes	none	3	1	47
persistent storage for graph data	sqlite-client	disambiguation	miss	hit	no	both	3	3	90
cached context goes stale on file save	context-cache	disambiguation	hit	hit	yes	none	3	3	96
which agent gets this task	agent-assignment	disambiguation	hit	hit	yes	none	3	2	58
upstream probe fails then what	provider-health	disambiguation	miss	hit	no	both	3	2	59
simple versus complex task split	triage	disambiguation	miss	hit	no	both	3	2	65
learn from what went wrong	reflector	disambiguation	miss	hit	no	episode	3	3	87
stop a hung run safely	cancellation	disambiguation	miss	hit	no	both	2	2	61
numbers behind the token savings	tokensavings	disambiguation	miss	hit	no	episode	3	3	82
find similar past problems	embeddings	disambiguation	miss	hit	no	both	3	1	33
fast nearest neighbor search	hnsw	disambiguation	miss	hit	no	both	3	1	39
did the run stay on target	goal-anchor	disambiguation	hit	hit	yes	none	3	3	93
execute the chosen provider	provider-executor	indirect	hit	hit	yes	none	3	3	80
what can each role do	role-capabilities	indirect	hit	hit	yes	none	3	2	68
protocol messages over the wire	atp-schema	indirect	miss	hit	no	both	3	3	90
describe the work before planning	task-profile	indirect	miss	hit	no	episode	3	3	77
transition states in order	state-machine	indirect	hit	hit	yes	none	3	3	95
break the goal into steps	planner	indirect	hit	hit	yes	none	3	3	87
pick between conflicting options	decision-engine	indirect	miss	hit	no	both	3	1	34
peek at the graph state	snapshot-view	indirect	miss	hit	no	episode	3	3	75
stats about the knowledge graph	graph-analysis	indirect	hit	hit	no	none	3	3	81
bootstrap skills from scratch	seed-skills	indirect	hit	hit	yes	none	3	2	60
export the learning dataset	exporter	indirect	miss	hit	no	both	3	3	77
detect app routes automatically	framework-routes	indirect	hit	hit	yes	none	2	1	35

Methodology & honest caveats

• Task set: 26 queries duplicated from tests/retrieval-golden.test.ts GOLDEN_SET (that file is owned by another agent and is not modified; this list mirrors benchmarks/run-skill-ab.ts). Each task's golden node id (file:src/golden/<module>.ts) contains an expectAny alternative verbatim. The 36 HARD tasks (cross-module blast radius / name disambiguation / indirect-morphological) are authored for this benchmark; their golden node content deliberately shares zero query tokens, so pure retrieval cannot rank the target.
• Success proxy: Top-K = the first 5 ranked anchors of the compressed context package (the retrieval channel); success there means the golden node id is within those 5 anchors (precise id membership — avoids substring false positives from distractor/decoy nodes). For Arm A, the injected hints + episode summaries form an additional channel an agent reads in full; hints/episodes reference modules by name, so injection success is the expectAny substring check. Arm A success = package Top-5 hit or injection hit. The package-only hit rate is reported separately for a like-for-like retrieval comparison.
• Package ranking differs slightly between arms: Arm A's graph additionally holds the history nodes (skills + episodes), which can shift the package top-5 for a few tasks; such differences are visible in the per-task detail (Pkg Top-5 (A) vs Top-5 (B)) and count as real flywheel side effects — the summary's rescued/hurt numbers include them.
• Attribution: for each rescued task, carryingChannel names the channel that actually carried the target; the top similar episode is the first entry of the real findSimilarEpisodes ranking, and similarity is recomputed with the exact formula that ranking uses internally (Jaccard over extractTaskTokens +0.1 for a pass outcome), since scores are not exported. The memory-contribution summary counts each distinct episode that rescued >= 1 task.
• Arm B success is deliberately imperfect: the 27 tasks the OFF arm misses (11 golden + the hard tasks) share zero query tokens with their golden file (realistic: module names are morphologically different from task wording, and symptoms cross module boundaries) — prior episodic experience is the only bridge.
• Both arms run through the real retrieval and learning paths (buildEnhancedContextPackage, applySkillLearning, recordEpisode, suggestSkillHints, findSimilarEpisodes, summarizeEpisodeForPrompt) on isolated in-memory graphs — no mocks, no network, no API key.
• Token counts use gpt-tokenizer (gpt-4o encoding), identical to the token benchmark. Hashing is the project's DJB2a (FNV-class) — fully deterministic within a run; episode ids embed Date.now() so ids differ across runs, but ranking depends on tokens/scores, not ids.
• This measures a mechanical success proxy, not LLM task completion. It validates that episodic memory moves the needle on finding the expected target, quantifies the exact token and wall-clock cost, and exposes which memories earned their keep.