Effect Validation

July 21, 2026 · View on GitHub

This document tracks small, reproducible KoutenDB effect-validation demos. These are generated local workloads, not universal performance claims.

The goal is to make KoutenDB's main claim testable:

reduce the candidate set that must be scanned;
reduce estimated downstream context tokens;
produce a compact prompt before any LLM is called;
keep the model choice optional and explicit.

Single Demo

examples/effect_validation_demo.sh

The demo generates a deterministic JSONL corpus, imports it into KoutenDB, and compares:

global retrieval across the whole generated corpus;
ring-routed retrieval scoped to docs/japan.

Representative local output:

docs	global scanned	routed scanned	global tokens	routed tokens
168	168	24	692	260

This means the demo reduced scanned records by 85.7% and estimated tokens by 62.4% before sending anything to an LLM.

Matrix Demo

examples/effect_validation_matrix.sh
KOUTEN_EFFECT_LARGE=1 examples/effect_validation_matrix.sh
KOUTEN_EFFECT_QUICK=1 examples/effect_validation_matrix.sh

This matrix is a manual validation path. It is intentionally not part of the default CI smoke suite because the default and large cases are meant to create millions of generated records.

The matrix uses several generated workload shapes:

small-balanced: small corpus with one useful ring and unrelated rings;
near-distractors: adds near-topic distractors to avoid a trivial clean split;
medium-noisy: adds a larger unrelated background corpus;
large-noisy: optional larger generated case enabled by KOUTEN_EFFECT_LARGE=1.

By default, the matrix uses KOUTEN_EFFECT_SCALE=1000, which reaches 13,500,000 documents in the standard run. KOUTEN_EFFECT_LARGE=1 adds a 98,000,000-document generated stress case. Use KOUTEN_EFFECT_QUICK=1 for the smaller fast matrix, or set KOUTEN_EFFECT_SCALE=N explicitly.

Bulk load uses chunked commits. The default matrix uses KOUTEN_EFFECT_BATCH_SIZE=10000; set it explicitly when comparing import behavior:

KOUTEN_EFFECT_SCALE=1000 KOUTEN_EFFECT_BATCH_SIZE=10000 examples/effect_validation_matrix.sh

The current matrix path uses disk-backed storage by default (KOUTEN_EFFECT_DISK_BACKED=1). Set KOUTEN_EFFECT_DISK_BACKED=0 only when you intentionally want to compare the legacy in-memory validation path.

Quick local sanity result from this repository state:

case	docs	global budget	routed budget	set latency us	set us/record	pack latency us	pack records/rings/bytes	scanned	tokens	retrieve latency us	scanned reduction	token reduction	prompt bytes
small-balanced	168	8	3	2250.065	13.393244	987.782	168 / 4 / 61352	168 -> 24	692 -> 260	321.089 -> 55.249	85.714%	62.428%	1356
focused-pinpoint	12045	30	3	166421.501	13.816646	56405.995	12040 / 4 / 4286760	12045 -> 5	2408 -> 260	19804.821 -> 60.418	99.958%	89.203%	1357
focused-small-context	12150	30	5	168488.497	13.867366	57476.127	12150 / 5 / 4330050	12150 -> 50	2595 -> 433	19583.824 -> 102.117	99.588%	83.314%	2064
near-distractors	1860	20	5	28328.164	15.230196	9541.160	1860 / 5 / 681730	1860 -> 120	1730 -> 433	3375.936 -> 221.486	93.548%	74.971%	2064
medium-noisy	13500	30	8	190560.775	14.115613	62926.742	13500 / 5 / 4855950	13500 -> 500	2595 -> 692	22898.644 -> 867.784	96.296%	73.333%	3124

Larger local validation with KOUTEN_EFFECT_SCALE=100 and KOUTEN_EFFECT_BATCH_SIZE=10000:

case	docs	global budget	routed budget	set latency us	set us/record	pack latency us	pack records/rings/bytes	scanned	tokens	retrieve latency us	scanned reduction	token reduction	prompt bytes
small-balanced	16800	8	3	235716.932	14.030770	82063.821	16800 / 4 / 6168360	16800 -> 2400	692 -> 260	27384.316 -> 4239.180	85.714%	62.428%	1360
focused-pinpoint	1204500	30	3	16409354.387	13.623374	6158327.700	1204500 / 5 / 432620950	1204500 -> 500	2595 -> 260	2730205.122 -> 901.451	99.958%	89.981%	1361
focused-small-context	1215000	30	5	16650765.840	13.704334	5555622.287	1215000 / 5 / 436784450	1215000 -> 5000	2595 -> 433	2353292.649 -> 8855.607	99.588%	83.314%	2068
near-distractors	186000	20	5	2529182.669	13.597756	835860.620	186000 / 5 / 68640450	186000 -> 12000	1730 -> 433	325546.670 -> 20660.522	93.548%	74.971%	2068
medium-noisy	1350000	30	8	17598824.700	13.036166	5875386.987	1350000 / 5 / 489914450	1350000 -> 50000	2595 -> 692	2530113.316 -> 86388.774	96.296%	73.333%	3128

The disk-backed path now separates two costs:

set latency: normal JSONL import into the WAL-backed store;
pack latency: an explicit physical-layout step that builds ring-local segment files for faster reads.

The WAL remains the source of truth. Ring segment files are rebuildable read layout, similar in operational role to a compaction or optimize step. They are not required for correctness.

The current segment-pack implementation uses buffered ring-local writes and skips very small rings. Large rings benefit from contiguous segment files, while tiny rings such as users/<id> can stay on WAL-offset reads without paying the cost of one segment file per ring. Explicit post-import packing is now fast enough for the scale-100 validation path, but it is still a visible cost at the largest multi-million stress scale. The effectPackRecords, effectPackRings, and effectPackBytes metrics expose that cost so future pack optimization can be measured directly. KOUTEN_EFFECT_PACK_DURING_IMPORT=1 can be used to test import-time segment construction, but it is not the default because it currently increases normal import latency too much.

Scale-1000 validation also completed on this machine with the current disk-backed segment path:

case	docs	global budget	routed budget	set latency us	set us/record	pack latency us	pack records/rings/bytes	scanned	tokens	retrieve latency us	scanned reduction	token reduction	prompt bytes
small-balanced	168000	8	3	2268312.838	13.501862	808368.180	168000 / 4 / 61929560	168000 -> 24000	692 -> 260	289622.808 -> 42232.274	85.714%	62.428%	1362
near-distractors	1860000	20	5	24940651.107	13.408952	9361622.001	1860000 / 5 / 689944450	1860000 -> 120000	1730 -> 433	3481719.378 -> 227453.701	93.548%	74.971%	2070
medium-noisy	13500000	30	8	203851078.694	15.100080	72425799.771	13500000 / 5 / 4955874450	13500000 -> 500000	2595 -> 692	30082232.678 -> 1101752.063	96.296%	73.333%	3130

The scale-1000 stress path shows that the generated 13.5M-record case completes without the previous OOM kill. It also shows the next performance target: post-import segment packing is much faster than the earlier unbuffered path, but the largest pack step is still large enough to deserve more optimization before calling the v0.9 line final.

The important part is not that these generated numbers are universal. They show how to test the effect: compare broad retrieval with placement-aware retrieval, then report import latency, scanned records, estimated tokens, retrieval latency, and prompt bytes.

Pinpoint User Read

examples/pinpoint_user_read_bench.sh measures a common application pattern: many users exist, but a request already knows the target user. It compares two layouts with the same logical records:

broad layout: every profile is stored in one users ring and read with an id filter;
local layout: each profile is stored in users/<id> and read by ring.

Run:

KOUTEN_PINPOINT_USERS=100000 examples/pinpoint_user_read_bench.sh

Local disk-backed result with 100,000 users:

layout	set latency us	set us/record	pack latency us	read mode	read latency us
broad users ring	783773.641	7.837736	360499.095	filter id in users	1906854.458
local users/ ring	10297185.434	102.971854	11704.754	read users/ limit=1	70.755
local users/ ring	10297185.434	102.971854	11704.754	read users/ limit=20	77.601

The local layout has a higher write/setup cost because it creates many ring metadata entries. The read path is the important result: once the application places data at users/<id>, the request does not need to scan the broad users ring and can read the target ring directly.

User Bundle Read Compared With PostgreSQL

examples/user_bundle_postgres_bench.sh measures a user-detail page shape rather than a single primary-key lookup. Each user has 20 logical records: profile, addresses, career entries, preferences, and orders. KoutenDB stores them under coordinate-local rings such as users/<id>/profile and users/<id>/orders, then reads the bundle with a narrowed stellar read:

N=100000 READS=500 examples/user_bundle_postgres_bench.sh

PostgreSQL stores the same logical data in normalized indexed tables and is measured in two shapes:

five prepared indexed SELECT statements in one transaction;
one prepared JSON aggregate query over the same indexed tables.

Local disk-backed results:

users	logical records	KoutenDB bundle read us	PostgreSQL 5 SELECT us	PostgreSQL JSON aggregate us
1,000	20,000	213.248	424	246
10,000	200,000	202.668	442	257
100,000	2,000,000	205.799	407	240

The KoutenDB read path stayed roughly flat in this benchmark because the query starts from users/<id> and only visits the requested subrings: profile, addresses, career, preferences, and orders. This is the intended locality model: related records can be retrieved together without turning every request into a broad collection scan or a multi-table join.

The write/setup side is still a performance target. At 100,000 users, KoutenDB inserted 2,000,000 logical records at 33.442073 us/record in this helper. That is good enough for the read-locality validation, but bulk load and many-ring metadata creation should be optimized before treating this shape as a final ingestion design.

Per-Subring Limit And Sort Case Study

The heterogeneous subring bundle benchmark checks a shape that is awkward to express as a plain relational join: one root user plus several related collections where each collection has a different limit and sort order.

Example read shape:

kouten get --ring=users/<id> \
  --subring=profile,addresses,career,preferences,orders,notifications \
  --subring-limit=profile:1,addresses:3,career:2,preferences:1,orders:10,notifications:5 \
  --subring-rsort=orders:time,notifications:time

Local disk-backed result, measured with N=10000 READS=1000 examples/subring_bundle_postgres_bench.sh on 2026-07-21:

users	logical records	KoutenDB subring bundle read us	PostgreSQL 6 SELECT us	PostgreSQL JSON aggregate us
10,000	1,050,000	196.859	515	236

The important distinction is the model. KoutenDB treats the related collections as nearby subrings and applies the bounds at the read boundary. PostgreSQL can still express the shape with indexed limited subqueries and JSON aggregation, but it is no longer a simple join and it pushes more response-shaping work into SQL.

Offline Real-Data Copy

Generated workloads are useful for repeatability, but the next pre-production step is to run the same measurement against copied or exported real data:

KOUTEN_REAL_JSONL=/path/to/corpus.jsonl \
QUERY_RING=docs/japan \
GLOBAL_BUDGET=40 \
ROUTED_BUDGET=10 \
examples/offline_effect_validation.sh

Expected JSONL shape:

{"ring":"docs/japan","body":{"id":"doc-1","title":"...","text":"..."},"embedding":[1.0,0.0,0.0,0.0]}

This keeps the validation offline. It does not require production traffic and it does not call an LLM unless the user separately chooses to run the generated prompt through a model.

Optional Trusted LLM Step

The effect validation does not require a model download. It always writes a prompt first. A trusted local model can be added explicitly:

ollama pull gemma4:e2b
KOUTEN_TRUSTED_LLM_CMD='ollama run gemma4:e2b' examples/effect_validation_demo.sh

Gemma 4 E2B is used as the documented example because it is an official Google Gemma 4 edge-size model available through Ollama.

References:

JMeter Load Smoke

examples/jmeter_load_smoke.sh
KOUTEN_JMETER_THREADS=64 KOUTEN_JMETER_LOOPS=1000 examples/jmeter_load_smoke.sh

This is an optional Apache JMeter load smoke for the koutend TCP listener. It sends concurrent HEALTH requests and writes a JTL result file.

It is deliberately separate from retrieval-locality benchmarks:

JMeter health load checks listener stability and request/response behavior;
effect-validation scripts check retrieval working-set and token reduction.

When JMeter is not installed, the wrapper exits successfully with a clear skip message so the repository remains easy to test in minimal environments.