Benchmarks

semble returns the top-50 ranked chunks. ripgrep+read splits the query into keywords (dropping stopwords and short words), runs rg --fixed-strings --ignore-case for each keyword, then reads matched files in full ranked by how many distinct keywords they contain. Both methods search the same set of file types and ignored directories. Tokens are counted with cl100k_base via tiktoken. A relevant file is "covered" once any retrieved unit overlaps its annotated span.

Languages: three repos per language (nine for Python): bash, C, C++, C#, Elixir, Go, Haskell, Java, JavaScript, Kotlin, Lua, PHP, Python, Ruby, Rust, Scala, Swift, TypeScript, Zig. Repos are pinned by revision in repos.json.

How the benchmark was built: queries and ground-truth relevance labels are generated by Claude Sonnet 4.6. The same model is used as LLM-as-judge to verify label quality.

uv run python -m benchmarks.run_benchmark
uv run python -m benchmarks.run_benchmark --repo fastapi --repo axios
uv run python -m benchmarks.run_benchmark --language python

Full runs write to benchmarks/results/semble-hybrid-<sha12>.json.

uv run python -m benchmarks.speed_benchmark

Writes to benchmarks/results/speed-<sha12>.json.

uv run python -m benchmarks.baselines.ablations
uv run python -m benchmarks.baselines.ablations --mode bm25
uv run python -m benchmarks.baselines.ablations --mode semble-semantic

Needs probe on $PATH (npm install -g @buger/probe).

uv run python -m benchmarks.baselines.probe
uv run python -m benchmarks.baselines.probe --repo fastapi --repo axios

Needs grepai on $PATH and Ollama running with nomic-embed-text pulled:

ollama pull nomic-embed-text

uv run python -m benchmarks.baselines.grepai
uv run python -m benchmarks.baselines.grepai --repo fastapi --repo axios

Large repos take several minutes to index. Use --timeout <seconds> (default 120) for repos with many files:

uv run python -m benchmarks.baselines.grepai --timeout 1800 --output results.json

The --output flag enables resume mode: already-completed repos are skipped on restart.

Needs rg on $PATH (brew install ripgrep / apt install ripgrep).

uv run python -m benchmarks.baselines.ripgrep
uv run python -m benchmarks.baselines.ripgrep --no-fixed-strings

Needs the colgrep binary on $PATH.

uv run python -m benchmarks.baselines.colgrep
uv run python -m benchmarks.baselines.colgrep --repo fastapi --repo axios

Runs with --code-only everywhere except bash repos (bash-it, bats-core, nvm), which use --no-code-only because ColGREP's code filter excludes .sh/.bash files.

Requires the benchmark extra (uv sync --extra benchmark).

uv run python -m benchmarks.baselines.coderankembed
uv run python -m benchmarks.baselines.coderankembed --mode semantic

Requires the benchmark extra (uv sync --extra benchmark) and rg on $PATH.

# Recall vs. token-budget across all queries; plots automatically.
uv run python -m benchmarks.token_efficiency recall
uv run python -m benchmarks.token_efficiency recall --repo fastapi

# Regenerate the plot from a saved recall payload.
uv run python -m benchmarks.token_efficiency plot

Writes benchmarks/results/token-efficiency-<sha12>.json and assets/images/token_efficiency.png.

uv run python -m benchmarks.plot

Writes speed_vs_ndcg_cold.png and speed_vs_ndcg_warm.png to assets/images/.