Performance architecture and benchmarking

Status: benchmark artifact version 6 covers public signal/audit workflows, the three native kernels, the v0.3 path-independent cost-stress surface, the v0.4 point-in-time as-of path, v0.5 CPCV/PBO/Reality Check/SPA reference paths, the v0.9 integrated strategy-audit workload, and the v0.10-v0.12 factor-diagnostic, event-window, projection, and adapter paths. The measurements establish reproducible baselines; no hardware-independent latency promise is claimed.

Performance is a product requirement, but only measured workloads justify optimization decisions. The Rust migration candidate register maps current observation-scale Python work to native, Polars-first, contract-blocked, and keep-Python dispositions.

Non-negotiable rules

1. No Python loop whose iterations scale with observations in a core analysis path.
2. Cross the Python/Rust boundary at coarse granularity.
3. Preserve lazy or streaming execution until an algorithm truly needs materialization.
4. Favor columnar traversal and contiguous reductions.
5. Use float64 by default; do not inflate or narrow dtypes accidentally.
6. Allocate output and scratch buffers deliberately.
7. Return batches, arrays, or tables rather than per-row Python objects.
8. Benchmark before and after any claimed optimization.

Execution planner

The initial planner is explicit dispatch, not a general query optimizer:

Workload	Preferred path
Small vectorized array	NumPy/reference
Lazy columnar projection/grouping	Polars
Large quant-specific repeated reduction	Rust
Mature statistical distribution/test	SciPy
Large Parquet scan	Polars streaming
Optional local SQL extraction	DuckDB Arrow stream

Thresholds come from benchmark crossover points. They are versioned configuration or internal constants with benchmark references, not guesses embedded throughout code.

Benchmark layers

Rust microbenchmarks

Criterion benchmarks isolate kernels such as:

• grouped rank IC;
• quantile assignment;
• interval purging;
• dependent bootstrap;
• turnover;
• parameter-grid reductions.

Record input distribution, group sizes, null/tie characteristics, and thread count. A single uniformly random array is not representative of panel finance data.

The implemented Criterion suite covers:

• grouped average-rank IC at 10,000 and 100,000 rows with deterministic ties;
• 200 bootstrap mean reductions over 1,000-observation samples;
• interval purging for 100,000 training and 1,000 test intervals.

Build or run it with:

cargo bench --bench kernels --no-run
cargo bench --bench kernels

Criterion is a development-only dependency of lacuna-core. The committed configuration uses ten samples, a 250 ms warm-up, and a one-second measurement window so the suite remains practical while still producing statistical timing evidence.

Python end-to-end benchmarks

Measure the public call, including:

• adapter and schema validation;
• sorting/rechunking/materialization;
• boundary conversion;
• kernel time;
• result construction.

Run equivalent Polars, pandas, NumPy, and Arrow inputs where supported. A fast kernel cannot compensate for an accidental full-data copy at its boundary.

The implemented runner measures forward labels, reference/native IC, quantiles, turnover, decay, reference/native bootstrap, reference/native interval purge, CPCV, PBO, Reality Check, SPA, the complete SignalStudy.audit workflow, a nine-point costs.stress grid, and the point-in-time bias.asof_join reference path. It invokes public APIs so validation, data movement, result construction, combinatorial enumeration, resampling, output checksums, and traced Python memory are included.

The cost benchmark currently supports the NumPy/Polars reference implementation. It does not imply a native cost path: native/reference differential testing becomes mandatory if measurements later justify one.

The v0.5 cases likewise describe reference implementations, not native performance claims. CPCV is measured in input intervals, PBO in enumerated symmetric combinations, and Reality Check/SPA in joint bootstrap replicates. Any later optimized path must reproduce the same checksums in differential tests before it can replace or accompany these baselines.

Artifact v6 adds grouped bucketing with deterministic null attrition, grouped neutralization, multi-lag turnover, explicit diagnostic portfolio projection, availability-anchored event windows, and chunk-preserving factor-panel adaptation. The shared generator records period/instrument scale, group cardinality, deterministic null stride, and chunk construction. These cases expose transfer, sorting, grouping, frame construction, and evidence costs; no new native path is implied.

Artifact v5 adds workflow.standard_audit.strategy. One measured call constructs forward labels, signal IC, purged K-fold evidence, a stationary bootstrap, a nine-scenario cost surface, a point-in-time join, vendor and backtest adapter evidence, and then the standardized strategy audit. Its throughput unit is source panel rows per second. That unit makes runs comparable; it is not the sum of every intermediate row visited by the cross-phase workflow.

lacuna bench --tier smoke --out benchmark.json

# The repository script exposes the same service.
python benches/python/bench_signal.py --tier small --repetitions 5

Use --no-native to isolate reference paths. Output is canonical JSON on stdout unless --out is given; existing artifacts require --overwrite.

Initial v5 integrated profile

The initial 0.9 profiling run used the smoke tier, Python 3.13, Polars 1.44, NumPy 2.5, reference paths, three timed repetitions, and one warm-up on Apple arm64. The integrated case produced the same checksum in every invocation, a 145.8 ms median, about 27,432 source panel rows per second, and a 5,283,858-byte traced Python peak. These values are environment evidence, not release budgets.

cProfile attributed about 78% of the integrated call's instrumented cumulative time to the required nine-scenario cost surface and 16% to the point-in-time join. Signal IC, stationary bootstrap, purged splitting, labels, adapters, and final audit composition each accounted for the remaining single-digit share. Profiling instrumentation magnifies Python canonicalization work, so those percentages are routing evidence rather than production latency estimates.

No production optimization accompanies v5. Inspection found no accidental repeated cross-phase call: cost evidence performs the declared scenario calculation and stable full-input fingerprint, while the as-of join executes once. Changing either persistence identity or calculation semantics for a smoke-only timing would be unjustified. A future optimization must reproduce this checksum at small and medium tiers and show an improvement outside profiler overhead.

Study benchmarks

Complete workflow benchmarks measure shared scans and repeated analyses. They catch regressions caused by recomputing labels, collecting lazy input multiple times, or creating excessive Python objects.

Dataset scales

Maintain deterministic generators for approximate scales:

Tier	Rows	Typical use
Smoke	4 thousand	correctness and runner integration
Small	100 thousand	local iteration and crossover behavior
Medium	5 million	pull-request benchmark subset
Large	50 million	scheduled CI or dedicated runner
XL	250 million	release/performance hardware only

Generators specify instruments, dates, group skew, missingness, ties, horizons, and label overlap. Save generator configuration rather than large proprietary data.

The CLI exposes smoke, small, and medium tiers. Large and XL runs require an explicit custom BenchmarkConfig on dedicated hardware so they cannot be launched accidentally in an ordinary developer loop.

Measurements

Track:

• wall and CPU time;
• throughput in rows or resamples per second;
• peak RSS;
• allocations and bytes allocated where available;
• conversion/copy volume;
• thread count and CPU model;
• library/compiler build profile;
• output equivalence checksum.

The versioned Python artifact records resolved configuration, Python/NumPy/Polars/native versions, platform identity, per-case min/median/max wall time, throughput with units, traced Python peak bytes, process peak RSS when the operating system exposes it, and a SHA-256 checksum of nonvolatile result evidence. The equivalence checksum excludes the backend selector and rounds finite floats to 12 significant digits, a tighter normalization than the declared native/reference tolerances. tracemalloc does not attribute every native allocation; the artifact says so explicitly rather than presenting it as total process memory.

Generated timestamps and timings are volatile. Checksums exclude result creation timestamps and must remain stable across repetitions on the same method/backend. A checksum mismatch fails the same-backend benchmark run because comparing timings for nondeterministic evidence would be meaningless. Cross-backend admission retains both exact checksums but does not misuse checksum equality as a floating-point tolerance test.

Report warm and cold behavior separately when caches or dynamic loading matter.

Native-migration evidence sidecar

The public benchmark-v6 dataclasses and JSON remain unchanged. Admission experiments write a separate private artifact with schema lacuna.native-migration-benchmark, version 1. It records effective case dimensions rather than presenting every workload as the top-level panel row count. Each measured backend includes all raw timings, median absolute deviation, baseline and incremental process RSS, Python-traced peak memory, input/output copy bytes, workspace and legacy-projection bytes, checksum/tolerance results, and configured/observed Polars, BLAS, and native thread counts. The correctness record compares complete evidence trees: mapping keys, value types, sequence lengths and ordering, states, findings, warnings, and signed zero are exact; only finite float values use the explicitly recorded method-specific absolute and relative tolerances. It also records whether the exact checksums matched, how many numerical values were compared, maximum absolute and relative error, and the first mismatch path. Backend provenance is excluded from this semantic comparison just as it is from benchmark-v6 equivalence checksums.

Admission runs create inputs before timing, isolate each case in a child process, execute two warm-ups and seven alternating reference/candidate measurements, and retain a separate instrumented phase-attribution run. The fixed environment uses one BLAS thread, two Polars threads, and one native thread. Normal admission uses the first representative tier whose optimized-reference median is at least 50 ms; medium evidence is reproduced in nightly CI and the release preflight. Large evidence is required only for asymptotic or bounded-memory claims.

A native path is admitted only after exact structural and method-tolerance equivalence and at least one transfer-inclusive result: 1.5 times throughput, 30% lower incremental RSS with no more than 10% latency regression, or bounded completion where the reference exceeds the same declared memory budget. The decision ledger links every result to an exact commit and run. Rejected experiments leave no unused production kernel.

Regression policy

For every pull request and main push, the baseline and candidate small tiers run in the same Ubuntu 24.04 job with fixed Polars and BLAS thread budgets. All legacy cases must remain present and retain their correctness checksum. An unexplained candidate median above 115% of its same-runner baseline blocks packaging when the absolute increase also exceeds the committed 1 ms timing-noise floor. Sub-millisecond changes remain visible in the artifact but are not treated as release evidence from a three-repetition single-call measurement; a repeatedly material micro-operation must first gain a batched benchmark. New cases are measured immediately but become regression-protected legacy cases after they exist in the comparison revision.

Historical artifacts and measurements from different hosted runners remain descriptive. Peak-memory changes require investigation and explanation but are not reduced to a noisy cross-machine hard threshold.

An accepted regression may be appropriate for correctness, stronger validation, or clearer semantics, but it is documented rather than hidden.

For v0.14, native admission additionally requires the result to reproduce in both scheduled CI and the exact-source non-publishing release preflight. Import latency above 115% of the v0.13 same-runner baseline or a core wheel more than 20% larger solely because of an unadmitted boundary experiment blocks that experiment from shipping.

Memory architecture

Prefer:

• streaming group accumulators;
• dictionary-encoded IDs;
• offsets rather than repeated group objects;
• reusable scratch buffers;
• bounded batches of bootstrap replicates;
• one projection of required columns before collection.

Avoid:

• materializing unused columns;
• duplicating an entire frame for naming convenience;
• converting to Python lists;
• allocating an observations × resamples matrix when a streamed reduction suffices;
• returning one Python object per group or row.

The private immutable ResolvedExecutionBudget parses an active Config.memory_limit, checks required fixed output bytes before allocation, and selects the largest batch fitting the remaining budget. With no configured limit, temporary batch workspace targets at most 64 MiB; a smaller public method cap still wins. An explicit limit is never ignored, and failure to fit the fixed output or one batch item raises ConfigurationError before that allocation. Bootstrap is the first consumer; other bounded operations adopt the same resolver as their allocation models stabilize.

Thread budget

Polars and BLAS may each own a thread pool. v0.14 does not claim to control those third-party pools: the resolved budget records the requested Lacuna setting plus observed Polars and BLAS configuration, while Lacuna's own native work stays single-threaded with native_threads=1. Rayon and any coordinated cross-library budget remain deferred.

Benchmark both single-thread correctness baselines and parallel scaling. More threads are not assumed to be faster for small groups or memory-bound scans.

Profiling workflow

1. Reproduce an end-to-end workload.
2. Capture a baseline with output checksum and memory.
3. Profile before changing code.
4. Identify computation, conversion, allocation, or scheduling cost.
5. Make the smallest architectural improvement.
6. Re-run correctness and differential tests.
7. Compare the same benchmark environment.
8. Document the crossover and tradeoff.

Do not merge an optimization whose only evidence is that the implementation “looks faster.”