v9.0

August 2, 2026 · View on GitHub

Status: Draft (binding contract for the v9.0 refactor). Owners: investing-algorithm-framework core. Scope: Replaces bundle format v2 with v3. Breaks the Backtest Python surface. Adds an iaf migrate-bundles --to v3 migration CLI.

This document is the single source of truth for the v9.0 refactor. Every implementation stage (Stages 2–8 of the v9.0 plan) must conform to what is specified here. Any deviation must come back to this document first.

1. Motivation

Bundle format v2 ships an engine_type field that picks one of two slots in the envelope (vector_runs / event_runs). In practice, one .iafbt file holds the results of one engine.

This is the limit we want to remove:

When we run a vector backtest into out/ and then run an event backtest with the same algorithm_id, we want the second run to merge into the existing bundle — replacing the matching engine's slot and preserving the other engine's slot — not silently overwrite the file.

The cleanest fix is to make every bundle dual-engine by construction: one .iafbt per algorithm_id, two engine slots inside it, either slot independently writable.

Treating this as a major release (v9.0) lets us drop the engine_type / backtest_runs / backtest_summary Python surface entirely and replace it with four canonical fields. The cost is paid in a one-shot migration CLI (iaf migrate-bundles --to v3) and a backward-compat reader for v1/v2 bundles.

2. New domain model: Backtest

2.1 Field set (canonical, v9.0)

@dataclass
class Backtest:
    algorithm_id: str

    # --- engine slots (NEW canonical fields) ----------------------
    vector_runs: List[BacktestRun] = field(default_factory=list)
    vector_summary: Optional[BacktestSummaryMetrics] = None
    event_runs: List[BacktestRun] = field(default_factory=list)
    event_summary: Optional[BacktestSummaryMetrics] = None

    # --- unchanged ------------------------------------------------
    backtest_monte_carlo_tests: List[BacktestMonteCarloTest] = \
        field(default_factory=list)
    metadata: Dict[str, str] = field(default_factory=dict)
    risk_free_rate: Optional[float] = None
    strategy_ids: List[Any] = field(default_factory=list)
    parameters: Dict = field(default_factory=dict)
    tag: Optional[str] = None
    ohlcv: Dict[str, object] = field(default_factory=dict, repr=False)

2.2 Removed (no shims, no deprecation aliases)

  • engine_type: str field
  • backtest_runs: List[BacktestRun] field
  • backtest_summary: BacktestSummaryMetrics field
  • vector_runs / event_runs / vector_metrics / event_metrics as @property accessors over a single underlying list — they are now real, independently-settable fields. The names vector_metrics and event_metrics are dropped in favour of vector_summary / event_summary for symmetry with the legacy backtest_summary naming.

A backtest whose engine slot is empty has vector_runs == [] and vector_summary is None. That is the explicit "no data for this engine" signal — no sentinel, no engine_type.

2.3 Invariants

  1. At least one slot must be populated for a bundle to be meaningful. Empty-empty Backtests are tolerated in-memory but save_bundle() raises OperationalException.
  2. Both slots may be populated simultaneously. Consumers must handle this case; nothing in v9.0 prevents it.
  3. Monte-Carlo tests are shared across engines. A Monte-Carlo test set sits on the bundle, not on an engine. (Future work may split them — out of scope for v9.0.)
  4. metadata, parameters, strategy_ids, risk_free_rate, tag are shared across engines. They describe the algorithm, not the engine.

2.4 Helper / accessor methods

MethodBehaviour in v9.0
get_all_backtest_runs(date_ranges=None)Returns vector_runs + event_runs (vector first, then event). Filter applies to the concatenated list.
get_backtest_run(date_range)Searches vector first, then event. Returns the first match.
get_all_backtest_metrics()Concatenated metrics across both engines (vector first).
get_backtest_summary()Removed. Replaced by get_vector_summary() / get_event_summary(). Callers that need "the summary" must pick an engine.
scalar_summary()Removed for the same reason.
get_runs(engine: str)NEW. engine in {"vector", "event"} → returns the matching list.
get_summary(engine: str)NEW. Returns the matching BacktestSummaryMetrics or None.
engines() -> List[str]NEW. Returns the list of engines with non-empty *_runs, in ["vector", "event"] order.
index_row(bundle_path=None)See §6 — emits one row per populated engine. Signature changes.

2.5 to_dict() shape

{
    "algorithm_id": "...",
    "vector_runs":     [run.to_dict(), ...] or None,
    "vector_summary":  summary.to_dict()    or None,
    "event_runs":      [run.to_dict(), ...] or None,
    "event_summary":   summary.to_dict()    or None,
    "backtest_monte_carlo_tests": [...]     or None,
    "metadata": {...},
    "risk_free_rate": 0.03,
    "strategy_ids": [...],
    "parameters": {...},
    "tag": "...",
}

No engine_type key. No backtest_runs. No backtest_summary.

2.6 from_dict() compatibility

The classmethod accepts three shapes:

  1. v9.0 canonical — reads vector_runs/event_runs/etc. directly. Used by the v3 bundle reader.
  2. Legacy backtest_runs + engine_type (v1/v2 era) — used by iaf migrate-bundles --to v3 and the v2 reader fallback in the bundle module. Routes the legacy list into the matching engine slot:
    • engine_type == "vector" → vector_runs / vector_summary
    • engine_type == "event" → event_runs / event_summary
    • engine_type is None → vector_runs / vector_summary (default — see §2.6.1)
  3. Legacy backtest_runs with no engine_type — same as (2) with engine defaulting to vector.

2.6.1 Default engine for legacy bundles

Legacy v1 bundles and pre-#487 directories never carried engine_type. We default them to vector because:

  • Historically the vector engine was the only engine.
  • The migration CLI tags the resulting v3 bundle's source engine in its metadata["v9_migrated_from"] field so analysts can tell.

A --default-engine event flag on iaf migrate-bundles --to v3 overrides this for repositories that know their legacy bundles came from the event engine.

2.7 merge(other) semantics

def merge(self, other: 'Backtest') -> 'Backtest':
    """Merge two dual-engine bundles, per engine."""
    merged = Backtest(algorithm_id=self.algorithm_id)
    merged.vector_runs = self.vector_runs + other.vector_runs
    merged.event_runs  = self.event_runs  + other.event_runs
    merged.vector_summary = _regenerate(merged.vector_runs)
    merged.event_summary  = _regenerate(merged.event_runs)
    merged.backtest_monte_carlo_tests = (
        self.backtest_monte_carlo_tests
        + other.backtest_monte_carlo_tests
    )
    merged.metadata   = {**self.metadata,   **other.metadata}
    merged.parameters = {**self.parameters, **other.parameters}
    # strategy_ids: union, order-preserving, self first
    merged.strategy_ids = list(dict.fromkeys(
        list(self.strategy_ids) + list(other.strategy_ids)
    ))
    merged.risk_free_rate = self.risk_free_rate \
        if self.risk_free_rate is not None else other.risk_free_rate
    merged.tag = self.tag if self.tag is not None else other.tag
    return merged

Where _regenerate(runs) calls generate_backtest_summary_metrics([r.backtest_metrics for r in runs if r.backtest_metrics]) when the list is non-empty, otherwise returns None.

merge() is commutative on data but not on metadata / parameters (later writes wins). This matches v8's behaviour.

3. Bundle format v3 — on-disk envelope

3.1 Envelope layout

+--------+--------+-----------------------------+
| magic  | uint32 | zstd(level=19)(msgpack body)|
| "IAFB" | LE     |                              |
+--------+--------+-----------------------------+
                                  ↑
                          format_version = 3

Magic stays IAFB. format_version is now 3. Zstd level stays 19 for envelope, 5 for blobs.

3.2 Msgpack body — top-level dict

{
    "format_version": 3,
    "framework_version": "9.0.0",
    "created_at": "<iso8601-utc>",

    "algorithm_id": "...",
    "tag": "...",
    "risk_free_rate": 0.03,
    "strategy_ids": [...],
    "parameters": {...},
    "metadata": {...},

    # --- dual engine slots ---------------------------------------
    "vector": {                       # OMITTED if no vector data
        "runs":    [<run-msgpack>...],
        "summary": <summary-msgpack> | None,
    },
    "event": {                        # OMITTED if no event data
        "runs":    [<run-msgpack>...],
        "summary": <summary-msgpack> | None,
    },

    "monte_carlo_tests": [<mc-msgpack>...],  # shared

    # --- side stores ---------------------------------------------
    "blobs": {
        "vector_runs/<idx>/metrics/<field>.parquet": <bytes>,
        "event_runs/<idx>/metrics/<field>.parquet":  <bytes>,
        ...
    },
    "ohlcv_manifest": {...},          # unchanged from v2
    "ohlcv_payload":  {...},          # unchanged from v2
}

Key changes vs v2:

  • Top-level engine_type removed.
  • backtest_runs / backtest_summary removed.
  • vector_runs / vector_metrics / event_runs / event_metrics top-level keys removed; replaced by nested vector / event dicts.
  • A slot dict (vector or event) is omitted entirely if its runs list would be empty. The reader treats absence as "no data for that engine" and materialises vector_runs = [], vector_summary = None (and similarly for event).

3.3 Blob key namespacing

Heavy-series Parquet blob keys gain an engine prefix:

EngineKey prefix
Vectorvector_runs/<idx>/metrics/<field>.parquet
Eventevent_runs/<idx>/metrics/<field>.parquet

<idx> is the run's position in its engine-local list (i.e. the first vector run is vector_runs/0/..., regardless of how many event runs exist).

The 8 heavy fields are unchanged from v2:

  • equity_curve
  • drawdown_series
  • cumulative_return_series
  • rolling_sharpe_ratio
  • monthly_returns
  • yearly_returns
  • twr_equity_curve
  • twr_drawdown_series

Reference markers inside the msgpack body remain {"@blob": "<full-key>"} — unchanged shape, namespaced key.

3.4 Reader compatibility matrix

Bundle versionv9.0 reader behaviour
v1 (no Parquet blobs)Decode envelope, materialise as Backtest with the engine inferred from legacy engine_type (default vector). Permanently supported.
v2 (single-engine, runs//...)Decode envelope, route into matching engine slot, accept legacy blob keys runs/<idx>/.... Permanently supported.
v3 (dual-engine, vector_runs//... and event_runs//...)Native path.

Writer policy: v9.0 only writes v3. v1 and v2 are read-only.

3.5 Merge-on-save semantics

Backtest.save_bundle(path) becomes idempotent under partial updates:

if target file does not exist:
    write v3 envelope from the in-memory Backtest's two engine slots
else if target is a v3 bundle:
    decode existing envelope (lazy; blobs not eagerly decoded)
    for each engine in {"vector", "event"}:
        if in-memory Backtest has that engine's runs (non-empty):
            replace the on-disk slot + all blobs under that engine
        else:
            preserve the on-disk slot + blobs untouched
    re-encode atomically (write to .tmp, fsync, rename)
else if target is v1 or v2:
    decode existing envelope into a dual-engine Backtest
    merge in-memory engines into the materialised Backtest using the
        "in-memory replaces non-empty engine, preserves empty engine"
        rule above
    write fresh v3 envelope

Atomicity: write to <path>.tmp.<pid>, fsync the temp file, os.replace over the target, fsync the directory. This matches the v2 writer's atomicity guarantees and adds no new failure modes.

Concurrency: Out of scope. Concurrent writes to the same .iafbt are undefined behaviour, same as v2. Consumers needing multi-writer safety should write to distinct paths and merge later.

4. Legacy directory save layout

When Backtest.save(directory) is called (the non-bundle path used by older tooling and tests), the on-disk shape changes from:

<dir>/
  backtest_summary.json
  backtest_runs/<idx>/...

to:

<dir>/
  vector_summary.json        # OMITTED if vector slot empty
  vector_runs/<idx>/...
  event_summary.json         # OMITTED if event slot empty
  event_runs/<idx>/...
  monte_carlo_tests/<idx>/...
  metadata.json              # unchanged

The directory reader keeps reading the old backtest_summary.json

  • backtest_runs/ layout and routes it into the vector slot (with the same --default-engine override semantics as the migration CLI).

5. SQLite index

SqliteBacktestIndex (services/backtest_index/sqlite_index.py) needs:

  1. SCHEMA_VERSION bump to 3.
  2. Primary key change from bundle_path (single) to composite (bundle_path, engine_type). engine_type is no longer optional — it is one of "vector" / "event".
  3. One row per (bundle, populated engine). A dual-engine bundle produces two rows. A vector-only bundle produces one.
  4. number_of_runs, summary_* columns are populated from the engine's slot, not from any cross-engine roll-up.
  5. Schema migration: if the existing DB has SCHEMA_VERSION < 3, drop the table and force a rebuild. Logged as a warning. The index is derived data; rebuild cost is acceptable.

iaf index / iaf index --rebuild walk the directory and call Backtest.open(path, summary_only=True).index_rows() (plural — see §6) per bundle.

6. BacktestIndexRow changes

The row contract is otherwise unchanged, but:

  • engine_type: Optional[str] becomes engine_type: str — required, one of "vector" / "event".
  • number_of_runs is the engine-specific count.
  • summary_metrics is the engine-specific summary.

Backtest.index_row(bundle_path=None) is renamed to index_rows(bundle_path=None) -> List[BacktestIndexRow] and returns one row per populated engine slot. Callers that previously got a single row now get a 1- or 2-element list.

The migration is mechanical for the ~6 production call sites identified in the survey; they all iterate the result.

7. BacktestReport (HTML dashboard)

The user picked option (b): duplicate the per-strategy entries in the report's left-hand nav, one entry per engine.

7.1 Strategy list shape

For a backtest with algorithm_id = "trend_following_v3":

Vector populatedEvent populatedNav entries
✓✓trend_following_v3 (vector), trend_following_v3 (event)
✓✗trend_following_v3 (vector)
✗✓trend_following_v3 (event)

The existing per-strategy tabs (Performance, Trading) stay exactly as they are. Each engine-specific entry shows that engine's runs and that engine's summary metrics.

7.2 Implementation sketch

  • _build_html in app/reporting/backtest_report.py expands its strategy iteration to yield (algorithm_id, engine) pairs.
  • dashboard.js ensureStratPage(stratIdx) is unchanged; the number of strategy indices simply doubles for dual-engine bundles.
  • Display name template: f"{algorithm_id} ({engine})".

7.3 Cross-strategy comparison views

Cross-strategy aggregate pages (correlation matrix, rank tables) treat each (algorithm_id, engine) pair as a distinct strategy. This matches what the SQLite index already does (one row per engine) and keeps the report consistent with the analyst CLI.

8. Migration CLI

iaf migrate-bundles --to v3 <dir> [--default-engine vector|event] \
                                  [--dry-run] [--workers N]

Walks <dir> recursively, opens every .iafbt whose format_version < 3, materialises it as a dual-engine Backtest (populated engine determined by legacy engine_type or the --default-engine override) and re-saves it as v3. Adds metadata["v9_migrated_from"] = <legacy-version> and metadata["v9_migrated_at"] = <iso8601>.

--dry-run lists what would be migrated without writing. --workers N runs the migration in a process pool (default 1). The CLI is idempotent — bundles already at v3 are skipped with an INFO log.

Migration does not delete the legacy engine_type / backtest_runs data from metadata unless it was synthesised during read; what's on disk is what was read, mapped to the new shape and written out.

9. Out of scope for v9.0

The following are explicitly not addressed by v9.0 and remain for future work:

  • Splitting Monte-Carlo tests per engine.
  • Concurrent write safety for .iafbt.
  • Streaming readers (the v2 read path is still fully eager except for the explicit summary_only=True flag).
  • A v3 → v3 differential update protocol (every save is still a full envelope re-encode).
  • Renaming BacktestSummaryMetrics to BacktestSummary.

10. Acceptance criteria for v9.0

Closing the v9.0 release requires:

  1. Backtest dataclass restructured as in §2; all callers updated.
  2. Bundle format v3 implemented per §3; v1 and v2 still readable; merge-on-save behaves per §3.5.
  3. Legacy directory save matches §4.
  4. SQLite index matches §5; iaf index --rebuild works against a directory mixing v1, v2 and v3 bundles.
  5. BacktestReport renders dual-engine bundles per §7.
  6. iaf migrate-bundles --to v3 round-trips v1, v2 and v3 bundles correctly on a fixture corpus.
  7. Tests under tests/scenarios/, tests/domain/backtests/, tests/services/, tests/infrastructure/services/, tests/cli/, tests/app/reporting/ and tests/analysis/ updated to the new surface; coverage does not regress.
  8. docs/design/backtest_storage.md updated; new docs/design/bundle-format-v3.md ships; bundle-format-v2.md marked frozen.
  9. CHANGELOG.md calls out the removed Python attributes and points at the migration CLI.

End of document.