8. Writer contract

July 26, 2026 ยท View on GitHub

A conforming writer is one whose output meets every MUST-level requirement in this section (and in the sections it references).

Write flow

To write a bundle to path p, a conforming writer MUST:

  1. Construct an in-memory body map conforming to 2. Data model.
  2. Set format_version to the highest version the writer knows how to emit (or, if the caller explicitly opts in, a supported lower version).
  3. Optionally extract eligible time-series fields into Parquet blobs and replace them with @blob references per 4. Blobs.
  4. MessagePack-encode the body with use_bin_type=True.
  5. zstd-compress the encoded body.
  6. Write the envelope to a sibling temp path <p>.tmp:
    • 4 bytes magic OBTF.
    • 4 bytes little-endian uint32 version.
    • The compressed body.
  7. Atomically rename <p>.tmp to p (os.replace on POSIX, MoveFileEx with MOVEFILE_REPLACE_EXISTING on Windows).

Partially-written bundles MUST NOT appear at the final path.

Content requirements

Conforming writers MUST:

  • Set format_version to a value the writer actually knows how to emit.
  • Set algorithm_id to a stable, producer-defined string. SHOULD document the fingerprinting scheme in metadata["algorithm_id_scheme"].
  • Ensure the blobs map contains every key referenced from the body. Dangling @blob references are non-conforming.
  • Ensure no blob key is referenced from more than one position in the body (unique-ownership rule).
  • Sort blob time-series by ts strictly ascending (no duplicate timestamps).
  • Preserve any unknown top-level keys and unknown x-* keys that were present on a bundle they read and are now writing back (round-trip preservation).

Conforming writers SHOULD:

  • Populate metadata["producer"], metadata["producer_version"], and metadata["writer_created_at"] (ISO 8601).
  • Populate EngineSlot.summary when populating EngineSlot.runs.
  • Extract time-series fields with more than ~50 entries into Parquet blobs.
  • Deduplicate Universe values in Study.universes so that runs sharing a universe reference an equal value.

Optional behaviour

  • Float32 quantization for OHLCV blobs. Writers MAY downcast float64 OHLCV columns to float32 before Parquet encoding as a space-saving measure. This is off by default; opt-in behaviour.
  • Compression level. Writers MAY expose the zstd compression level as a caller-visible knob. RECOMMENDED: 7 for archive/share, 3 for hot-write.

Downgrade writes

Writers MAY support writing a bundle at a lower format_version than their maximum, for compatibility with older readers. When downgrading:

  • Fields introduced in a newer version MUST be dropped or converted according to that version's data model.
  • The envelope version MUST match the body's format_version.
  • Round-trip preservation of unknown newer-version fields is NOT REQUIRED on downgrade.