4. Blobs & the @blob reference protocol

Large time-series fields in the msgpack body MAY be extracted into Parquet payloads and stored in the top-level blobs map. This keeps the msgpack body small and fast to read while preserving lossless round-trip.

Reference protocol

Any position in the msgpack body where a producer has extracted a value into a blob MUST contain a single-key map:

{"@blob": "<blob_key>"}

The reader resolves the reference by looking up <blob_key> in the bundle's top-level blobs map (map[str, bin]) and decoding the Parquet bytes it finds there.

Rules:

• The @blob key MUST be the only key in the placeholder map. Readers MUST reject placeholders that contain additional keys.
• Blob keys are producer-chosen strings. The RECOMMENDED convention is a slash-delimited path that mirrors the position of the field in the body, e.g. studies/ema_cross/vector/runs/0/metrics/equity_curve.parquet.
• The blobs map is authoritative. If a field is present both inline and as a @blob reference, the reference wins.
• Blob keys MUST be unique within a bundle.
• Readers MUST ignore unknown blob keys (i.e. keys not referenced from anywhere in the body). Producers MAY include them for producer-internal use, but SHOULD NOT.

Parquet blob column schema (time-series metrics)

Time-series metrics stored as blobs MUST use a 2-column Parquet file with the following schema:

Column	Type	Semantics
ts	int64	UTC epoch milliseconds, strictly increasing
value	float64	The metric value

• Compression: zstd, level RECOMMENDED 5.
• Row-group size: producer-chosen.
• Nulls: value MAY be null; ts MUST NOT be null.
• Ordering: rows MUST be sorted by ts ascending, strictly increasing (no duplicate timestamps).

Which fields SHOULD be extracted

Producers SHOULD extract any time-series field of shape array[[value, timestamp]] with more than approximately 50 entries to a Parquet blob. For shorter series, inline msgpack storage is more compact due to Parquet's per-file overhead.

Specific fields recommended for extraction are enumerated in 3. Metrics. Extraction is per-field, per-run: a producer MAY freely mix inline and blob-referenced storage within a single bundle.

Reader fallback

Readers MUST handle both storage forms for every time-series field:

• Inline — a msgpack array of [value, iso_timestamp] pairs.
• Referenced — a {"@blob": "<key>"} map that resolves to a Parquet payload with the schema above.

Summary-only reads

Readers MAY support a "summary-only" mode that skips Parquet decoding entirely: each @blob reference is replaced with an empty array, and all scalar metrics remain populated. This is useful for bulk listing / ranking pipelines that don't draw charts. Summary-only mode MUST NOT be presented as a lossless round-trip.

Non-time-series blobs

Future revisions of this specification MAY define blob schemas for non-time-series payloads (e.g. very large Monte-Carlo null distributions, portfolio_snapshots tables). Any such extension MUST be backwards-compatible: unknown blob key patterns MUST be ignored by readers that don't know about them, and any field promoted to a blob MUST leave an inline placeholder using the @blob convention above.