ZXC Compressed File Format (Technical Specification)

July 15, 2026 · View on GitHub

Date: July 2026 Format Version: 7

This document describes the on-disk binary format of a ZXC compressed file. It formalizes the current reference implementation of format version 7.

1. Conventions

  • Byte order: all multi-byte integers are little-endian.
  • Unit: offsets are in bytes, zero-based from the start of each structure.
  • Checksum mode: enabled globally by a flag in the file header.
  • Block model: a file is a sequence of blocks terminated by an EOF block, then a footer.

2. Full File Layout

+----------------------+ 16 bytes
| File Header          |
+----------------------+
| Block #0             |
|  - 8B Block Header   |
|  - Block Payload     |
|  - Optional 4B CRC32 |
+----------------------+
| Block #1             |
|  ...                 |
+----------------------+
| EOF Block            | 8 bytes (type=255, comp_size=0)
+----------------------+
| SEK Block (Optional) | table of contents for random access
+----------------------+
| File Footer          | 12 bytes
+----------------------+

3. File Header (16 bytes)

Offset  Size  Field
0x00    4     Magic Word
0x04    1     Format Version
0x05    1     Chunk Size Code
0x06    1     Flags
0x07    7     Reserved (must be 0)
0x0E    2     Header CRC16

3.1 Field definitions

  • Magic Word (u32): 0x9CB02EF5.
  • Format Version (u8): 7. Any other value is rejected (ZXC_ERROR_BAD_VERSION);
  • Chunk Size Code (u8):
    • The value is an exponent in the range [12, 21]: block_size = 2^code.
      • 12 = 4 KB, 13 = 8 KB, ..., 19 = 512 KB (default), ..., 21 = 2 MB.
    • All other values are rejected (ZXC_ERROR_BAD_BLOCK_SIZE).
    • Valid block sizes are powers of 2 in the range 4 KB – 2 MB.
  • Flags (u8):
    • Bit 7 (0x80): HAS_CHECKSUM.
    • Bit 6 (0x40): HAS_DICTIONARY — a pre-trained dictionary is required for decompression.
    • Bits 0..3: checksum algorithm id (0 = RapidHash-based folding).
    • Bits 4..5: reserved.
  • Reserved / Dictionary ID: 7 bytes.
    • When HAS_DICTIONARY is set: bytes 0x07..0x0A contain a dict_id (u32 LE), a 32-bit hash of the dictionary content. Bytes 0x0B..0x0D remain zero.
    • When HAS_DICTIONARY is clear: all 7 bytes are zero.
  • Header CRC16 (u16): computed with zxc_hash16 on the 16-byte header where bytes 0x0E..0x0F are zeroed.

4. Generic Block Container

Each block starts with a fixed 8-byte block header.

Offset  Size  Field
0x00    1     Block Type
0x01    1     Block Flags
0x02    1     Reserved
0x03    4     Compressed Payload Size (comp_size)
0x07    1     Header CRC8

4.1 Header semantics

  • Block Type:
    • 0 = RAW
    • 1 = GLO
    • 2 = GHI
    • 254 = SEK
    • 255 = EOF
  • Block Flags: currently not used by implementation (written as 0).
  • Reserved: must be 0.
  • comp_size: payload size in bytes (does not include the optional trailing 4-byte block checksum).
  • Header CRC8: zxc_hash8 over the 8-byte header with byte 0x07 forced to zero before hashing.

4.2 Block physical layout

[8B Block Header] + [comp_size bytes payload] + [optional 4B checksum]

When checksums are enabled at file level, each non-EOF block carries one trailing 4-byte checksum of its compressed payload.

5. Block Types and Payload Formats

5.1 RAW block (type=0)

Payload is uncompressed data.

Payload = raw bytes
raw_size = comp_size

No internal sub-header.

5.2 GLO block (type=1)

General LZ-style format with separated streams.

GLO payload layout

+-------------------------------+
| GLO Header (16 bytes)         |
+-------------------------------+
| 4 Section Descriptors (32B)   |
+-------------------------------+
| Literals stream               |
+-------------------------------+
| Tokens stream                 |
+-------------------------------+
| Offsets stream                |
+-------------------------------+
| Extras stream                 |
+-------------------------------+

GLO Header (16 bytes)

Offset  Size  Field
0x00    4     n_sequences (u32)
0x04    4     n_literals (u32)
0x08    1     enc_lit    (0=RAW, 1=RLE, 2=HUFFMAN, 3=HUFFMAN_DICT)
0x09    1     enc_litlen (0=RAW tokens, 2=HUFFMAN tokens; level 7 only)
0x0A    1     enc_mlen   (reserved; match lengths share the token byte)
0x0B    1     enc_off    (0=16-bit offsets, 1=8-bit offsets)
0x0C    4     reserved
``$

### \text{GLO} \text{section} \text{descriptors} (4  \times  8 \text{bytes})

\text{Descriptor} \text{format} (\text{packed} $u64`):
- low 32 bits: compressed size
- high 32 bits: raw size

Section order:
1. Literals
2. Tokens
3. Offsets
4. Extras

### GLO stream content

- **Literals stream**:
  - raw literal bytes if `enc_lit=0`, or
  - RLE tokenized if `enc_lit=1`, or
  - Huffman-coded if `enc_lit=2`
    (see [§ 5.2.1 Huffman literal section](#521-huffman-literal-section)), or
  - Huffman-coded with the dictionary's shared code lengths if
    `enc_lit=3` (dictionary-compressed archives only; same section layout,
    no inline lengths header).
- **Tokens stream**:
  - one byte per sequence: `(LL << 4) | ML`, `LL` and `ML` being 4-bit fields.
  - if `enc_litlen=0` (all levels ≤ 6), these `n_sequences` bytes are stored
    verbatim and the Tokens section's compressed size equals `n_sequences`.
  - if `enc_litlen=2` (level 7 only), the token bytes are Huffman-coded
    over the token alphabet using the exact § 5.2.1 layout (inline 128-byte
    lengths header included); the section's compressed size is the encoded
    payload size and the decoder expands it back to `n_sequences` bytes.
- **Offsets stream**:
  - `n_sequences × 1` byte if `enc_off=1`, else `n_sequences × 2` bytes LE.
  - Values are **biased**: stored value = `actual_offset - 1`. Decoder adds `+ 1`.
  - This makes `offset == 0` impossible by construction (minimum decoded offset = 1).
- **Extras stream**:
  - Prefix-varint overflow values for token saturations:
    - if `LL == 15`, read varint and add to LL
    - if `ML == 15`, read varint and add to ML
  - actual match length is `ML + 5` (minimum match = 5).

### 5.2.1 Huffman literal section

`enc_lit=2` carries a length-limited **canonical Huffman code** over the
literal bytes. The bits are placed on the wire with the **PivCo layout**
(level-ordered Huffman, after
[Giesen 2026](https://fgiesen.wordpress.com/2026/06/21/pivco-huffman-merge-operations/)):
the encoding is ordinary Huffman — same code lengths, same code bits, same
size — and PivCo only reorders those bits, grouping them by TREE LEVEL rather
than by symbol so decoding runs data-parallel list merges instead of a serial
bit chain.

```text
Offset  Size  Field
0x00    128   256 x 4-bit code lengths, packed two-per-byte (low nibble first).
              code_len[i] in [0, 11] (0 means symbol absent).
0x80    var   One run per EMITTING node of the canonical code tree, in BFS
              order (parents before children, left before right). Runs are
              LSB-first within bytes and each run is padded to a byte
              boundary. Emitting nodes and run contents are defined below.

Canonical code. Codes are length-limited at L = 11 bits (levels ≤ 6 never emit codes longer than 8; level 7 / ULTRA may emit up to 11). Symbols are ordered by (code_len, symbol) and assigned consecutive code values in that order, starting at 0 and left-shifting when the length increases — the standard canonical construction. The code tree is the binary trie of these codewords read MSB-first: at depth d, a codeword's bit code_len-1-d selects the left (0) or right (1) child. The Kraft equality (validated below) makes this trie complete: every internal node has exactly two children.

Emitting nodes and flat subtrees. Both sides derive, from the code lengths alone, the set of flat roots: an internal node is a flat root iff

  1. it is not itself inside another flat subtree (BFS order resolves this: parents are classified first), and
  2. every leaf below it sits at the same relative depth D, and
  3. D >= 2.

(Completeness of the trie makes condition 2 imply a perfect binary subtree of 2^D leaves. Every maximal complete subtree of depth D >= 2 is a flat root — a fixed format rule. The reference decoder unpacks the packed codes directly, with SIMD kernels for D in 2..6 and a scalar table lookup for D >= 7, instead of running the level-merge cascade.)

Every internal node that is neither a flat root nor a descendant of one is a bitmap node: its run holds one branch bit per symbol routed through it, in symbol-sequence order (0 = left child, 1 = right child). A flat root's run instead holds D packed bits per symbol routed through it, in symbol sequence order: bit j (bit 0 first) is the branch taken at relative depth j below the flat root. Strict descendants of a flat root emit no run at all — they are skipped in the BFS enumeration.

Derived sizes. There is no stored size field of any kind: the root handles n_literals symbols, the popcount of a bitmap node's bits equals its right child's symbol count, and a flat root consuming c symbols occupies exactly ceil(c*D/8) bytes, so every run length is derived while walking the BFS order once.

Selection is an encoder policy, not a format rule: the reference encoder requires level ≥ 6 and at least 1024 literals, prices every candidate (RAW / RLE / Huffman / shared-table Huffman) as J = size + premium(level) * n_decoded_bytes, and picks the minimum — a space-speed Lagrangian with a per-level decode-time premium.

Decoder validation requirements:

  • Every code length must satisfy code_len[i] ≤ 11.
  • At least one symbol must be present (code_len[i] != 0 for some i).
  • The Kraft sum Σ 2^(11 − code_len[i]) over present symbols must equal 2^11, except for the single-present-symbol degenerate case where exactly one symbol has code_len = 1 and the Kraft sum is 2^10.
  • Every node's run (bitmap or flat) must lie within the section payload.
  • A popcount that routes symbols to an absent child is a corruption error.
  • A failure on any of the above results in ZXC_ERROR_CORRUPT_DATA.

5.2.2 Shared-table Huffman literal section

enc_lit=3 is only valid in archives compressed with a dictionary (HAS_DICTIONARY set): the payload is the same Huffman/PivCo section as § 5.2.1 with the 128-byte lengths header omitted — the code lengths come from the shared literal table carried by the .zxd dictionary (see § 12.4), validated once when the dictionary is attached (same rules as § 5.2.1). Decoders MUST reject enc_lit=3 sections with ZXC_ERROR_DICT_REQUIRED when no dictionary table is attached. The archive's dict_id binds the (content, table) pair, so a matching table is guaranteed present whenever the dictionary check passed.

The shared table is trained on the corpus' post-LZ literal distribution and covers only the symbols seen in training; the encoder falls back to a per-block table (enc_lit=2) or RAW/RLE for any block containing a literal byte without a code.

The level-7 token section reuses the § 5.2.1 layout (enc_litlen=2, with the inline lengths header) over the token byte alphabet.

5.3 GHI block (type=2)

High-throughput LZ format with packed 32-bit sequences.

GHI payload layout

+-------------------------------+
| GHI Header (16 bytes)         |
+-------------------------------+
| 3 Section Descriptors (24B)   |
+-------------------------------+
| Literals stream               |
+-------------------------------+
| Sequences stream (N * 4B)     |
+-------------------------------+
| Extras stream                 |
+-------------------------------+

GHI Header (16 bytes)

Same binary layout as GLO header:

  • n_sequences, n_literals, enc_lit, enc_litlen, enc_mlen, enc_off, reserved.

In practice for GHI:

  • enc_lit = 0 (raw literals)
  • enc_off is metadata (sequence words always store 16-bit offsets)

GHI section descriptors (3 × 8 bytes)

Section order:

  1. Literals
  2. Sequences
  3. Extras

Each descriptor uses the same packed size encoding as GLO (u64: comp32|raw32).

GHI sequence word format (32 bits)

Bits 31..24 : LL (literal length, 8 bits)
Bits 23..16 : ML (match length minus 5, 8 bits)
Bits 15..0  : Offset - 1 (16 bits, biased; decode: stored + 1)

Memory order (little-endian word):

byte0 = offset low
byte1 = offset high
byte2 = ML
byte3 = LL

Overflow rules:

  • if LL == 255, read varint from Extras and add it to LL.
  • if ML == 255, read varint, then add minimum match (+5).
  • otherwise decoded match length is ML + 5.

5.4 EOF block (type=255)

EOF marks end of block stream.

Constraints:

  • block header is present (8 bytes)
  • comp_size must be 0
  • no payload
  • no per-block trailing checksum

Immediately after EOF block header comes the Optional SEK block, followed by the 12-byte file footer.

5.5 SEK block (type=254)

The Seek Table block is an optional block appended between the EOF block and the File Footer. It provides O(1) random-access capabilities by recording the compressed size of every block in the archive. Decompressed sizes and block indices are derived from the file header's block_size (all blocks are block_size except the last, which may be smaller).

Layout of a SEK Block:

  Offset             Size    Field
  0x00               8       Block Header (type=254, comp_size=N*4)
  0x08               4       Block 0 Compressed Size (u32 LE)
  0x0C               4       Block 1 Compressed Size (u32 LE)
  ...                ...     ...
  8 + (N-1)*4        4       Block N-1 Compressed Size (u32 LE)

Backward Detection Strategy:

  1. Read the File Header (first 16 bytes) -> extract block_size.
  2. Read the File Footer (last 12 bytes) -> extract total_decompressed_size.
  3. Derive num_blocks = ceil(total_decompressed_size / block_size).
  4. Calculate seek_block_size = 8 + (N × 4).
  5. Seek backward by seek_block_size bytes from the start of the footer to read the Block Header.
  6. Validate block_type == 254 (SEK) and comp_size == N × 4.

6. Prefix Varint (Extras stream)

ZXC extras use a prefix-length varint.

The length is encoded in unary form in the high bits of the first byte: the number of leading 1 bits, followed by a terminating 0, indicates how many additional payload bytes follow. The scheme generalizes to N bytes (11110xxx = 5, 111110xx = 6, ...), but the current ZXC spec caps the encoding at 3 bytes because no legitimate value exceeds 21 bits (see below).

Encodings used:

  • 0xxxxxxx -> 1 byte total (7 bits payload, value < 128)
  • 10xxxxxx -> 2 bytes total (14 bits, value < 16384)
  • 110xxxxx -> 3 bytes total (21 bits, value < 2 MiB)

Payload bits from the following bytes are concatenated little-endian style (low bits first). Used by GLO/GHI to carry LL/ML overflows beyond token/sequence inline limits.

Value bound: a varint encodes (LL - MASK) or (ML - MASK). Since LL/ML are bounded by ZXC_BLOCK_SIZE_MAX = 2 MiB (2212^{21}), every legitimate varint value is strictly less than 2212^{21} and therefore fits in at most 3 bytes.

Any prefix indicating a length >= 4 bytes (first byte >= 0xE0) is out of spec for this format version: encoders must never emit such a varint, and conforming decoders reject it as corrupt input. This caps the varint surface to the format-defined block size limit and neutralizes integer-overflow attacks in downstream bounds arithmetic. A future version of the format that raises ZXC_BLOCK_SIZE_MAX would also extend the accepted prefix lengths.

7. Checksums and Integrity

7.1 Header checksums

  • File header: 16-bit (zxc_hash16).
  • Block header: 8-bit (zxc_hash8).

These protect metadata/navigation fields.

7.2 Per-block checksum (optional)

When file header has HAS_CHECKSUM=1:

  • each data block appends a 4-byte checksum after payload.
  • checksum input is compressed payload bytes only (not block header).
  • algorithm id currently 0 (RapidHash folded to 32-bit).

7.3 Global stream hash

A rolling global hash is maintained from per-block checksums in stream order:

global = 0
for each data block checksum b:
    global = ((global << 1) | (global >> 31)) XOR b

This value is stored in the file footer (or zeroed when checksum mode is disabled).

Footer is mandatory and placed immediately after EOF block header.

Offset  Size  Field
0x00    8     original_source_size (u64)
0x08    4     global_hash (u32)
  • original_source_size: full uncompressed size of the file.
  • global_hash:
    • valid when checksum mode is active;
    • set to zero when checksum mode is disabled.

9. Decoder Validation Checklist (Practical)

  1. Validate file header magic/version/CRC16.
  2. Parse blocks sequentially:
    • validate block header CRC8,
    • check block bounds using comp_size,
    • if enabled, verify trailing block checksum.
  3. Decode payload according to block type.
  4. On EOF:
    • require comp_size == 0,
    • read footer,
    • compare footer original_source_size with produced output size,
    • if enabled, compare footer global_hash with recomputed rolling hash.

10. Versioning Policy

10.1 Format version field

The format version is a single byte at offset 0x04 of the file header. A conforming decoder MUST reject any file whose version it does not support.

10.2 Version bump criteria

ZXC has no forward compatibility: the set of block types and the meaning of every field are fixed per format version. Any change a decoder must understand — adding a block type, assigning meaning to a reserved field/flag bit, changing an encoding, layout, or the checksum algorithm — requires a version bump.

Change classVersion actionExample
New block type addedVersion bump (decoders reject unknown types)Adding a hypothetical GLR block type
Reserved field/flag bit assigned meaningVersion bumpDefining a reserved flag bit
Existing block encoding changedVersion bumpChanging GLO token layout
Header/footer layout changedVersion bumpResizing the file header
Checksum algorithm changedVersion bumpReplacing RapidHash with Komihash

10.3 Compatibility rules

  • Version compatibility: a decoder accepts only the format version it implements and MUST reject any other version with ZXC_ERROR_BAD_VERSION. Because block-type numbering and payload formats may change between versions, a decoder MUST NOT attempt to interpret an archive whose version byte it does not recognise.
  • Unknown block types: a decoder MUST reject any block whose type is not defined for its format version (ZXC_ERROR_BAD_BLOCK_TYPE). The block-type set is fixed per version; introducing a new type is a version bump (decoders do not skip unknown blocks — silently advancing past untrusted, unrecognised data is unsafe).
  • Reserved fields: all reserved bytes and flag bits MUST be written as zero by encoders. The current decoder tolerates (ignores) non-zero reserved values — they are covered by the header CRC, so accidental corruption is still caught — but assigning a reserved field any meaning is a version bump, never a same-version extension.
  • Defined-but-bounded fields: where only specific values are defined (e.g. the checksum-algorithm id, currently 0 = RapidHash only), the decoder rejects out-of-range values (ZXC_ERROR_BAD_HEADER).

10.4 Minimum conforming decoder

A minimal conforming decoder for version 7 MUST support:

  • File header parsing and CRC16 validation
  • RAW blocks (type 0) - passthrough copy.
  • GLO blocks (type 1) - full LZ decode with extras varint, including Huffman entropy sections (§5.2.1, PivCo layout) with code lengths up to 11 bits.
  • GHI blocks (type 2) - full LZ decode with extras varint.
  • EOF block (type 255) - stream termination.
  • File footer validation (source size check).

Support for checksum verification is RECOMMENDED but not strictly required for a minimal implementation.

11. Error Handling

11.1 Error classes

Decoders MUST detect and handle the following error conditions. The recommended behavior for each class is specified below.

ErrorDetection pointRequired behavior
Bad magicFile header, offset 0x00Reject immediately. Not a ZXC file.
Unsupported versionFile header, offset 0x04Reject immediately. Version not supported.
Header CRC16 mismatchFile header, offset 0x0EReject. Header is corrupt or truncated.
Invalid chunk size codeFile header, offset 0x05Reject. Code outside the valid range [12..21].
Block header CRC8 mismatchBlock header, offset 0x07Reject block. Stream is corrupt.
Unknown block typeBlock header, offset 0x00Skip block using comp_size (see §10.3), or reject.
Block payload truncatedDuring fread of comp_size bytesReject. Unexpected end of stream.
Block checksum mismatchTrailing 4-byte checksumReject block. Payload is corrupt.
EOF block with non-zero comp_sizeEOF block headerReject. Malformed EOF marker.
Footer source size mismatchFile footer, offset 0x00Reject. Output size does not match declared original size.
Footer global hash mismatchFile footer, offset 0x08Reject (if checksum mode active). Integrity failure.
Decompressed output exceeds chunk sizeDuring LZ decodeReject. Corrupt or malicious payload.
Match offset out of boundsDuring LZ copyReject. Offset references data before output start.
Varint exceeds maximum lengthExtras streamReject. Overflow or corrupt extras data.

11.2 Severity levels

  • Fatal: the decoder MUST stop processing and report an error. All errors in the table above are fatal by default.
  • Warning: not currently defined. Future versions may introduce non-fatal conditions (e.g. unknown flag bits set in reserved positions).

11.3 Partial output

When a fatal error occurs mid-stream, the decoder SHOULD:

  1. Stop producing output immediately.
  2. Report the specific error condition (see zxc_error_name in the reference implementation).
  3. Not return partially decompressed data as a valid result.

Buffer-mode decoders MUST return a negative error code. Stream-mode decoders MUST signal the error and cease writing to the output.

11.4 Decoder hardening recommendations

For decoders processing untrusted input (e.g. network data, user uploads):

  • Validate all header checksums before processing payloads.
  • Enforce maximum allocation limits based on comp_size and chunk size code.
  • Reject files where comp_size exceeds zxc_compress_bound(chunk_size).
  • Use bounded memory copies - never trust decoded lengths without cross-checking against output buffer capacity.

12. Pre-Trained Dictionary Support

12.1 Overview

A pre-trained dictionary improves compression ratio on small, similar payloads (e.g. JSON API responses, game assets, structured logs) by prefilling the LZ77 sliding window at the start of each block. The dictionary is an external file (.zxd format) referenced by a 32-bit ID in the ZXC file header.

12.2 Mechanism

The dictionary contains raw byte content (max 64 KB, bounded by the 64 KB LZ sliding window). At compression time, the dictionary is logically prepended to each block's input, seeding the hash tables so the match finder can reference dictionary content immediately. At decompression time, the dictionary is prepended to the output buffer so match copies that reference dictionary bytes resolve naturally via pointer arithmetic.

Since each block is independent, the dictionary prefill happens per-block. This preserves O(1) seekable random-access: load the dictionary once, then decompress any block independently.

12.3 File header encoding

When HAS_DICTIONARY (flag bit 6) is set, the reserved bytes at offsets 0x07..0x0A contain the dict_id (u32 LE). A decoder MUST:

  1. Verify that a dictionary is provided (ZXC_ERROR_DICT_REQUIRED if not).
  2. Verify that the dictionary id matches header.dict_id (ZXC_ERROR_DICT_MISMATCH if not). For a raw in-memory dictionary without a shared table, the id is zxc_dict_id(dict, dict_size, NULL). When a shared literal table is attached, the id also binds the table: id = fold32(hash(table_128_bytes, seed = hash(content))) (i.e. zxc_dict_id(content, size, table)).

Older decoders that do not recognize the HAS_DICTIONARY flag will ignore it (per §10.3: reserved flag bits are ignored). However, blocks compressed with a dictionary contain match offsets that reference dictionary content; decoding without the dictionary produces corrupt output. Per-block and global checksums (when enabled) will detect this corruption.

12.4 Dictionary file format (.zxd)

Dictionaries are stored as standalone .zxd files with the following layout:

Offset  Size  Field
0x00    4     Magic Word (0x9CB0D1C7 LE)
0x04    1     Dictionary format version (currently 1)
0x05    1     Flags (bits 0..3: checksum algorithm id; bits 4..7 reserved)
0x06    2     Content size (u16 LE, max 65535)
0x08    4     dict_id (u32 LE, binds content AND shared table, see below)
0x0C    2     Reserved (0)
0x0E    2     Header CRC16 (zxc_hash16, computed with bytes 0x0C-0x0F zeroed)
0x10    N     Dictionary content (raw bytes)
0x10+N  128   Shared literal Huffman table (256 × 4-bit packed code lengths,
              same layout as the § 5.2.1 code-length header; always present)
  • Magic Word: 0x9CB0D1C7. Allows immediate rejection of non-dictionary files.
  • Version: 1. Decoders reject any other version with ZXC_ERROR_BAD_VERSION.
  • Flags: bits 0..3 carry the checksum algorithm id (0 = RapidHash-based folding), matching the ZXC file header flags; bits 4..7 are reserved (must be 0).
  • Shared literal Huffman table: code lengths for the enc_lit=3 literal sections (§ 5.2.2), trained on the corpus' post-LZ literal distribution.
  • dict_id: fold32(hash(table_128_bytes, seed = hash(content))) — binds the exact (content, table) pair. Must match the dict_id stored in any ZXC file header that references this dictionary.
  • Header CRC16: zxc_hash16 checksum of the 16-byte header with bytes 0x0C..0x0F zeroed before hashing — same method as the ZXC file header.
  • Content: raw bytes that prefill the LZ77 window. Not compressed.

12.5 Dictionary training

The zxc_train_dict() function analyzes a corpus of representative samples to select byte segments that maximize LZ77 match coverage. The most frequently matched segments are placed at the end of the dictionary so they produce the shortest offsets (closest to the block start in the virtual window).

12.6 Naming convention

The .zxd extension is cosmetic — files are identified by the magic word at offset 0x00, never by extension. This is a tooling convention, not a format requirement; it does not affect bytes on the wire. The reference CLI applies it as follows:

  • zxc --train -o <dir>/ <files> writes the trained dictionary as <dir>/dictionary_<dict_id>.zxd, where <dict_id> is the lowercase 8-digit hex of the dictionary id (e.g. dictionary_bc46eec1.zxd). Embedding the id keeps the name unique per dictionary and easy to match against the Dict ID reported by zxc -l. With no -o, the file is written to the current directory; with -o <file> it is written there verbatim.
  • On decompression, a dictionary is not auto-located: an archive that was compressed with a dictionary must be decompressed by passing that dictionary explicitly with -D. Without it, decompression fails with ZXC_ERROR_DICT_REQUIRED (the dict_id in the header is still verified against the supplied dictionary, yielding ZXC_ERROR_DICT_MISMATCH on a mismatch).

13. Summary of Useful Fixed Sizes

  • File header: 16 bytes
  • Block header: 8 bytes
  • Block checksum (optional): 4 bytes
  • GLO header: 16 bytes
  • GHI header: 16 bytes
  • Section descriptor: 8 bytes
  • GLO descriptors total: 32 bytes
  • GHI descriptors total: 24 bytes
  • File footer: 12 bytes
  • Dictionary file header (.zxd): 16 bytes

Magic words — both are little-endian u32 at offset 0x00 and deliberately share the 0x9CB0... family prefix, so check the full value (or the file extension) to tell them apart:

FileMagic (value)On-disk bytes (LE)
ZXC archive (.zxc)0x9CB02EF5F5 2E B0 9C
ZXC dictionary (.zxd)0x9CB0D1C7C7 D1 B0 9C

14. Worked Example (Real Hexdump)

This example was produced with the CLI from a 10-byte input (Hello ZXC\n) using:

zxc -z -C -1 sample.txt

Generated archive size: 58 bytes.

14.1 Full hexdump

00000000: F5 2E B0 9C 07 13 80 00 00 00 00 00 00 00 3E 5D
00000010: 00 00 00 0A 00 00 00 69 48 65 6C 6C 6F 20 5A 58
00000020: 43 0A 90 BB A1 75 FF 00 00 00 00 00 00 02 0A 00
00000030: 00 00 00 00 00 00 90 BB A1 75

14.2 Byte-level decoding

A) File Header (offset 0x00, 16 bytes)

F5 2E B0 9C | 07 | 13 | 80 | 00 00 00 00 00 00 00 | 3E 5D
  • F5 2E B0 9C -> magic word (LE) = 0x9CB02EF5.
  • 07 -> format version 7.
  • 13 -> chunk-size code 19 (exponent encoding: 2^19 = 524288 bytes, i.e. 512 KiB, the default).
  • 80 -> checksum enabled (HAS_CHECKSUM=1, algo id 0).
  • next 7 bytes are reserved zeros.
  • 3E 5D -> header CRC16 (LE value 0x5D3E).

B) Data Block #0 (RAW)

Block header at offset 0x10:

00 | 00 | 00 | 0A 00 00 00 | 69
  • type 00 = RAW.
  • flags 00, reserved 00.
  • comp_size = 0x0000000A = 10 bytes.
  • header CRC8 = 0x69.

Payload at 0x18..0x21 (10 bytes):

48 65 6C 6C 6F 20 5A 58 43 0A

ASCII: Hello ZXC\n.

Trailing block checksum at 0x22..0x25:

90 BB A1 75

LE value: 0x75A1BB90.

C) EOF Block (offset 0x26, 8 bytes)

FF | 00 | 00 | 00 00 00 00 | 02
  • type FF = EOF.
  • comp_size = 0 (mandatory).
  • header CRC8 = 0x02.
0A 00 00 00 00 00 00 00 | 90 BB A1 75
  • original source size = 10 bytes.
  • global hash = 0x75A1BB90.

Since there is exactly one data block, the global hash equals that block checksum:

global0 = 0
global1 = rotl1(global0) XOR block_crc = block_crc

14.3 Structural view with absolute offsets

0x00..0x0F  File Header (16)
0x10..0x17  RAW Block Header (8)
0x18..0x21  RAW Payload (10)
0x22..0x25  RAW Block Checksum (4)
0x26..0x2D  EOF Block Header (8)
0x2E..0x39  File Footer (12)

14.4 Seekable Variant (with Seek Table)

Same 10-byte input (Hello ZXC\n), compressed with seekable mode enabled:

zxc -z -C -1 -S sample.txt

Generated archive size: 70 bytes (12 bytes larger than the non-seekable variant).

Full hexdump

00000000: F5 2E B0 9C 07 13 80 00 00 00 00 00 00 00 3E 5D
00000010: 00 00 00 0A 00 00 00 69 48 65 6C 6C 6F 20 5A 58
00000020: 43 0A 90 BB A1 75 FF 00 00 00 00 00 00 02 FE 00
00000030: 00 04 00 00 00 D2 16 00 00 00 0A 00 00 00 00 00
00000040: 00 00 90 BB A1 75

Byte-level decoding

A) File Header (offset 0x00, 16 bytes) - identical to non-seekable.

B) Data Block #0 (RAW) (offset 0x10, 22 bytes) - identical to non-seekable.

C) EOF Block (offset 0x26, 8 bytes) - identical to non-seekable.

D) SEK Block (offset 0x2E, 12 bytes)

Block header at 0x2E:

FE | 00 | 00 | 04 00 00 00 | D2
  • FE -> type 254 = SEK (Seek Table).
  • flags 00, reserved 00.
  • comp_size = 0x00000004 = 4 bytes (one entry x 4 bytes/entry).
  • header CRC8 = 0xD2.

Seek table entry at 0x36:

16 00 00 00
  • Entry #0: compressed block size = 0x00000016 = 22 bytes. This is the total size of data block #0 including its header (8) + payload (10) + checksum (4) = 22. ✓

E) File Footer (offset 0x3A, 12 bytes)

0A 00 00 00 00 00 00 00 | 90 BB A1 75
  • original source size = 10 bytes.
  • global hash = 0x75A1BB90.

Structural view with absolute offsets

0x00..0x0F  File Header (16)
0x10..0x17  RAW Block Header (8)
0x18..0x21  RAW Payload (10)
0x22..0x25  RAW Block Checksum (4)
0x26..0x2D  EOF Block Header (8)
0x2E..0x35  SEK Block Header (8)    <- seek table
0x36..0x39  SEK Entry #0 (4)        <- comp_size of block #0
0x3A..0x45  File Footer (12)

Compatibility note: The SEK block is inserted between the EOF block and the file footer. The footer always remains the last 12 bytes of the file, so decoders that locate the footer from the end of the file (e.g. src + src_size - 12 for buffer APIs, or fseek(END - 12) for file APIs) work unchanged with seekable archives. However, streaming decoders that read the footer sequentially immediately after the EOF block must be updated to detect and skip the SEK block. In practice, all ZXC decoders since v0.9.0 handle both seekable and non-seekable archives transparently.

15. Worked Example: Dictionary File (.zxd Hexdump)

A minimal dictionary whose content is the 5 ASCII bytes hello. Total file size: 149 bytes (16-byte header + 5-byte content + 128-byte shared Huffman table). This is the on-disk form produced by zxc_dict_save() (see §12.4); the table is always present.

15.1 Full hexdump

00000000: C7 D1 B0 9C 01 00 05 00 23 58 DF 6F 00 00 63 65
00000010: 68 65 6C 6C 6F 00 00 00 00 00 00 00 00 00 00 00
00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000040: 00 00 00 00 00 00 00 20 00 02 00 02 20 00 00 00
00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000090: 00 00 00 00 00

15.2 Byte-level decoding

A) Dictionary Header (offset 0x00, 16 bytes)

C7 D1 B0 9C | 01 | 00 | 05 00 | 17 0F 72 9A | 00 00 | 4A D9
  • C7 D1 B0 9C -> magic word (LE) = 0x9CB0D1C7 (.zxd dictionary).
  • 01 -> dictionary format version 1.
  • 00 -> flags (bits 0..3 = checksum algorithm id 0 = RapidHash; bits 4..7 reserved).
  • 05 00 -> content size (LE) = 5 bytes.
  • 23 58 DF 6F -> dict_id (LE) = 0x6FDF5823. Binds the (content, table) pair (see §12.4) and must match the dict_id stored in the file header of any .zxc archive compressed with this dictionary.
  • 00 00 -> reserved.
  • 63 65 -> header CRC16 (LE) = 0x6563, computed over the 16-byte header with bytes 0x0C..0x0F zeroed (same method as the ZXC file header — the CRC is the last 2 bytes of the header).

B) Dictionary Content (offset 0x10, 5 bytes)

68 65 6C 6C 6F

ASCII: hello. Raw bytes that prefill the LZ77 window — not compressed.

C) Shared Huffman Table (offset 0x15, 128 bytes)

... 20 00 02 00 02 20 ...   (remaining bytes 0x00)

256 × 4-bit code lengths, packed two-per-byte (low nibble first), for the shared literal table (§5.2.2). Symbols absent from the training distribution have length 0; here only the four bytes of hello carry codes (e.g. the nibble at table index 'e'=0x65 gives length 2), so all other entries are zero.

15.3 Structural view with absolute offsets

0x00..0x0F  Dictionary Header (16)
0x10..0x14  Dictionary Content (5)
0x15..0x94  Shared Huffman Table (128)