serialize.dart

A bitpacking serialization library for Dart. Part of the serialize family, wire compatible with the C++, C, Go, C#, Rust, JavaScript, Java and Elixir libraries — the same values produce the same bytes in every implementation, so a stream written by one reads in any other. STANDARD.md is the authority on every byte. The copy here is vendored verbatim from mas-bandwidth/serialize, along with the shared conformance corpus in conformance/; CI fails if either drifts from upstream.

If this library helps you, please support it

Getting it

serialize.dart ships as source today. The implementation is ready and the package is not yet published: pubspec.yaml names it serialize, and nothing under that name exists on pub.dev. Publishing it there is a separate round.

Depend on it from git, which gives you the same package:serialize/ imports a published package would:

dependencies:
  serialize:
    git:
      url: https://github.com/mas-bandwidth/serialize.dart.git
      ref: v1.1.2

import 'package:serialize/serialize.dart';

Pure Dart, zero dependencies, SDK 3.9 or newer. Pin a release tag as above rather than tracking main. The newest is on the releases page: a release states a format version, and two endpoints interoperate only when they carry the same one.

The surface

The complete family operation set, on three streams sharing one bool-returning serialize surface — WriteStream, ReadStream and MeasureStream — so a single serialize function writes, reads and measures. Values travel in Ref<T> holders (Dart has no by-reference parameters). USAGE.md teaches every operation by example.

• Raw bits: serializeBits (1–32), serializeBits64 (1–64), serializeAlign.
• Ranged integers: serializeInt, serializeInt64, serializeInt128 — offset from min in exactly the bit length of the range, unsigned-domain arithmetic so ranges wider than $2^{63}$/$2^{127}$ are exact, zero bits for a degenerate min == max range on every width.
• Unsigned helpers and bool: serializeUint8 / 16 / 32 / 64, serializeUint128 (the UInt128 pair type), serializeBool.
• Floats: serializeFloat and serializeDouble, bit transparent both ways — every pattern legal, NaN payloads ride a software narrowing/widening that never sets the quiet bit; serializeCompressedFloat, quantizing in float32 with the standard's two roundings on each side.
• Bytes and strings: serializeBytes (aligned bulk copy, count agreed, not transmitted); serializeString (UTF-8 on the wire, payload validated on read in every mode); serializeWideString (one 32-bit group per UTF-16 code unit, no alignment anywhere).
• The relative integer: serializeIntRelative — the flag ladder for strictly increasing sequences, one bit for a difference of 1. The domain is 0 to $2^{31}$ - 1 inclusive, checked on read in every tier.
• Fixed point: serializeFixed at 8/16/32/64-bit storage and serializeFixed128 at 128-bit storage — Q formats, the raw scaled integer as an exact ranged offset, byte identical to serializeInt64 wherever storage fits 64 bits.
• Nested objects: serializeObject over a Serializable — inline composition that contributes no bytes of its own, with no framing, length prefix or alignment inserted around it.
• Range pricing: bitsRequired, bitsRequired64, bitsRequired128, and valueFitsInBits, the write-side bound on a raw bit field.
• 128-bit values: Int128 and UInt128, two's complement pairs of 64-bit halves, mirroring the family's emulated pair types.
• The bitpacker underneath: BitWriter and BitReader, the family wire on Dart's native 64-bit integers. The reader prices its windows inside the buffer — any data length is supported, no slack past the data required.

Pure Dart, zero dependencies.

Quick example

class Player {
  final health = Ref<int>(0);
  final alive = Ref<bool>(false);
  final heading = Ref<double>(0);
}

bool serializePlayer(BitStream stream, Player player) {
  return stream.serializeInt(player.health, 0, 100) &&
      stream.serializeBool(player.alive) &&
      stream.serializeFloat(player.heading);
}

final writer = WriteStream(Uint8List(64)); // length a multiple of 8
serializePlayer(writer, player);           // -> true
writer.flush();                            // always flush before touching the bytes
final wire = writer.data();                // 5 bytes: 7 + 1 + 32 bits

final reader = ReadStream(wire);           // any length, no slack required
serializePlayer(reader, decoded);          // -> true

Toolchain

The Dart SDK is pinned per project, not taken from the system: tending/PINS.md records the exact version, download URL and SHA-256. dist/ is gitignored — re-fetch by the pinned URL, verify the hash, and unpack so the SDK sits at dist/dart-sdk-3.13.2/bin/dart. Every command below uses that binary.

Testing

dist/dart-sdk-3.13.2/bin/dart run test/all.dart                    # release shape
dist/dart-sdk-3.13.2/bin/dart --enable-asserts run test/all.dart   # checked shape

The suite runs every vector in conformance/ — the shared corpus vendored from the C++ reference, one file per covered operation — through this reader, writer and measure. The runner discovers the directory rather than naming its files, and a vector whose operation it cannot drive fails rather than being skipped. An accepted vector must yield the stated value and consume the stated bits; a vector marked writer = canonical must re-emit the whole stream byte for byte, flush included; a sequence stating measure_at_least must measure at least that; and a refused vector must refuse, leave the caller's scalar destination untouched, and leave the stream terminal, which is checked by issuing a further read and requiring it to fail, consume no bits and write nothing. Nothing in it is regenerated here, so it can convict this port rather than agree with it.

interop/ takes it further: the CI interop job builds the C++ reference at a pinned release and runs it head to head with this port. Both halves write the same boundary message — every operation the standard defines, at its boundary values — and the files must be byte identical; each then decodes the other's bytes and re-encodes them exactly; both must refuse every truncation of the other's stream; and both run the corpus. The release candidate in this repository exchanges bytes with the reference on every push, so wire compatibility is measured rather than asserted.

Beside it the suite pins the family's golden vectors byte for byte — the golden wire message covering every operation class, the discriminating compressed-float vectors (bit patterns, not tolerances), the string and wide-string pins, every relative-integer tier, and the fixed point shapes at every group count — plus per-primitive unit suites, refusal proofs for hostile input, terminal-failure proofs, and the measure bound. Run it in both assert modes: writer contracts live in asserts, so the two runs cover both the checked and release shapes of the library (see USAGE.md).

Benchmarking for the serialize family lives in mas-bandwidth/schema's data-driven bench, which measures the generated codecs across every language on one corpus.

License

BSD 3-Clause, © Más Bandwidth LLC.