Wirehair

Fast and Portable Fountain Codes in C

Wirehair produces a stream of error correction blocks from a data source using an erasure code. When enough of these blocks are received, the original data can be recovered.

Packet equations are versioned separately from the C ABI. Applications that persist or exchange legacy packets across builds should use the explicit wire profile APIs and follow LEGACY_WIRE_PROFILES.md. The public V2 API instead emits a canonical endian-stable descriptor documented in V2_WIRE_PROFILE.md; a decoder is created from that descriptor alone. Raw packets and profile IDs do not authenticate either their contract or recovered payload.

As compared to other similar libraries, an unlimited number of error correction blocks can be produced, and much larger block counts are supported. Furthermore, it gets slower as O(N) in the amount of input data rather than O(N Log N) like the Leopard block code or O(N^2) like the Fecal fountain code, so it is well-suited for large data.

This is not an ideal MDS code, so sometimes it will fail to recover N original data packets from N symbol packets. It may take N + 1 or N + 2 or more. On average it takes about N + 0.02 packets to recover. Overall the overhead from the code inefficiency is low, compared to LDPC and many other fountain codes.

Legacy decoders are safe to use with datagram retransmission: the first accepted payload for a packet ID wins, identical duplicates are idempotent, and conflicting duplicates are rejected. Identity tracking is bounded to N + 1024 accepted IDs with no silent eviction; a novel ID beyond that limit returns Wirehair_ExtraInsufficient. Equality uses a fixed-key 128-bit fingerprint (and only meaningful bytes of a partial final block), so it is a duplicate detector rather than authentication. Applications must still verify their own trusted digest or MAC before accepting recovered data.

A simple C API is provided to make it easy to incorporate into existing projects. No external dependencies are required.

Building: Quick Setup

The source code in this folder (gf256 and wirehair code) can be incorporated into your project without any other external dependencies.

To build the software in this repo:

On Windows, make sure CMake and Git Bash are installed. Open up git bash and then:

git clone git@github.com:catid/wirehair.git
cd wirehair
mkdir build
cd build
cmake .. -G "Visual Studio 16 2019"
ls
explorer .

Then you can use Visual Studio Community Edition to open up the wirehair.sln file and build the software.

The default CMake build is portable and produces one static library. Standard CMake configuration and compiler variables are left under caller/toolchain control. Wirehair adds only target-scoped /W4 on MSVC or -Wall -Wextra on other compilers; Release optimization remains CMake's standard policy.

Useful build options are:

• BUILD_SHARED_LIBS=ON selects the shared library instead.
• WIREHAIR_BUILD_BOTH=ON explicitly produces static and shared variants. On Unix-like platforms they share one PIC object compilation.
• The canonical wirehair Python package and compatible whirehair module are installed only by shared or dual builds, because they require a loadable Wirehair dynamic library. Static-only installs deliberately omit them.
• WIREHAIR_STATIC_PIC=OFF disables PIC for a static-only build. It defaults to ON, so the installed archive can be embedded in plugins/shared objects.
• MARCH_NATIVE=ON opts into -march=native after a compiler capability check. Such a build is host-specific and must not be deployed to older or otherwise different CPUs; the default is OFF.
• BUILD_TESTS, BUILD_CODEC_V2, WIREHAIR_BUILD_TOOLS, and WIREHAIR_BUILD_BENCHMARKS control their named developer/test groups. The public V2 C/C++ API remains part of the library when BUILD_CODEC_V2=OFF; that option controls internal V2 tests, fuzz targets, and benchmarks. Offline generators and the V2 benchmark remain available as explicit targets when their options are OFF, but are excluded from the default build.
• WIREHAIR_ENABLE_SCHEDULED_TESTS=ON registers resource-bounded high-N, packet-size, and large-block E2E profiles in addition to the fast tests.
• WH_LTO and WH_PGO_MODE add target-scoped optimization flags without rewriting caller compiler or linker cache variables.

For example, build a generator or benchmark explicitly without adding all offline tools to routine builds:

cmake -S . -B build
cmake --build build --target gen_tables wirehair_v2_bench
./build/gen_tables --no-benchmarks --heavy-trials 0
./build/codec/wirehair_v2_bench compare --nlo 2 --nhi 2 --trials 1 --bb-list 8 --max-message-mib 1 --loss 0

An installed tree exports a relocatable CMake package. Downstream C and C++ projects use the same target for static or shared installs:

find_package(wirehair CONFIG REQUIRED)
target_link_libraries(my_target PRIVATE wirehair::wirehair)

C consumers include <wirehair/wirehair.h>. C++ consumers may use the same C API or include <wirehair/wirehair.hpp> for move-only RAII V2 encoder/decoder wrappers and the fixed-size serialized profile owner.

CMAKE_INSTALL_PREFIX and CMAKE_INSTALL_LIBDIR may both be customized. The imported target supplies include paths, thread linkage, and the Windows DLL/static API definitions; consumers should not define Wirehair export macros manually.

Non-CMake Unix consumers can use the relocatable pkg-config metadata installed beside the library:

cc app.c $(pkg-config --cflags --libs wirehair)
cc app.c $(pkg-config --static --cflags --libs wirehair)

The static query adds Wirehair's private C++ runtime and threading dependencies. Static-only packages include those unavoidable flags in the normal query as well. Binary installs also carry the exact project license at share/licenses/wirehair/LICENSE and both normative wire-profile documents at share/doc/wirehair; the CMake and pkg-config metadata report the project version.

The public shared-library ABI is recorded in the versioned abi/wirehair.map allowlist. Linux shared builds assign those C entry points to the WIREHAIR_2.0 symbol version and hide every unlisted implementation symbol. An intentional public API addition must update that manifest; CI compares it with both the public header and the complete dynamic export table. The same manifest drives the exact Windows DLL and MinGW import-library checks.

Standard Python distribution

The wirehair-fec PEP 517 distribution provides the canonical import wirehair name while preserving import whirehair compatibility:

python -m pip install .
python -m pip install -e .

Its portable py3-none-any wheel contains no native binary. Deploy the Wirehair shared library from the same release separately; major API version 2 alone does not guarantee that an older library exports every required symbol. WIREHAIR_LIBRARY selects an exact file and WIREHAIR_PREFIX selects an exact installation prefix. Both are authoritative and never silently fall back to another copy. Normal CMake- prefix and operating-system loader discovery remain available when neither is set. The complete installation and platform-name contract is in the Python distribution README.

Python reusable output buffers

The Python binding provides Encoder.encode_into, Decoder.recover_into, and Decoder.recover_block_into for packet loops that reuse application storage. Each method accepts a bytearray or writable, C-contiguous memoryview and returns the exact number of bytes written. For example:

packet = bytearray(1200)
written = encoder.encode_into(packet_id, packet)
send(packet_id, memoryview(packet)[:written])

message = bytearray(decoder.message_bytes)
assert decoder.recover_into(message) == len(message)

Repair packets require block_bytes capacity. Systematic packets and recover_block_into require only the original block's exact length, so the final partial block can use a smaller buffer. recover_into requires at least message_bytes, writes exactly that prefix, and preserves any larger-buffer tail. Read-only, released, noncontiguous, undersized, or native-size- overflowing buffers are rejected before the C function is called. The older encode, recover, and recover_block methods remain convenient bytes-returning wrappers over the same checks.

The binding pins the output buffer only for the duration of each call; it can be resized again when the method returns. A ctypes.CDLL foreign call releases the Python GIL, while validation before it and result handling after it hold the GIL. Use independent codec objects for parallel work, or provide your own lock around a shared codec and its output buffer: neither a codec nor a buffer may be mutated concurrently by another thread during a native call.

Legacy encoders materialize their recovery columns during creation. After that succeeds, wirehair_encoder_detach_input() (or Python Encoder.detach_input()) can release an owned message copy or end the lifetime requirement for a borrowed message. The call is idempotent and all later packets remain byte-identical. It trades memory for CPU: systematic packets must then be regenerated from recovery columns instead of copied from the source. Do not race detach with encoding, reuse, conversion, or destruction.

To compare steady-state Python allocation peaks and throughput on a particular interpreter and Wirehair build, run:

python3 python/benchmark_buffers.py --library build/libwirehair.so

The report excludes reusable buffers allocated before tracing and includes the small memoryview/ctypes call scaffolding. The into methods avoid the packet- or message-sized output allocation and copy; exact throughput varies with interpreter, block size, and CPU.

Example Usage

Here's an example program using Wirehair. It's included in the UnitTest project and demonstrates both the sender and receiver, which are normally separate programs. For example the data sender might be a file server and the data receiver might be downloading a file from the sender. ApplicationSha256 and ApplicationDigestMatches below stand for the application's cryptographic implementation; a real receiver obtains the expected digest through authenticated or otherwise trusted metadata, not from the unauthenticated FEC packet stream.

#include <wirehair/wirehair.h>

static bool ReadmeExample()
{
    // Size of packets to produce
    static const int kPacketSize = 1400;

    // Note: Does not need to be an even multiple of packet size or 16 etc
    static const int kMessageBytes = 1000 * 1000 + 333;

    vector<uint8_t> message(kMessageBytes);

    // Fill message contents
    memset(&message[0], 1, message.size());

    // Sender metadata.  A real receiver gets this value through a trusted or
    // authenticated channel, independently of the Wirehair packet equations.
    const auto trustedDigest = ApplicationSha256(message);

    // Create encoder
    WirehairCodec encoder = wirehair_encoder_create(nullptr, &message[0], kMessageBytes, kPacketSize);
    if (!encoder)
    {
        cout << "!!! Failed to create encoder" << endl;
        return false;
    }

    // Create decoder
    WirehairCodec decoder = wirehair_decoder_create(nullptr, kMessageBytes, kPacketSize);
    if (!decoder)
    {
        // Free memory for encoder
        wirehair_free(encoder);

        cout << "!!! Failed to create decoder" << endl;
        return false;
    }

    unsigned blockId = 0, needed = 0;

    for (;;)
    {
        // Select which block to encode.
        // Note: First N blocks are the original data, so it's possible to start
        // sending data while wirehair_encoder_create() is getting started.
        blockId++;

        // Simulate 10% packetloss
        if (blockId % 10 == 0) {
            continue;
        }

        // Keep track of how many pieces were needed
        ++needed;

        vector<uint8_t> block(kPacketSize);

        // Encode a packet
        uint32_t writeLen = 0;
        WirehairResult encodeResult = wirehair_encode(
            encoder, // Encoder object
            blockId, // ID of block to generate
            &block[0], // Output buffer
            kPacketSize, // Output buffer size
            &writeLen); // Returned block length

        if (encodeResult != Wirehair_Success)
        {
            cout << "wirehair_encode failed: " << encodeResult << endl;
            return false;
        }

        // Attempt decode
        WirehairResult decodeResult = wirehair_decode(
            decoder, // Decoder object
            blockId, // ID of block that was encoded
            &block[0], // Input block
            writeLen); // Block length

        // If decoder returns success:
        if (decodeResult == Wirehair_Success) {
            // Decoder has enough data to recover now
            break;
        }

        if (decodeResult != Wirehair_NeedMore)
        {
            cout << "wirehair_decode failed: " << decodeResult << endl;
            return false;
        }
    }

    vector<uint8_t> decoded(kMessageBytes);

    // Recover original data on decoder side
    WirehairResult decodeResult = wirehair_recover(
        decoder,
        &decoded[0],
        kMessageBytes);

    if (decodeResult != Wirehair_Success)
    {
        cout << "wirehair_recover failed: " << decodeResult << endl;
        return false;
    }

    // FEC success proves only that the equations were solvable.  A corrupt,
    // mixed-profile, or adversarially self-consistent stream can recover the
    // wrong bytes successfully, so authenticate the result before using it.
    if (!ApplicationDigestMatches(decoded, trustedDigest))
    {
        cout << "recovered message failed application digest" << endl;
        return false;
    }

    // Free memory for encoder and decoder
    wirehair_free(encoder);
    wirehair_free(decoder);

    return true;
}

int main()
{
    const WirehairResult initResult = wirehair_init();

    if (initResult != Wirehair_Success)
    {
        SIAMESE_DEBUG_BREAK();
        cout << "!!! Wirehair initialization failed: " << initResult << endl;
        return -1;
    }

    if (!ReadmeExample())
    {
        SIAMESE_DEBUG_BREAK();
        cout << "!!! Example usage failed" << endl;
        return -2;
    }
...

Benchmarks

Some quick comments:

Benchmarks on my PC do not mean a whole lot. Right now it's clocked at 3 GHz and has Turbo Boost on, etc. To run the test yourself just build and run the UnitTest project in Release mode.

For small values of N < 128 or so this is a pretty inefficient codec compared to the Fecal codec. Fecal is also a fountain code but is limited to repairing a small number of failures or small input block count.

For N = 2 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 11 usec (236.364 MBPS)
+ Average wirehair_encode() time: 0 usec (7435.7 MBPS)
+ Average wirehair_decode() time: 2 usec (476.205 MBPS)
+ Average overhead piece count beyond N = 0.0105
+ Average wirehair_recover() time: 0 usec (9319 MBPS)

For N = 4 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 8 usec (650 MBPS)
+ Average wirehair_encode() time: 0 usec (8353.43 MBPS)
+ Average wirehair_decode() time: 1 usec (695.102 MBPS)
+ Average overhead piece count beyond N = 0.0225
+ Average wirehair_recover() time: 0 usec (11219 MBPS)

For N = 8 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 13 usec (800 MBPS)
+ Average wirehair_encode() time: 0 usec (7916.2 MBPS)
+ Average wirehair_decode() time: 1 usec (704.359 MBPS)
+ Average overhead piece count beyond N = 0.0045
+ Average wirehair_recover() time: 1 usec (8973.25 MBPS)

For N = 16 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 27 usec (770.37 MBPS)
+ Average wirehair_encode() time: 0 usec (7993.4 MBPS)
+ Average wirehair_decode() time: 1 usec (707.211 MBPS)
+ Average overhead piece count beyond N = 0.036
+ Average wirehair_recover() time: 2 usec (9116.81 MBPS)

For N = 32 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 41 usec (1014.63 MBPS)
+ Average wirehair_encode() time: 0 usec (7062.93 MBPS)
+ Average wirehair_decode() time: 1 usec (908.097 MBPS)
+ Average overhead piece count beyond N = 0.0195
+ Average wirehair_recover() time: 5 usec (8057.33 MBPS)

For N = 64 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 81 usec (1027.16 MBPS)
+ Average wirehair_encode() time: 0 usec (7159.51 MBPS)
+ Average wirehair_decode() time: 1 usec (1033.95 MBPS)
+ Average overhead piece count beyond N = 0.017
+ Average wirehair_recover() time: 10 usec (7640.74 MBPS)

For N = 128 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 192 usec (866.667 MBPS)
+ Average wirehair_encode() time: 0 usec (5662.07 MBPS)
+ Average wirehair_decode() time: 1 usec (870.14 MBPS)
+ Average overhead piece count beyond N = 0.015
+ Average wirehair_recover() time: 25 usec (6419.38 MBPS)

For N = 256 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 319 usec (1043.26 MBPS)
+ Average wirehair_encode() time: 0 usec (6333.2 MBPS)
+ Average wirehair_decode() time: 1 usec (1018.77 MBPS)
+ Average overhead piece count beyond N = 0.022
+ Average wirehair_recover() time: 50 usec (6602.26 MBPS)

For N = 512 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 670 usec (993.433 MBPS)
+ Average wirehair_encode() time: 0 usec (6483.91 MBPS)
+ Average wirehair_decode() time: 1 usec (1028.85 MBPS)
+ Average overhead piece count beyond N = 0.022
+ Average wirehair_recover() time: 100 usec (6600.1 MBPS)

For N = 1024 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 1697 usec (784.443 MBPS)
+ Average wirehair_encode() time: 0 usec (5309.05 MBPS)
+ Average wirehair_decode() time: 1 usec (671.005 MBPS)
+ Average overhead piece count beyond N = 0.022
+ Average wirehair_recover() time: 207 usec (6404.05 MBPS)

For N = 2048 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 3227 usec (825.039 MBPS)
+ Average wirehair_encode() time: 0 usec (5202.3 MBPS)
+ Average wirehair_decode() time: 1 usec (683.141 MBPS)
+ Average overhead piece count beyond N = 0.021
+ Average wirehair_recover() time: 441 usec (6026.08 MBPS)

For N = 4096 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 7614 usec (699.343 MBPS)
+ Average wirehair_encode() time: 0 usec (4334.08 MBPS)
+ Average wirehair_decode() time: 2 usec (577.674 MBPS)
+ Average overhead piece count beyond N = 0.0215
+ Average wirehair_recover() time: 1208 usec (4405.65 MBPS)

For N = 8192 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 17208 usec (618.875 MBPS)
+ Average wirehair_encode() time: 0 usec (3277.17 MBPS)
+ Average wirehair_decode() time: 2 usec (521.665 MBPS)
+ Average overhead piece count beyond N = 0.075
+ Average wirehair_recover() time: 2916 usec (3651.35 MBPS)

For N = 16384 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 42512 usec (501.016 MBPS)
+ Average wirehair_encode() time: 0 usec (2646.89 MBPS)
+ Average wirehair_decode() time: 2 usec (435.173 MBPS)
+ Average overhead piece count beyond N = 0.015
+ Average wirehair_recover() time: 7282 usec (2924.63 MBPS)

For N = 32768 packets of 1300 bytes:
+ Average wirehair_encoder_create() time: 111287 usec (382.78 MBPS)
+ Average wirehair_encode() time: 0 usec (2378.29 MBPS)
+ Average wirehair_decode() time: 3 usec (342.556 MBPS)
+ Average overhead piece count beyond N = 0.0195
+ Average wirehair_recover() time: 16326 usec (2609.23 MBPS)

Credits

Software by Christopher A. Taylor mrcatid@gmail.com

Please reach out if you need support or would like to collaborate on a project.