streamvbyte

July 27, 2026 ยท View on GitHub

StreamVByte is a high-performance integer compression library for encoding and decoding streams of 32-bit integers. It supports multiple architectures and implements two encoding schemes: the standard 1234 scheme (using between 1-4 bytes to represent the integers) suitable for most data, and the alternative 0124 scheme, which is optimized for data with lots of zeroes.

This library is based on two repositories:

The C code from lemire's codebase has been adjusted and transpiled to Go using gocc. Architectures other than amd64 and arm64 use a pure Go implementation based on bmkessler's code.

Features

  • High-speed encoding and decoding of integer streams
  • Support for both signed and unsigned 32-bit integers (signed using zigzag encoding)
  • SIMD support for amd64 (SSE4.1) and arm64 (NEON)
  • Optimized encoding schemes for different data patterns
  • Delta encoding for efficient compression of sequences

Installation

To install the library, use go get:

go get github.com/mhr3/streamvbyte

Usage

Basic Encoding and Decoding

The library provides two encoding schemes via the Scheme type:

  • Scheme1234: Standard scheme using 1-4 bytes per value
  • Scheme0124: Alternative scheme optimized for data with many zeros
package main

import (
    "github.com/mhr3/streamvbyte"
)

func main() {
    // Unsigned integers
    input := []uint32{1, 2, 3, 4, 5}
    
    // Basic encoding with default options
    encoded := streamvbyte.EncodeUint32(input, nil)
    decoded := streamvbyte.DecodeUint32(encoded, len(input), nil)

    // Encoding with specific scheme
    encoded = streamvbyte.EncodeUint32(input, &streamvbyte.EncodeOptions[uint32]{
        Scheme: streamvbyte.Scheme0124,  // Use alternative scheme
    })

    // Signed integers
    signedInput := []int32{-1, -2, -3, -4, -5}

    encoded = streamvbyte.EncodeInt32(signedInput, nil)
    decodedSigned := streamvbyte.DecodeInt32(encoded, len(signedInput), nil)
}

Delta Encoding

Delta encoding is useful for compressing sequences of integers with small differences:

package main

import (
    "github.com/mhr3/streamvbyte"
)

func main() {
    // Unsigned delta encoding
    input := []uint32{100, 101, 102, 103, 104}
    
    encoded := streamvbyte.EncodeDeltaUint32(input, nil)    
    decoded := streamvbyte.DecodeDeltaUint32(encoded, len(input), nil)

    // Signed delta encoding
    signedInput := []int32{-100, -98, -96, -94, -92}
    
    encoded = streamvbyte.EncodeDeltaInt32(signedInput, nil)
    decodedSigned := streamvbyte.DecodeDeltaInt32(encoded, len(signedInput), nil)
}

Buffer Reuse

For better performance, you can reuse buffers across encoding/decoding operations:

func processData(data []uint32) {
    var encBuf []byte
    var decBuf []uint32
    
    for {
        // Reuse the same buffers
        encBuf = streamvbyte.EncodeUint32(data, &streamvbyte.EncodeOptions[uint32]{
            Buffer: encBuf,
        })
        
        decBuf = streamvbyte.DecodeUint32(encBuf, len(data), &streamvbyte.DecodeOptions[uint32]{
            Buffer: decBuf,
        })
    }
}

Benchmarks

The following table shows the benchmark results for different encoding and decoding operations on two different architectures: ARM64 (Graviton 2) and AMD64 (Intel Xeon & AMD EPYC). The results include both accelerated and non-accelerated (noasm tag) versions (median of 3 runs).

OperationCPUPure Go (MB/s)SIMD (MB/s)Speedup
Encode/uint32/stdGraviton 2494.35397.310.9x
Encode/uint32/altGraviton 2510.66263.212.3x
Encode/int32/stdGraviton 2476.84129.78.7x
Encode/int32/altGraviton 2470.44998.410.6x
EncodeDelta/uint32/stdGraviton 2607.74312.37.1x
EncodeDelta/uint32/altGraviton 2756.85142.96.8x
EncodeDelta/int32/stdGraviton 2518.03365.16.5x
EncodeDelta/int32/altGraviton 2565.43987.47.1x
Decode/uint32/stdGraviton 2545.610459.119.2x
Decode/uint32/altGraviton 2554.410334.018.6x
Decode/int32/stdGraviton 2503.77223.914.3x
Decode/int32/altGraviton 2549.27144.913.0x
DecodeDelta/uint32/stdGraviton 21623.86889.64.2x
DecodeDelta/uint32/altGraviton 21792.67248.14.0x
DecodeDelta/int32/stdGraviton 21400.94700.53.4x
DecodeDelta/int32/altGraviton 21529.94889.23.2x
Encode/uint32/stdXeon 8375C565.013640.024.1x
Encode/uint32/altXeon 8375C581.610116.517.4x
Encode/int32/stdXeon 8375C541.810606.019.6x
Encode/int32/altXeon 8375C554.38079.914.6x
EncodeDelta/uint32/stdXeon 8375C739.310812.914.6x
EncodeDelta/uint32/altXeon 8375C923.68255.88.9x
EncodeDelta/int32/stdXeon 8375C619.68885.014.3x
EncodeDelta/int32/altXeon 8375C719.76920.49.6x
Decode/uint32/stdXeon 8375C590.419564.233.1x
Decode/uint32/altXeon 8375C578.618347.331.7x
Decode/int32/stdXeon 8375C577.013161.222.8x
Decode/int32/altXeon 8375C577.513335.323.1x
DecodeDelta/uint32/stdXeon 8375C3580.723219.06.5x
DecodeDelta/uint32/altXeon 8375C3073.314771.34.8x
DecodeDelta/int32/stdXeon 8375C2913.710991.03.8x
DecodeDelta/int32/altXeon 8375C2826.011146.13.9x
Encode/uint32/stdEPYC 7R13672.810561.515.7x
Encode/uint32/altEPYC 7R13680.18225.212.1x
Encode/int32/stdEPYC 7R13631.510292.816.3x
Encode/int32/altEPYC 7R13638.67733.312.1x
EncodeDelta/uint32/stdEPYC 7R13859.010674.712.4x
EncodeDelta/uint32/altEPYC 7R131050.57664.57.3x
EncodeDelta/int32/stdEPYC 7R13717.010156.814.2x
EncodeDelta/int32/altEPYC 7R13763.07329.19.6x
Decode/uint32/stdEPYC 7R13697.721223.530.4x
Decode/uint32/altEPYC 7R13755.122265.529.5x
Decode/int32/stdEPYC 7R13685.917496.925.5x
Decode/int32/altEPYC 7R13738.917809.024.1x
DecodeDelta/uint32/stdEPYC 7R133125.027192.98.7x
DecodeDelta/uint32/altEPYC 7R134031.618448.54.6x
DecodeDelta/int32/stdEPYC 7R132927.014262.04.9x
DecodeDelta/int32/altEPYC 7R133561.214648.74.1x