fin-protoc

September 8, 2025 ยท View on GitHub

fin-protoc is a powerful multi-language protocol compiler that transforms PacketDSL definitions into executable code for binary packet serialization and deserialization across six programming languages: Java, Rust, Lua (Wireshark), Go, Python, and C++.

This tool enables developers to define binary communication protocols once and generate consistent, type-safe implementations across multiple platforms, eliminating the tedious and error-prone process of writing protocol codecs manually for each target language.

In modern distributed systems, especially in financial trading, gaming, and network communication, binary protocols are essential for performance-critical applications. However, implementing the same protocol across multiple programming languages presents significant challenges:

  • Code duplication: Writing and maintaining serialization/deserialization logic in multiple languages
  • Consistency risks: Protocol drift between language implementations can cause subtle bugs
  • Development overhead: Protocol changes require updates across all language implementations
  • Testing complexity: Ensuring all implementations behave identically requires extensive cross-language testing

fin-protoc addresses these challenges by providing a single source of truth for protocol definitions that generates optimized, idiomatic code for each target language.

Architecture Overview

The fin-protoc system follows a three-stage compilation pipeline: parsing DSL files using ANTLR-generated components, transforming parse trees into internal models via the visitor pattern, and generating language-specific code through pluggable generators. 2

graph TB
    subgraph "Input Layer"
        A[PacketDSL Files]
    end

    subgraph "Parsing Layer"
        B[ANTLR Lexer]
        C[ANTLR Parser]
        D[Parse Tree]
    end

    subgraph "Model Layer"
        E[Visitor Pattern]
        F[BinaryModel]
    end

    subgraph "Code Generation"
        G[Java Generator]
        H[Rust Generator]
        I[Lua Generator]
        J[Go Generator]
        K[Python Generator]
        L[C++ Generator]
    end

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H
    F --> I
    F --> J
    F --> K
    F --> L

Core Components

Parser Architecture

The parsing system uses ANTLR 4.13.2 to generate lexer and parser components from the PacketDsl.g4 grammar file 3 . The PacketDslLexer handles tokenization with 36 token types, while the PacketDslParser implements 13 parser rules for different grammar constructs 4 .

Visitor Pattern Implementation

The visitor pattern transforms ANTLR parse trees into strongly-typed Go data structures 5 . The PacketDslVisitorImpl processes different context types through specialized methods:

classDiagram
    class PacketDslVisitor {
        <<interface>>
        +VisitPacket()
        +VisitPacketDefinition()
        +VisitFieldDefinition()
        +VisitMatchField()
    }

    class PacketDslVisitorImpl {
        +BinModel BinaryModel
        +VisitPacket()
        +VisitPacketDefinition()
        +VisitFieldDefinition()
        +VisitInerObjectField()
    }

    class BinaryModel {
        +PacketsMap map[string]Packet
        +MetaDataMap map[string]MetaData
        +Options map[string]string
    }

    PacketDslVisitor <|-- PacketDslVisitorImpl
    PacketDslVisitorImpl --> BinaryModel

Multi-Language Code Generation

The code generation system supports six target languages through a unified Generator interface . Each generator produces language-specific serialization code:

LanguageOutputKey Features
JavaBinaryCodec classesNetty ByteBuf integration, JUnit tests
RustBinaryCodec traitsZero-copy serialization, bytes crate
LuaWireshark dissectorsTCP port binding, ProtoField definitions
GoStructs with methodsNative Go serialization
PythonClasses with methodsPython serialization
C++Classes with methodsC++ serialization support

Field Type System

The DSL supports various field types with language-specific mappings:

graph LR
    subgraph "DSL Types"
        A[Primitive Types]
        B[Complex Types]
        C[Match Fields]
    end

    subgraph "Primitive Types"
        D["u8, u16, u32, u64"]
        E["i8, i16, i32, i64"]
        F["string, char[n]"]
    end

    subgraph "Complex Types"
        G["repeat field"]
        H["nested objects"]
        I["metadata fields"]
    end

    subgraph "Match Fields"
        K["key-value matching"]
    end

    A --> D
    A --> E
    A --> F
    B --> G
    B --> H
    B --> I
    C --> K

Usage

The primary interface is through the compile command, which processes DSL files and generates target language code :

# generate code for rust
fin-protoc -f input.dsl -r ./src
# generate code for lua (Wireshark)
fin-protoc -f input.dsl -l ./src
# generate code for java
fin-protoc -f input.dsl -j ./src
# generate code for go
fin-protoc -f input.dsl -g ./src
# generate code for python
fin-protoc -f input.dsl -p ./src
# generate code for c++
fin-protoc -f input.dsl -c ./src

The compilation process:

  1. Parses DSL files using ANTLR-generated components
  2. Transforms parse trees via visitor pattern
  3. Generates language-specific code through appropriate generators
  4. Organizes output files with proper module structure

Applications

The fin-protoc compiler has been applied across multiple language implementations to ensure consistent binary protocol definitions and codec logic:

  • fin-proto

    • A comprehensive financial protocol library
    • Supports SSE, SZSE, and risk protocols
    • Includes Lua dissectors for Wireshark

  • fin-proto-rs

    • High-performance binary codec in Rust
    • Zero-copy serialization/deserialization
    • Supports SSE, SZSE, and risk protocols
    • Includes unit testing infrastructure

  • fin-proto-go

    • Native Go implementation of the protocols
    • Standardized codec interface
    • Modular, exchange-specific architecture
    • This repository has been integrated into the gt-auto repository, an automated testing tool for financial systems(gateway,engine and so on)

  • fin-proto-cpp

    • Efficient C++ implementation
    • Protocol support for SSE, SZSE, risk
    • Optimized serialization logic

  • fin-proto-java

    • Binary protocol codec for Java
    • Netty ByteBuf integration
    • Gradle build system
    • Java 17+ compatible

  • fin-proto-py

    • Python implementation for financial protocols
    • SSE, SZSE, and risk protocol support
    • Easy-to-use parsing and serialization API

  • fin-proto-vscdoe

    • Visual Studio Code extension for protocol development
    • Intuitive syntax highlighting
    • Code formatting
    • Code completion
    • Error reporting

  • fin-proto-plugin

    • intellij idea plugin for fin-proto
    • integration with fin-protoc .so file

Together, these projects demonstrate how fin-protoc enables protocol definitions to be shared and consistently executed across different ecosystems.

Notes

The codebase demonstrates a well-structured compiler architecture with clear separation between parsing, model transformation, and code generation phases. The ANTLR integration provides robust grammar processing, while the visitor pattern enables clean transformation logic. The multi-language generator system allows consistent code generation across diverse target platforms.

Ask DeepWiki