Wingfoil

Wingfoil is a blazingly fast, highly scalable stream processing framework designed for latency-critical use cases such as electronic trading and real-time AI systems.

It ships with a growing library of production-ready I/O adapters covering tick stores, message buses, market protocols, and observability backends — so you can plug graphs into real data sources and sinks with a single line.

Wingfoil simplifies receiving, processing, distributing and monitoring streaming data across your entire stack.

Features

• Fast: Ultra low latency and high throughput with an efficient DAG based execution engine.
• Backtesting: Replay historical data to backtest and optimise strategies.
• Simple and obvious to use: Define your graph of calculations; Wingfoil manages its execution.
• I/O Adapters: production-ready integrations for iceoryx2, KDB+, Kafka, Fluvio, FIX, ZeroMQ, etcd, Prometheus, OpenTelemetry, CSV, and more.
• Multi-language: currently available as a Rust crate, python package and a TypeScript/JavaScript client.
• Graph dynamism: rewire your graph in response to incoming data.
• Async/Tokio: seamless integration, allows you to leverage async at your graph edges.
• Multi-threading: distribute graph execution across cores.

Quick Start

In this example we build a simple, linear pipeline with all nodes ticking in lock-step.

use wingfoil::*;
use std::time::Duration;
fn main() {
    let period = Duration::from_secs(1);
    ticker(period)
        .count()
        .map(|i| format!("hello, world {:}", i))
        .print()
        .run(RunMode::RealTime, RunFor::Duration(period*3)
    );
}

This output is produced:

hello, world 1
hello, world 2
hello, world 3

Order Book Example

Wingfoil lets you easily wire up complex business logic, splitting and recombining streams, and modulating the frequency of data. I/O adapters make it easy to plug in real data sources and sinks. In this example we load a CSV of AAPL limit orders, maintain an order book using the lobster crate, derive trades and two-way prices, and export back to CSV — all in a few lines:

let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);
let (fills, prices) = csv_read("aapl.csv", get_time, true)
    .map(move |chunk| process_orders(chunk, &book))
    .split();
let prices_export = prices
    .filter_value(|price: &Option<TwoWayPrice>| !price.is_none())
    .map(|price| price.unwrap())
    .distinct()
    .csv_write("prices.csv");
let fills_export = fills.csv_write("fills.csv");
Graph::new(vec![prices_export, fills_export], RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)
    .print()
    .run()
    .unwrap();

This output is produced:

Full example.

More Examples

Short code snippets for each adapter live in the examples README. The examples below are all runnable — see each one's README.md for setup and commands.

Core concepts

Example	Description
order_book	Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, export to CSV.
breadth_first	Why wingfoil's BFS execution avoids the O(2^N) node explosion of naive depth-first DAGs.
run_mode	Swap RunMode::RealTime and RunMode::HistoricalFrom with the same graph wiring for backtesting.
async	Integrate Tokio async/await at graph edges (I/O adapters) while keeping the core graph synchronous.
threading	Distribute graph execution across worker threads with producer() / mapper().
dynamic	Add and remove nodes at runtime. Includes demux, dynamic-group, and dynamic-manual variants.
tracing	Instrumentation modes (log, tracing, instruments) for event and span handling.
latency	Per-hop latency stamping with Traced<T, L> and LatencyReport, transported over iceoryx2.

I/O adapters

Example	Description
kdb	KDB+ integration: time-sliced reads, cached reads (LRU file cache), and round-trip write/read/validate.
kafka	Kafka / Redpanda adapter — subscribe, transform, publish pipeline via rdkafka.
fluvio	Fluvio distributed streaming — subscribe, transform, publish pipeline.
fix	FIX 4.4 protocol: self-contained loopback, client, echo server, and live LMAX market data over TLS.
zmq	ZeroMQ pub/sub with direct addressing or etcd-based service discovery.
etcd	etcd key-value store adapter for sub/pub with transformation.
redis	Redis adapter — Pub/Sub channels (subscribe, transform, republish) and persistent Streams (snapshot + tail).
iceoryx2	Zero-copy IPC over shared memory (spin, threaded, signaled polling modes).
aeron	Low-latency Aeron UDP/IPC transport — publish and subscribe to i64 values with spin and threaded polling modes.
web	WebSocket adapter streaming synthetic prices and receiving UI events.
telemetry	Metrics export via Prometheus scraping (pull) and OpenTelemetry OTLP (push).

Links

• Checkout the examples
• Download from crates.io
• Read the documentation
• Review the benchmarks
• Download the wingfoil Python module from pypi.org
• Download the @wingfoil/client browser client from npmjs.com

Get Involved!

We want to hear from you! Especially if you:

• are interested in contributing
• know of a project that wingfoil would be well-suited for
• would like to request a feature or report a bug
• have any feedback

Please do get in touch:

• ping us on discord
• email us at hello@wingfoil.io
• submit an issue
• get involved in the discussion