Wingfoil

Wingfoil is a blazingly fast stream processing engine for latency-critical systems: electronic trading, real-time decisioning and streaming ML features.

Wire a graph of calculations once and Wingfoil runs it — interpreted, compiled into a single monomorphized function, or as compiled islands inside an interpreted graph. Backtest it over history, then run it live without changing the wiring.

It ships with production-ready adapters covering tick stores, message buses, market protocols and observability backends, so graphs plug into real data sources and sinks in a line.

9.0 replaces the engine. Coming from 8.x, start with the release notes and the migration guide.

Languages

Wire the graph in Rust or Python — the same engine underneath, the same combinator surface — and stream it to a browser over the web adapter.

	Install	Package	Docs	Source
Rust	cargo add wingfoil	crates.io	docs.rs	crates/wingfoil/
Python	pip install wingfoil	PyPI	readthedocs	crates/wingfoil-python/
TypeScript	npm install @wingfoil/client	npm	js/README.md	js/

Rust is the engine itself — all three Nitro execution tiers, every op and adapter, and #[op] to add your own. Python gets the same graph model, combinators and adapters in the wheel, with nodes written in Python and results out as a pandas frame. TypeScript is a browser client for the web adapter, decoding the wire format with the server's own code compiled to wasm. The Python wheel and the browser client both track the engine version, so one number covers all three registries.

Features

• Fast: ~27 ns of engine overhead per node cycle, from a topologically sorted DAG execution engine that visits each node once per tick.
• Nitro — three execution tiers, one wiring: interpreted, compiled, or a compiled island — all derived from the same definition, so they cannot drift. Compiled runs 4.4×–37× faster.
• Backtesting: replay historical data deterministically off source-driven engine time, then run the identical graph live. Same-instant values ride a single burst — never coalesced, never latest-wins, never dropped, in either mode.
• Adapters: production-ready integrations for KDB+, PostgreSQL, Kafka, Redis, Fluvio, etcd, ZeroMQ, FIX 4.4, iceoryx2, Aeron, WebSocket, Prometheus, OpenTelemetry, CSV, augurs and more — one runnable example each.
• Latency tracing: per-hop wall-clock stamps aggregating into one report, across shared memory and the wire.
• Multi-language: a Rust crate, a Python package and a TypeScript client — one line each to install.
• Graph dynamism: add and remove nodes on a running graph, between cycles.
• Async/Tokio: seamless integration at your graph edges.
• Multi-threading: distribute graph execution across cores, with no locks on the graph execution path.
• Errors are values: every lifecycle function returns a Result; a producer error propagates into the graph and aborts the run with context.
• Extensible without forking: sources, combinators, statistics and adapters are extension traits, and your own ops get interpreted and compiled coverage from #[op].

Quick Start

A simple linear pipeline, with all nodes ticking in lock-step:

use std::time::Duration;
use wingfoil::{RunFor, RunMode};
use wingfoil::prelude::*;

fn main() {
    GraphBuilder::new()
        .ticker(Duration::from_secs(1))
        .count()
        .map(|i| format!("hello, world {i}"))
        .print()
        .build()
        .run(RunMode::RealTime, RunFor::Cycles(3))
        .unwrap();
}

The same graph from Python — run defaults to deterministic historical replay, so this one finishes instantly rather than taking three seconds:

import wingfoil as wf

g = wf.Graph()
(
    g.counter(period_nanos=1_000_000_000)   # tick every second: 1, 2, 3, …
     .map(lambda n: f"hello, world {n}")
     .print()                               # print each value, pass it through
)
g.run(cycles=3)

Either way, this output is produced:

hello, world 1
hello, world 2
hello, world 3

Order Book Example

Wingfoil lets you wire up complex business logic, splitting and recombining streams and modulating the frequency of data. Adapters make it easy to plug in real data sources and sinks. Here we load a CSV of AAPL limit orders, maintain an order book with the lobster crate, derive trades and two-way prices, and export both back to CSV:

let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);

let g = GraphBuilder::new();
let (fills, prices) = csv_read(&g, &source_path, get_time, true, None)?
    .map(move |chunk: &Burst<Message>| process_orders(chunk, &book))
    .split();

let _prices_sink = prices.filter_none().distinct().csv_write(&prices_path)?;
let _fills_sink = fills.csv_write(&fills_path)?;

g.build().run(RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)?;

The frequencies of the inputs and outputs are all different to each other — messages arrive in same-timestamp bursts, the top of book changes less often, and trades are sparser still. This output is produced:

An hour of market data — 91,998 messages — replays in about a tenth of a second. Full example.

Execution tiers

Nitro is the tier system: one wiring function, wrapped in nitro! { fn my_graph(g: &GraphBuilder) -> ... }, expands to a module offering all three tiers:

Tier	Entry point	What it is
Interpreted	fluent chaining directly, or my_graph::interpreted()	One dyn boundary per op; open world — threaded/busy-poll sources, feedback, bursts.
Compiled	my_graph::compiled(run_mode, run_for)	The whole graph monomorphized into one function, state in locals — fastest, static DAGs.
Nested (island)	my_graph::nested(&g, inputs...)	A compiled sub-graph mounted as one node of an interpreted graph — hot core compiled, edges stay open.

Semantics live once, in each op's cycle function — the tiers differ only in how the engine reaches it, so there is no duplicated execution logic behind those three doors. core/dual_mode has the rules governing what a nitro! wiring accepts.

Performance

Read the ratios, not the absolute times: these were captured on shared 4-core cloud VMs, each comparison measured back to back in the same run. Full method, caveats and per-workload tables: benches/README.md.

	Measurement
Engine overhead per node cycle	~27 ns (10×10 graph, 100 nodes, every node ticking every cycle)
Compiled vs interpreted	4.4×–37× faster across eight workloads
Nested island vs interpreted	2.2×–10.2× faster
Interpreted vs the legacy engine	0.56×–0.84× — the port is faster on all eight
vs rxrust / tokio async streams	~79× / ~134× faster at depth 10, and the gap grows with depth

Wingfoil visits every node once per tick, in topological order. Libraries that propagate along one path at a time re-visit shared nodes once per path — so on a branch-and-recombine graph their cost doubles with every level while Wingfoil's stays flat. core/topological_sort explains the mechanism in 40 lines.

Where the engine sits against FPGA, kernel-bypass and GC'd stacks — and what is deliberately not claimed — is in where wingfoil currently sits, which ends with the four projects that move that line. All four are open: see Get Involved.

Examples

44 runnable examples, each in its own directory with a README covering what it teaches, the wiring, and its expected output. Full index: examples/README.md.

If you are new, run these three in order — they cover the whole model between them:

cargo run -p wingfoil --example hello_graph   # wire → build → run
cargo run -p wingfoil --example ema_crossover # fold/join/map/filter at backtest scale
cargo run -p wingfoil --features csv --example order_book

Core concepts

No services, no feature flags — these run with a plain cargo run.

Example	Description
hello_graph	The smallest complete program: wire, build, run.
ema_crossover	A backtest-shaped graph — fold, join, map and filter over a price series.
order_book	Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, write both back out.
run_mode	Swap RunMode::RealTime and RunMode::HistoricalFrom over the same wiring, for backtesting.
dual_mode	One wiring, three execution tiers — interpreted, compiled, and a compiled island — proven to agree.
topological_sort	Why topologically sorted execution avoids the O(2^N) node explosion of naive per-path propagation.
dynamism	Add and remove nodes on a running graph — one price book, four wirings.
feedback	Close a loop between two nodes with feedback — a proportional control loop a plain DAG cannot express.
statistics	Streaming statistics: EWMA, cumulative and rolling mean/variance/std/min/max/median, over sample- and time-based windows.
async	Tokio async/await at the graph's edges, with the core graph staying synchronous.
async_source	An async quote feed driving the graph through an external source.
threading	Distribute graph execution across worker threads, with no locks on the execution path.
spawn	Offload slow work off the graph thread with spawn / spawn_map.
tracing	Observability: the logged debug tap and the engine's own spans.
introspect	Read back the graph you wired — text, Mermaid, DOT, JSON or GML.

Adapters

One directory per adapter, each behind its cargo feature. See each README for the service to start and the command to run.

Example	Description
kdb	KDB+ in three parts: time-sliced reads, LRU-cached reads, and a round-trip write/read/validate.
postgres	PostgreSQL — time-sliced historical reads and streaming writes, round-tripped and asserted to tie out.
kafka	Consume a Kafka topic, transform each record, produce to another.
fluvio	Fluvio — seed a topic, consume it, transform, write to a second topic, from one GraphBuilder.
redis	Redis Pub/Sub end to end: publish, subscribe, transform, republish.
etcd	Watch an etcd key prefix, transform the values, write them back under another.
zmq	ZeroMQ pub/sub, with direct addressing or etcd service discovery.
fix	FIX 4.4 — an acceptor and an initiator in one process, over a loopback session.
iceoryx2	Zero-copy IPC over shared memory, in spin, threaded and signaled polling modes.
aeron	Low-latency Aeron UDP/IPC transport — publish and subscribe over aeron:ipc.
web	Stream a synthetic mid-price to a browser over WebSocket, and take UI events back in.
ws	A reconnecting WebSocket client feeding a graph — the transport half of a venue adapter.
prometheus	Serve GET /metrics in the Prometheus text format for a scraper or Grafana.
otlp	Push stream values to an OpenTelemetry backend over OTLP.
telemetry	The two exporters side by side — pull-based scraping vs push — with a Grafana stack.
csv	Replay a CSV as a deterministic historical burst stream, transform it, write it back. The one to read first — it needs no server.
lines	Line-oriented files in both directions — the smallest complete I/O edge.
augurs	On-graph time-series analysis with Grafana's augurs: forecasting, outliers, changepoints, seasonality, DTW, clustering.

Showcase

Example	Description
latency	A two-process pipeline over iceoryx2 with per-hop stamping and an end-of-run report.
trading_e2e	Browser to live venue and back: WebSocket in, shared memory across processes, FIX/TLS out, with Grafana dashboards over the whole path.

Links

• Explore the examples
• Read the release notes
• Compare the field: stream processing, dataflow and trading frameworks
• Browse the crates
• Read the benchmarks
• Use it from Python: wingfoil-python
• Use it from the browser: @wingfoil/client
• See CONTRIBUTING to build, test and contribute

Get Involved!

Four projects are open, and none of them has landed. Each is separable enough to be carried end to end by one person, and each links to a design rather than starting from a blank page:

Project	What it moves	Where it stands
Core pin — pin the graph thread to an isolated core, with the NUMA and warm-up knobs beside it	Deployment discipline — the dominant end-to-end win in the showcase deployment	#392. A working Linux implementation already sits in examples/showcase/trading_e2e/shared.rs; the job is promoting it into runtime/
Kernel bypass — Onload validation, then a raw ef_vi/DPDK source	Ingress, and the wire-to-trade number the benchmarks currently decline to claim	Items 1 and 7 of the trading roadmap. The first rung needs a Solarflare NIC and a measurement run, not a diff
Project Lightning — compiled graphs generated from procedurally wired ones	Config-driven topologies onto Nitro's compiled tier, where nitro! structurally cannot follow	#726, implemented on #769 — open and unmerged, so none of it is on main yet
Project Metal — FPGA/Verilog emission (RHDL) behind that same front-end	The sub-microsecond class: the graph as gateware, with the backtest as its testbench	#727 — exploratory, gated behind Lightning on a hand-written de-risk spike

The fuller picture, with what each one is worth against measured numbers, is in what moves the line.

We want to hear from you! Especially if you:

• are interested in contributing
• know of a project that Wingfoil would be well-suited for
• would like to request a feature or report a bug
• have any feedback

Please do get in touch:

• ping us on discord
• email us at hello@wingfoil.io
• submit an issue
• get involved in the discussion