A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects.
https://manuzhang.github.io/awesome-streaming/ is a more dynamic website where you can find updates of the awesome projects here.
- Aeron
- Reliable UDP unicast, multicast, and IPC message transport.
- Apache Apex
- Unified platform for big data stream and batch processing.
- Apache Flink - Distributed engine for stateful computation over bounded and unbounded data streams.
- Apache Heron - Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.
- Apache Kafka
- Distributed event streaming platform for high-performance data pipelines and applications.
- Apache Pulsar - Distributed pub-sub messaging and event streaming platform.
- Apache RocketMQ - Cloud-native messaging and streaming platform for event-driven applications.
- Apache Samza - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.
- Apache Spark Streaming
- Scalable, fault-tolerant stream processing on Apache Spark.
- Apache Storm - Distributed real-time computation system for reliably processing unbounded streams of data.
- Apache StreamPipes
- Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.
- ArkFlow
- High-performance stream processing engine with pluggable sources, processors, and sinks.
- Arroyo - Distributed stream processing engine for stateful SQL computations over unbounded data.
- AthenaX - Uber's retired SQL-based streaming analytics platform.
- AutoMQ - Diskless Kafka-compatible streaming platform that stores durable data in object storage.
- Bytewax
- Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.
- eKuiper
- Lightweight data stream processing engine for resource-constrained IoT edge devices.
- Esper - Complex event processing, Streaming SQL, and event series analysis engine.
- Fluvio
- Composable, stateful data streaming system with programmable in-line computation.
- Gazette - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.
- Gearpump
- Lightweight real-time distributed streaming engine built on Akka.
- hailstorm
- Distributed stream processing with exactly-once semantics based on Storm.
- Hazelcast Jet - Stream and batch processing engine whose development moved into Hazelcast Platform.
- HStreamDB - Cloud-native streaming database for IoT data storage and real-time processing.
- ksqlDB - Source-available database purpose-built for stream processing applications.
- LaminarDB - Embeddable streaming SQL engine built on Apache Arrow and DataFusion.
- LightSaber
- Multi-core stream processing engine using code generation for window aggregation.
- LogDevice - Facebook's archived distributed storage system for sequential data.
- Maki Nage
- Stream processing framework for data scientists based on Kafka and ReactiveX.
- Mantis - Netflix platform for building real-time, operations-focused stream processing applications.
- Materialize - Incremental SQL engine for maintaining continuously updated views over changing data.
- Metaq - Taobao distributed messaging system.
- mupd8 (Muppet)
- MapReduce-style framework for processing fast-moving data streams.
- NATS Streaming - Deprecated disk-backed messaging system superseded by NATS JetStream.
- NebulaStream - Data management system for cloud-edge-sensor environments.
- NSQ - Real-time distributed messaging platform designed to operate at scale.
- Numaflow
- Kubernetes-native, language-agnostic platform for scalable event-driven applications.
- Onyx
- Distributed, masterless, fault-tolerant data processing platform.
- PipelineDB
- Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.
- Proton - Unified streaming and historical data analytics database powered by ClickHouse.
- Redpanda - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.
- RisingWave - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.
- S4 - Retired distributed platform for processing continuous unbounded data streams.
- SABER
- Window-based hybrid CPU/GPU stream processing engine.
- Scramjet Cloud Platform
- Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.
- SensorBee - Discontinued lightweight stream processing engine for IoT.
- Siddhi - Streaming SQL and complex event processing engine.
- SPQR - Dynamic framework for processing high-volume data streams through pipelines.
- Squall - Online SQL query processing engine built on Apache Storm.
- StreamCQL - Storm-based continuous query language for real-time computation.
- Teknek - Stream processing platform with the interactive Stream Operator Language shell.
- Tigon
- High-throughput real-time stream processing framework built on Hadoop and HBase.
- Trill
- Single-node query processor for temporal and streaming data from Microsoft Research.
- Wally
- Distributed stream processing framework formerly named Wallaroo.
- Wingfoil
- Graph-based stream processing engine for latency-critical systems.
- Akka - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.
- Akka Streams - Reactive Streams implementation built on Akka actors.
- Apache Beam
- Unified programming model and language-specific SDKs for batch and streaming data processing.
- Apache Edgent - Retired programming model and runtime for streaming analytics on gateways and edge devices.
- Apache Kafka Streams - Stream processing library included with Apache Kafka.
- Apache Pekko - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.
- Apache SAMOA - Retired distributed streaming machine learning framework.
- Apache StormCrawler - Scalable web crawler SDK based on Apache Storm.
- CapyMOA
- Machine learning library for data streams with a Python API and MOA backend.
- coast - DSL that builds exactly-once dataflow graphs on top of Apache Samza.
- Daggy - Utility and developer library for aggregating and capturing data streams.
- DataSketches - Apache library of stochastic streaming algorithms known as sketches.
- FastStream - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.
- Faust - Deprecated Python stream processing library inspired by Kafka Streams.
- FS2 - Compositional streaming I/O library for Scala.
- Jubatus - Distributed processing framework and library for online machine learning.
- Kzmlabs StateFun Actors - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.
- MediaPipe
- Cross-platform, customizable ML solutions for live and streaming media.
- Monix - High-performance Scala and Scala.js library for asynchronous and event-based programs.
- Numalogic - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.
- Pulsar - Actor-based event-driven concurrency framework for Python.
- Quix Streams - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.
- River - Online machine learning library for Python.
- samza-luwak - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.
- Scramjet C++ - Reactive stream programming framework for C++.
- Scramjet Python - Reactive stream programming framework for Python.
- Scramjet TypeScript
- Reactive stream programming framework for asynchronous JavaScript and TypeScript applications.
- StormCV - Computer vision operations and data models for Apache Storm.
- Stream Ops - Embeddable data streaming engine and stream processing API for Java.
- streamDM - Huawei library for mining data streams with Spark Streaming.
- Streamiz.Kafka.Net - .NET stream processing library for Apache Kafka.
- Streamparse - Python API, command-line tools, and topology DSL for Apache Storm.
- Streamz - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.
- Summingbird - Retired library for expressing streaming MapReduce computations over Storm and Scalding.
- Swave - Lightweight Reactive Streams infrastructure toolkit for Scala.
- Tributary - Python library for constructing synchronous, reactive, and lazy dataflow graphs.
- trident-ml - Online machine learning library based on Apache Storm Trident.
- WindFlow - Data stream processing parallel library for multi-core CPUs and GPUs.
- Yurita - PayPal anomaly detection framework built on Spark Structured Streaming.
- Apache Flume - Distributed service for collecting, aggregating, and moving large amounts of log-like data.
- Brooklin - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.
- Bruin - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.
- Camus - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.
- CocoIndex
- Incremental data transformation engine for continuously updated AI and agent workloads.
- Databus - LinkedIn source-agnostic distributed change data capture system.
- faucet-stream - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.
- inGestr
- Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.
- Pathway - Python ETL framework for stream processing, real-time analytics, and AI data pipelines.
- Redpanda Connect - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.
- RudderStack - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.
- Suro - Netflix data pipeline for collecting, aggregating, and dispatching application events.
- yasdb - Durable Streams protocol server backed by SlateDB object storage, with offset-based catch-up reads and SSE or long-poll tailing.
- beava - Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.
- Eventum
- Data generation platform for producing synthetic event streams.
- javactrl-kafka - Code-first distributed workflow engine for microservice orchestration and business process automation.
- Nussknacker - Visual tool for defining and running real-time decision algorithms.
- straw - Platform for real-time streaming search.
- StreamAlert - Airbnb serverless framework for real-time security log analysis and alerting.
- Streamdal - Code-native data privacy controls for detecting PII in application data flows.
- StreamFlow - Tool for building and monitoring stream processing workflows.
- StreamingBandit - Web application for setting up and evaluating contextual multi-armed bandit experiments.
- Streamline - Hortonworks visual stream analytics framework built around engines such as Apache Storm.
- Substation - Toolkit for routing, normalizing, and enriching security event and audit logs.
- Turbine - Netflix tool for aggregating Server-Sent Event JSON streams.
- Zilla - Multi-protocol gateway for connecting applications, APIs, agents, and devices to event streams.
- Amazon Kinesis Data Streams - Fully managed service for ingesting and processing real-time data streams on AWS.
- Azure Stream Analytics
- Fully managed service for serverless real-time analytics in the cloud and at the edge.
- Concord - Historical distributed stream processing framework built on Apache Mesos.
- Google Cloud Dataflow - Fully managed service for running Apache Beam batch and streaming pipelines.
- IBM Streams
- Discontinued proprietary platform for distributed stream processing and real-time analytics.
- NVIDIA DeepStream SDK
- GStreamer-based toolkit with open-source components and proprietary NVIDIA libraries for real-time AI streaming analytics and multi-sensor processing.
- Flotilla - Automated message queue orchestration for scaled-up benchmarking.
- storm-perf-test - Apache Storm performance and stress test.
- streaming-benchmarks - Benchmarks for low-latency stream processing systems including Storm, Spark, and Flink.
- In-Stream Big Data Processing
- The world beyond batch: Streaming 101 by Tyler Akidau.
- Real Time Analytics: Algorithms and Systems (VLDB 2015)
- Grokking Streaming Systems by Josh Fischer & Ning Wang
- Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Reuven Lax, Slava Chernyak, and Tyler Akidau
- Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter
- MillWheel: Fault-Tolerant Stream Processing at Internet Scale
Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
