My Messaging Resources - James Lavin

(Extracted from My Tech Resources due to Github README.markdown length limit & truncation)

DESCRIPTION

Links to messaging (message queues, streams, brokers, etc) resources I have found useful or think might be helpful to future me or developers like me.

Related: Event Sourcing, CQRS, DDD, & Microservices

MESSAGING

MESSAGING - AMQP

• Keynote: AMQP Ten Years After - Alvaro Videla (RabbitMQ Summit 2023)

MESSAGING - KAFKA

• kafka.apache.org
  • Intro
  • Quickstart
  • Use Cases
  • Documentation
  • Performance
  • Powered By
  • Papers & Presentations
  • Wiki
  • Ecosystem
• Examples - Confluent
• Streaming Audio Podcast - Confluent
• Event-Driven Architectures - Jay Kreps
• An introduction to Apache Kafka - this is Event Streaming, not just Messaging - Katherine Stanley (Jfokus 2019)
• Commit Logs in an Age of Microservices by Tim Berglund (#MeetTheExperts)
• Kafka: The Definitive Guide - Neha Narkhede, Gwen Shapira & Todd Palino (Confluent)
• Embrace the Anarchy : Apache Kafka’s Role in Modern Data Architectures - Robin Moffatt
• 7 Reasons why your microservices should use Event Sourcing & CQRS - Hugh McKee (Reactive Summit 2018)
• Apache Kafka as an Event Store - Guido Schmutz (Kafka Summit NYC 2019)
• The Present and Future of the Streaming Platform - Neha Narkhede (Kafka Summit London 2018)
• Kafka and Event-Oriented Architecture - Jay Kreps (Kafka Summit SF 2018)
• The Log: What every software engineer should know about real-time data's unifying abstraction - Jay Kreps
• Thorough Introduction to Apache Kafka™ - Stanislav Kozlovski
• Toward a Functional Programming Analogy for Microservices - Bobby Calderwood
• Is Kafka a Database? - Martin Kleppmann (Kafka Summit SF 2018)
• Amy Boyle (NewRelic): Using Apache Kafka for Real-Time Event Processing at New Relic | Effective Strategies for Kafka Topic Partitioning
• Goodbye history tables, hello full audit- exploring message streams & event sourcing - Michele Leroux Bustamante (NDC {Oslo} 2017)
• Building Event-Driven Microservices with Event Sourcing and CQRS - Lidan Hifi (NDC {Oslo} 2018)
• Kafkapocalypse: Monitoring Kafka Without Losing Your Mind - Ben Summer (NewRelic)
• Publishing with Apache Kafka at The New York Times - Boerge Svingen (Confluent)
• Event sourcing using Kafka - Adam Warski
• Intro to Apache Kafka as Event-Driven Open Source Streaming Platform - Kai Waehner (VoxxedDays Zurich)
• Lessons learned from Kafka in production - Tim Berglund
• Kafka Summit NYC 2017 videos & slides
• Ben Stopford (Confluent): The Data Dichotomy- Rethinking the Way We Treat Data and Services | Build Services on a Backbone of Events | Using Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures | Chain Services with Exactly Once Guarantees | Messaging as the Single Source of Truth | Leveraging the Power of a Database Unbundled | Building a Microservices Ecosystem with Kafka Streams and KSQL
• KSQL Developer Guide - Confluent
• Building Event-Driven Services with Apache Kafka | Putting the Micro into Microservices with Stateful Stream Processing
• Distributed Commit Log: Application Techniques for Transaction Processing - David McNeil
• Distributed Commit Logs with Apache Kafka - James Ward (Devoxx Belgium 2016)
• Confluent.io: Blog | Resources
• Exactly-Once Semantics in Kafka: Listen up, I will only say this once! - Jason Gustafson
• Transactions in Apache Kafka - Apurva Mehta and Jason Gustafson
• When Microservices Meet Event Sourcing - Vinicius Gomes (O'Reilly Software Architecture Conference NYC 2017)
• Apache Kafka and the Next 700 Stream Processing Systems - Jay Kreps
• Reducing Microservice Complexity with Kafka and Reactive Streams - Jim Riecken (Scala Days New York 2016)
• Stream All Things - Patterns of Modern Data Integration - Gwen Shapira (GOTO Chicago 2017)
• A practical introduction to event sourcing and CQRS - Robert Lemke (Neos CMS 2017) (discusses Neos Flow for PHP)
• How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka - Kai Waehner
• Standing on the Distributed Shoulders of Giants - Pat Helland (Kafka Summit 2017)
• fast-data-dev: Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors - Landoop
• Understanding Kafka with Legos - Jesse Anderson
• Apache Kafka and the Rise of Stream Processing - Guozhang Wang
• Developing Real-Time Data Pipelines with Apache Kafka - Joe Stein
• Reactive Kafka - Rajini Sivaram (SpringOne Platform 2016)
• Introduction to Apache Kafka - James Ward (DevoxxUS 2017)
• Processing Streaming Data at a Large Scale with Kafka - Thijs Cadier (RailsConf 2017)
• Event sourcing in practise - lessons learned - Jfokus 2017
• Intro to Apache Kafka and Real-Time ETL for DBAs - Gwen Shapira and Jignesh Shah

MESSAGING - KAFKA - BLOGS

• Confluent blog

MESSAGING - KAFKA - CONFLUENT PLATFORM

• Confluent Platform Demo
• Helm 101: Apache Kafka and Confluent Platform for Kubernetes Explained - Viktor Gamov
• Confluent Blog
• Demo: Building a data pipeline with Confluent Cloud and Google BigQuery - Gwen Shapira

MESSAGING - KAFKA - DELETING DATA

• Handling GDPR: How to make Kafka Forget - Ben Stopford
• Delete Arbitrary Messages from a Kafka - Ben Stopford

MESSAGING - KAFKA - EXAMPLES

• Experiences from Building an Event-Sourced System with Kafka Streams - Jan Stenberg
• We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL - Robin Moffatt: Part 1: Filtering | Part 2: Event-Driven Alerting With Slack | Part 3: Enriching events with external data

MESSAGING - KAFKA - KAF

• Kaf: Modern CLI for Apache Kafka

MESSAGING - KAFKA - KAFKA CONNECT

• Kafka Connect Deep Dive – Converters and Serialization Explained - Robin Moffatt

MESSAGING - KAFKA - KAFKA MANAGER

• kafka-manager: tool for managing Apache Kafka - Yahoo
• kafka-manager Helm chart

MESSAGING - KAFKA - KAFKA STREAMS

• Apache Kafka Streams API - Tim Berglund (Confluent): 1. Intro to Streams | 2. Creating a Streams Application | 3. Transforming Data, Part I | 4. Transforming Data II
• How Kafka Enables Real-Time Stream Processing - André Melo: Part 1 | Part 2
• Building event sourced systems with Kafka Streams - Amitay Horwitz, Wix (JEE Conf)
• Beyond the DSL—Unlocking the power of Kafka Streams with the Processor API - Antony Stubbs (Devoxx 2018)
• Building Streaming Microservices with Apache Kafka - Tim Berglund (Devoxx Poland 2018) & Processing Streaming Data with KSQL - Tim Berglund (Devoxx Poland 2018) & Processing Streaming Data with KSQL - Tim Berglund (GOTO 2019)
• Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! - Robin Moffatt (Devoxx 2018)
• Inventory Microservice example with Kafka Streams and CQRS - Sanjay Acharya
• Putting Events in Their Place with Dynamic Routing - Yeva Byzek
• Kafka Streams for Event Driven Microservices - Marius Bogoevici (DevNation Live)
• Kafka based Microservices with Akka Streams and Kafka Streams - Dean Wampler (ScalaDays 2018)
• Of Streams and Tables in Kafka and Stream Processing, Part 1 - Michael G. Noll
• Getting Started With the Kafka Streams API
• Kafka Streams documentation
• Kafka Music demo application for the Kafka Streams API - Confluent
• Data Stream Processing: Concepts and Implementations - Matthias Niehoff (Codecentric AG)
• Kafka and the Service Mesh - Gwen Shapira (Confluent)
• Kafka’s Streams API for Highly Scalable Machine Learning & Deep Learning in Real Time - Kai Waehner (Confluent)
• Ranking Websites in Real-time with Apache Kafka’s Streams API - Hunter Kelly
• How to use Apache Kafka to transform a batch pipeline into a real-time one - Stéphane Maarek
• Rethinking Stream Processing with Apache Kafka - Michael Noll (Berlin Buzzwords 2017)
• Introducing Kafka Streams the New Stream Processing Library of Apache Kafka
• ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka - Neha Narkhede
• Unifying Stream Processing and Interactive Queries in Apache Kafka - Eno Thereska
• Crossing the Streams: Joins in Apache Kafka - Florian Troßbach & Matthias J Sax
• Interactive Queries in Apache Kafka Streams - Florian Troßbach
• Apache Samza: Distributed stream processing framework
• Real-time Financial Alerts at Rabobank with Apache Kafka’s Streams API - Jeroen van Disseldorp
• Pinot: Realtime distributed OLAP datastore (LinkedIn)
• Streaming Data Pipelines with Brooklin - Samarth Shetty, LinkedIn (5/24/17)

MESSAGING - KAFKA - KAFKA STREAMS - KSQL

• Github
• KSQL and Other Stream Processing Tools in Kafka - Nick Dearden (Confluent) Level Up Your KSQL - Tim Berglund (Confluent):
  • Demo: Build a Streaming Application with KSQL
  • KSQL from Confluent
  • KSQL Intro
  • KSQL and Core Kafka
  • KSQL Use Cases
  • Installing & Running KSQL
  • KSQL Streams and Tables
  • Reading Kafka Data From KSQL
  • Streaming and Unbounded Data in KSQL
  • Enriching Data with KSQL
  • Aggregations in KSQL
  • Taking KSQL to Production
  • Insert Into
  • Struct (Nested Data)
  • Stream-Stream Joins
  • UDFs / UDAFs
• Streaming SQL for Apache Kafka - Confluent
• Ask Confluent With Gwen Shapira
• Reading Kafka Data from KSQL
• Building a Real-Time Streaming Platform with Oracle, Apache Kafka, and KSQL - Gwen Shapira
• Introducing KSQL: Open Source Streaming SQL for Apache Kafka - Neha Narkhede
• KSQL: A Streaming SQL Engine for Apache Kafka - Tim Berglund
• Confluent Releases KSQL, a Distributed Streaming SQL Engine for Apache Kafka - Srini Penchikala
• KSQL from Confluent: Streaming SQL for Apache Kafka® - Tim Berglund
• KSQL 201: A Deep Dive into Query Processing - Hojjat Jafarpour, Confluent
• Level Up Your KSQL - Tim Berglund

MESSAGING - KAFKA - MIRRORMAKER

• Kafka MirrorMaker - Confluent
• Transfer data across clusters in Kafka with MirrorMaker - Parisa Moghaddam
• Kafka Replication: The case for MirrorMaker 2.0 - Renu Tewan

MESSAGING - KAFKA - SECURITY

• Kafka Security 101 & Real World Tips - Stephane Maarek
• Transparently Securing Kafka, Istio-style, with up to 300% Higher Performance than Native TLS in Microservice Environments - Jayanth Gummaraju | original article
• Apache Kafka Security: SSL, Kerberos & Authorization - Sriharsha Chintalapani & Parth Brahmbhatt (2016 Hadoop Summit San Jose)

MESSAGING - KAFKA - PRODUCTIONIZING

• Kafka Helm Chart: Kubeapps.com & Helm.sh & Github
• Kubernetes-kafka: Kafka cluster as Kubernetes StatefulSet, plain manifests and config
• Kafka in Docker - Spotify
• Kafka-docker
• Ches/kafka (Docker)
• Kafka on Kubernetes: From Evaluation to Production at Intuit - Shrinand Javadekar (KubeCon + CloudNativeCon 2018)
• Deploying Kafka Streams Applications with Docker and Kubernetes - Gwen Shapira + Matthias J. Sax - (Kafka Summit SF 2018)
• Demo: Scaling Kafka Streams with Kubernetes - Gwen Shapira (Confluent) | Code for demo
• Creating the Adobe Experience Platform Pipeline with Kafka - Jaemi Bremner
• Kafka in Containers in Docker in Kubernetes in The Cloud - Christopher Vollick, Shopify (Kafka Summit 2018)
• Best Practices for Running Kafka on Docker Containers - Nanda Vijaydev
• Disaster Recovery for Multi-Datacenter Apache Kafka Deployments - Yeva Byzek
• Things We Wish We Knew Before Operationalising Kafka - Sreekanth Ramakrishnan (BigData.SG & Hadoop.SG)
• Tuning Kafka for low latency guaranteed messaging -- Jiangjie (Becket) Qin (LinkedIn), 6/15/16
• Running Kafka in Production: lessons learned - Ivan Dyachkov (Klarna the KonferenSE 2016)
• Scaling Slack’s Job Queue - Saroj Yadav, et al. (Slack)

MESSAGING - KAFKA - PRODUCTIONIZING - CONFLUENT + GOOGLE CLOUD

• GCP for Apache Kafka Users: Stream Ingestion and Processing - Ricardo Ferreira & Karthi Thyagarajan (Cloud Next '19)

MESSAGING - KAFKA - PRODUCTIONIZING - STRIMZI

• Strimzi: Kubernetes Kafka Operator
• Kafka on Kubernetes - April 2019 - Marius Bogoevici & Slides
• Strimzi: Running Apache Kafka on Kubernetes and OpenShift - Jakub Scholz, Stanislav Knot
• Strimzi: Kubernetes Operator for running Kafka (and OpenShift): Github | Strimzi.io | Minikube quickstart | Using Helm to Deploy the Strimzi Kafka Operator
• Using OperatorHub.io to installing Strimzi operator for Apache Kafka on Kubernetes - Jakub Scholz

MESSAGING - KAFKA - LOG PERSISTENCE

• Secor: Service implementing Kafka log persistence
• Bifrost: Safely archive data from Apache Kafka to S3 with no Hadoop dependencies
• Building a Distributed Message Log from Scratch - Tyler Treat (Scale 16x Pasadena 2018)

MESSAGING - KAFKA - VIDEOS

• Kafka Summit New York 2019
• Kafka Summit London 2018
• Kafka Summit San Francisco 2017

MESSAGING - MESSAGES

• Messages: Patterns and Pitfalls - Richard Wellum (µCon London 2018)

MESSAGING - PINOT

"Realtime distributed OLAP datastore, designed to answer OLAP queries with low latency"

• Pinot.apache.org | Docs
• Github

MESSAGING - PULSAR

• Apache Pulsar - Distributed pub-sub messaging system | Concepts | Schema | Functions | IO | SQL | Deployment | Administration | Security | Admin API | Adapters | Cookbooks
• Apache Pulsar 101: architecture, concepts & benchmarks - Quentin Adam & Steven Le Roux (Devoxx Belgium 2019)
• Apache Pulsar: The Next Generation Messaging and Queuing System - Karthik Ramasamy (YOW! Data 2020)
• Ten reasons to choose Apache Pulsar over Apache Kafka for Event Sourcing (CQRS) — Robert van Mölken (StreamNative 2020)
• Pulsar Summit Virtual Conference 2020: Day 1 presentations | Day 2 presentations
• Pulsar vs. Kafka — Part 1 — A More Accurate Perspective on Performance, Architecture, and Features - Carolyn King, Addison Higham, and Sijie Guo
• Infinite Topic Backlogs with Apache Pulsar - Ivan Kelly: (Prezi Conference Team 2019) | (J On the Beach 2019)
• Apache Pulsar and the Legacy of Telegraphy - Jowanza Joseph (StrangeLoop 2019)
• Apache Pulsar 101 - Streamlio
• Apache Pulsar Tutorials - Streamlio
• Using Apache Pulsar Functions - Streamlio
• Apache Pulsar and Apache BookKeeper - Streamlio
• DB Lunch: Unifying Messaging, Queuing, Streaming & Light Weight Compute with Apache Pulsar - Karthik Ramasamy (Microsoft Research)
• Elastic Data Processing with Apache Flink and Apache Pulsar - Sijie Guo (Flink Forward)
• Pulsar in Action - David Kjerrumgaard (💰NON-FREE book💰) | Code (Github)

MESSAGING - RABBITMQ

• Update from RabbitMQ Core Team by Howard Twine & Roger Freixa (RabbitMQ Summit 2023)

MESSAGING - RABBITMQ - CLIENTS

• Client Documentation - RabbitMQ.com
• Erlang RabbitMQ Client library - RabbitMQ.com

MESSAGING - RABBITMQ - PERFORMANCE

• RabbitMQ Performance Improvements - David Ansari (Code BEAM America 2022)

MESSAGING - RABBITMQ - STREAMS

• Streams: past, present, and future - Arnaud Cogoluègnes & Karl Nilsson (RabbitMQ Summit 2023)
• Using RabbitMQ Steams In Go - ProgrammingPercy
• What's New in RabbitMQ Streams? - Arnaud Cogoluègnes (RabbitMQ Summit 2022)
• Introducing Stream Support In RabbitMQ - Erlang Admin
• Introducing stream support in RabbitMQ - Arnaud Cogoluègnes (RabbitMQ Summit 21)
• RabbitMQ Streams: The Architectural Possibilities - Alastair Turner (horrible video quality, 360p)
• Panel discussion: The new RabbitMQ Streams (RabbitMQ Summit 2021)

MESSAGING - RABBITMQ - VIDEOS

• RabbitMQ Summit 2023
• RabbitMQ Summit 2022
• RabbitMQ Summit 2021