Blumchen - Push-based Outbox for PostgreSQL
July 3, 2024 · View on GitHub
Outbox Pattern with CDC and .NET based on Postgres logical replication with Npgsql integration.
Read more details in:
- Push-based Outbox Pattern with Postgres Logical Replication.
- How to get all messages through Postgres logical replication.
Features:
- Publication filter is enabled to trigger only INSERTed rows;
- AOT compliant compilation enforced by design
Main logic is placed in EventsSubscription.
Running source code locally
- Start Postgres with WAL enabled from Docker image.
docker-compose up
- Run(order doesn't matter) Publisher and Subscriber apps, under 'demo' folder, from vs-studio, and follow Publisher instructions.
Testing (against default docker instance)
Run tests
dotnet test
Links
Postgres Logical Replication
- Postgres Docs - Postgres logical replication
- Npgsql - Logical Replication
- Robert Treat - Logical Replication...LIVE!
- Dmitry Narizhnykh - PostgreSQL Change Data Capture and Golang Sample Code
WAL
- Devrim Gündüz -WAL: Everything you want to know
- Postgres Documentation - Write Ahead Log
- The Internals of PostgreSQL - Write Ahead Logging — WAL
- Hevo - Working With Postgres WAL Made Easy
Logical Replication
General Introduction
- Gunnar Morling - The Wonders of Postgres Logical Decoding Messages
- Gunnar Morling - Open-source Change Data Capture With Debezium - video
- Gunnar Morling - Open-source Change Data Capture With Debezium - slides
- Several9s - Using PostgreSQL Logical Replication to Maintain an Always Up-to-Date Read/Write TEST Server
- Matt Tanner - PostgreSQL CDC: A Comprehensive Guide
Other
- Dmitry Narizhnykh - PostgreSQL Change Data Capture and Golang Sample Code
- Fujistsu - How PostgreSQL 15 improved communication in logical replication
- Kinsta - PostgreSQL Replication: A Comprehensive Guide
- Amit Kapila - Replication Improvements In PostgreSQL-14
- Postgresql Wiki - Logical Decoding Plugins
- Npgsql - Logical Replication
- Konstantin Evteev - Recovery use cases for Logical Replication in PostgreSQL 10
- EDB - PostgreSQL Write-Ahead Logging (WAL) Trade-offs: Bounded vs. Archived vs. Replication Slots
- Percona - The 1-2-3 for PostgreSQL Logical Replication Using an RDS Snapshot
- 2nd Quadrant - Basics of Tuning Checkpoints
- Thiago - How to use Change Data Capture (CDC) with Postgres
- Ramesh naik E - Change Data Capture(CDC) in PostgreSQL
- Wal2Json - JSON output plugin for changeset extraction
- AWS Database Blog - Using logical replication to replicate managed Amazon RDS for PostgreSQL and Amazon Aurora to self-managed PostgreSQL
- AWS Database Blog - Stream changes from Amazon RDS for PostgreSQL using Amazon Kinesis Data Streams and AWS Lambda
- PGDeltaStream - Streaming Postgres logical replication changes atleast-once over websockets
- Hrvoje Milković - Replicate PostreSQL data to Elasticsearch via Logical replication slots
- Azure - Azure Database for PostgreSQL : Logical Replication
- Azure Database for PostgreSQL—Logical decoding and wal2json for change data capture
- Amit Langote - Postgresql To Kinesis For Java - Disney Streaming
Queue
Performance
Snapshots
Ordering
- Virender Singla - Postgres — Logical Replication and long running transactions
- Ordering with Advisory Locks
Partitioning
Locks
- Depesz - Picking task from queue – revisit
- Alvaro Herrera - Waiting for 9.5 – Implement SKIP LOCKED for row-level locks
- Chris Hanks - Turning PostgreSQL into a queue serving 10,000 jobs per second
- Vlad Mihalcea - How do PostgreSQL advisory locks work
- Marco Slot - When Postgres blocks: 7 tips for dealing with locks
- Marco Slot - PostgreSQL rocks, except when it blocks: Understanding locks
- Nickolay Ihalainen - PostgreSQL locking, Part 1: Row Locks
- Nickolay Ihalainen - PostgreSQL locking, part 2: heavyweight locks
- Nickolay Ihalainen - PostgreSQL locking, part 3: lightweight locks