ApeDTS

A production-ready Rust binary (~17 MB compressed) for multi-engine snapshot migration, CDC, and data validation.

ApeDTS is a lightweight, standalone data migration and CDC engine. It handles snapshot migration, continuous synchronization, data validation, correction, and in-flight transformation across databases, message queues, and analytical engines. Designed for cloud-native, stateless deployments.

Key features

• Lightweight by design — Runs as a single standalone process, with no external runtime components or intermediate storage.
• Mature multi-engine coverage — Supports currently validated homogeneous and heterogeneous paths spanning MySQL, PostgreSQL, MongoDB, Redis, Kafka, TiDB, StarRocks, ClickHouse, and Doris.
• Production-ready reliability — Provides checkpoint-based recovery for snapshot and CDC tasks, Prometheus-compatible metrics, and task-level observability.
• Source- and task-specific parallelism — Uses dedicated parallel strategies for different sources, targets, and task types to improve throughput.
• Advanced data capabilities — On supported paths, validates, corrects, and reviews data; filters and routes at the database, table, and column levels; and applies lightweight Lua transformations in flight.

Supported task types

Currently supported mature task types:

	mysql -> mysql	pg -> pg	mongo -> mongo	redis -> redis	mysql -> kafka	pg -> kafka	mysql -> starrocks	mysql -> clickhouse	mysql -> tidb	pg -> starrocks	pg -> clickhouse	mysql -> doris	pg -> doris
Snapshot	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
CDC	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
Data check/revise/review	✔	✔	✔						✔
Structure migration	✔	✔	✔				✔	✔	✔	✔	✔	✔	✔

Quick starts

CLI

dtscli is a lightweight local CLI for creating and managing ApeCloud DTS tasks. It can generate task configs, start dt-main, list tasks, stream logs, and stop, restart, or delete local task records.

For installation and detailed usage, see dt-cli/README.md.

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> doris
• pg -> clickhouse
• pg -> kafka -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations
  • config details
• Error handling
  • error codes
• Structure tasks
  • migration
  • check
  • check by Liquibase
• Snapshot tasks
  • data migration
  • data check
  • data revise
  • data review
  • resume at breakpoint
  • multiple tables in parallel
• CDC tasks
  • data sync
  • heartbeat to source database
  • two-way data sync
  • generate sqls from CDC
  • resume at breakpoint
• Custom consumers
  • mysql/pg -> kafka -> consumer
• Data processing
  • modify data by lua
• Monitor
  • monitor info
  • position info
• Task templates
  • mysql -> mysql
  • pg -> pg
  • mongo -> mongo
  • redis -> redis
  • mysql/pg -> kafka
  • mysql -> starrocks
  • mysql -> doris
  • mysql -> clickhouse
  • pg -> starrocks
  • pg -> doris
  • pg -> clickhouse

Benchmark

• MySQL -> MySQL, Snapshot

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	71428	8.2% / 5.2%	211% / 5.1%
ape_dts	2c4g	99403	14.0% / 5.2%	359% / 5.1%
ape_dts	4c8g	126582	13.8% / 5.2%	552% / 5.1%
debezium	4c8g	4051	21.5% / 5.2%	51.2% / 5.1%

• MySQL -> MySQL, CDC

Method	Node Specs	RPS(rows per second)	Source MySQL Load (CPU/Memory)	Target MySQL Load (CPU/Memory)
ape_dts	1c2g	15002	18.8% / 5.2%	467% / 6.5%
ape_dts	2c4g	24692	18.1% / 5.2%	687% / 6.5%
ape_dts	4c8g	26287	18.2% / 5.2%	685% / 6.5%
debezium	4c8g	2951	20.4% / 5.2%	98% / 6.5%

• more benchmark details

Contributing

Structure

Modules

• dt-main: program entry

• dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure

• dt-connector: extractors + sinkers for databases

• dt-pipeline: pipeline to connect extractors and sinkers

• dt-parallelizer: parallel algorithms

• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations

• dt-common: common utils, basic data structures, metadata management

• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Crate features

The dt-main crate provides several optional components which can be enabled via Cargo [features]:

• metrics: Enable Prometheus format task metrics HTTP service interface. See the task metrics reference for metric names, units, and semantics. After enabling this feature, you can customize the metrics service with the following configuration:

  [metrics]
  # http service host
  http_host=127.0.0.1
  # http service port
  http_port=9090
  # http service worker count
  workers=2
  # prometheus metrics const labels
  labels=your_label1:your_value1,your_label2:your_value2
  

Build

• Minimum supported Rust version (MSRV) The current minimum supported Rust version (MSRV) is 1.85.0.
• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Community

If you have any questions, you can reach out to us through:

• ApeDTS GitHub Discussions

• ApeDTS Wechat Account with note ape-dts: