sql-splitter

August 11, 2026 · View on GitHub

Crates.io Downloads Build License: MIT Amp

Split large SQL dump files into individual table files. Fast, memory-efficient, multi-dialect.

  • ~1 GB/s end-to-end split throughput on large dumps (0.8–1.4 GB/s across dialects on Apple M2 Max; hardware-dependent)
  • MySQL, PostgreSQL, SQLite, MSSQL support (including COPY FROM stdin, GO batches)
  • Compressed files — gzip, bzip2, xz, zstd, zip (single .sql member) auto-detected
  • Streaming architecture — handles files larger than RAM
  • 10x faster than shell-based alternatives

Installation

From crates.io

cargo install sql-splitter

Cargo features (library consumers)

sql-splitter enables all features by default. To reduce dependency footprint:

sql-splitter = { version = "1", default-features = false }

Optional features:

  • compression (gzip/bzip2/xz/zstd support, for compressed input and per-file output)
  • archive (single-file tar.*/zip output, plus .zip archive input; implies compression)
  • duckdb-query (enables the query command and DuckDB integration)

From source

git clone https://github.com/helgesverre/sql-splitter
cd sql-splitter
just install  # Installs binary + shell completions + man pages

Or download pre-built binaries from GitHub Releases.

Man pages (optional)

After installation, view documentation with man sql-splitter or man sql-splitter-diff.

For cargo install users, install man pages manually:

git clone https://github.com/helgesverre/sql-splitter
cd sql-splitter
just install-man

Usage

# MySQL/MariaDB dump (default)
sql-splitter split dump.sql -o tables/

# PostgreSQL pg_dump
sql-splitter split pg_dump.sql -o tables/ --dialect=postgres

# SQLite dump
sql-splitter split sqlite.sql -o tables/ --dialect=sqlite

# MSSQL/T-SQL dump (SSMS "Generate Scripts", sqlcmd)
sql-splitter split mssql_dump.sql -o tables/ --dialect=mssql

# Compressed input (auto-detected)
sql-splitter split backup.sql.gz -o tables/
sql-splitter split backup.sql.zst -o tables/

# Zipped input — a .zip with exactly one .sql member (deflated or stored)
sql-splitter split backup.zip -o tables/

# Compressed output — one compressed file per table (users.sql.zst, ...)
sql-splitter split dump.sql -o tables/ --compress zstd   # or gzip, bzip2, xz

# Single-archive output — all tables in one file (format from the extension)
sql-splitter split dump.sql -o dump.tar.gz               # .tgz/.tar.zst/.tar.bz2/.tar.xz/.tar
sql-splitter split dump.sql -o dump.zip

# Split specific tables only
sql-splitter split dump.sql --tables users,posts,orders

# Select one schema-qualified table without merging it with namesakes
sql-splitter split postgres_dump.sql --dialect postgres --tables tenant_a.users

# Schema only (CREATE TABLE, indexes, etc.)
sql-splitter split dump.sql -o schema/ --schema-only

# Data only (INSERT/COPY statements)
sql-splitter split dump.sql -o data/ --data-only

# Merge split files back into single dump
sql-splitter merge tables/ -o restored.sql

# Merge specific tables only
sql-splitter merge tables/ -o partial.sql --tables users,orders

# Merge with transaction wrapper
sql-splitter merge tables/ -o restored.sql --transaction

# Analyze without splitting
sql-splitter analyze dump.sql

# Convert between SQL dialects
sql-splitter convert mysql_dump.sql -o postgres_dump.sql --to postgres
sql-splitter convert pg_dump.sql -o mysql_dump.sql --to mysql
sql-splitter convert dump.sql -o sqlite_dump.sql --to sqlite
sql-splitter convert mssql_dump.sql -o mysql_dump.sql --to mysql

# Convert with explicit source dialect
sql-splitter convert dump.sql --from postgres --to mysql -o output.sql
sql-splitter convert dump.sql --from mssql --to postgres -o output.sql

# Validate SQL dump integrity
sql-splitter validate dump.sql

# Validate with strict mode (warnings = errors)
sql-splitter validate dump.sql --strict

# Validate with JSON output for CI
sql-splitter validate dump.sql --json

# Batch operations with glob patterns
sql-splitter validate "dumps/*.sql" --fail-fast
sql-splitter analyze "**/*.sql"
sql-splitter split "*.sql" -o output/
sql-splitter convert "*.sql" --to postgres -o converted/

# Compare two SQL dumps for changes
sql-splitter diff old.sql new.sql

# Diff with schema-only or data-only
sql-splitter diff old.sql new.sql --schema-only
sql-splitter diff old.sql new.sql --data-only

# Diff with JSON or SQL migration output
sql-splitter diff old.sql new.sql --format json -o diff.json
sql-splitter diff old.sql new.sql --format sql -o migration.sql

# Diff with verbose PK samples and ignore timestamp columns
sql-splitter diff old.sql new.sql --verbose --ignore-columns "*.updated_at,*.created_at"

# Override primary key for tables without PK
sql-splitter diff old.sql new.sql --primary-key logs:timestamp+message

# Redact sensitive data using inline patterns
sql-splitter redact dump.sql -o safe.sql --null "*.ssn" --hash "*.email" --fake "*.name"

# Redact using YAML config file
sql-splitter redact dump.sql -o safe.sql --config redact.yaml

# Generate redaction config by analyzing input file
sql-splitter redact dump.sql --generate-config -o redact.yaml

# Reproducible redaction with seed
sql-splitter redact dump.sql -o safe.sql --null "*.password" --seed 42

# Generate synthetic SQL directly from a dump (profile, infer, generate)
sql-splitter generate production.sql -o synthetic.sql

# Emit an editable model instead of SQL, then generate from it later
sql-splitter generate production.sql --emit-config synthetic.yaml
sql-splitter generate --config synthetic.yaml -o synthetic.sql

# Generate from a model only, no source dump
sql-splitter generate --config synthetic.yaml -o synthetic.sql

# Apply overrides, freeze the resolved model, and scale row counts
sql-splitter generate production.sql --config overrides.yaml \
  --emit-config resolved.yaml --scale 0.1 -o synthetic.sql

# Validate a model, or generate/audit/publish atomically
sql-splitter generate --config synthetic.yaml --check
sql-splitter generate --config synthetic.yaml --verify -o synthetic.sql

# Generate ERD (Entity-Relationship Diagram)
sql-splitter graph dump.sql -o schema.html        # Interactive HTML (default)
sql-splitter graph dump.sql -o schema.dot         # Graphviz DOT format
sql-splitter graph dump.sql -o schema.mmd         # Mermaid erDiagram
sql-splitter graph dump.sql -o schema.json        # JSON with full schema details

# Graph with filtering
sql-splitter graph dump.sql --tables "user*,order*" -o filtered.html
sql-splitter graph dump.sql --exclude "log*,audit*" -o clean.html
sql-splitter graph dump.sql --cycles-only         # Only tables in circular dependencies

# Focus on specific table and its relationships
sql-splitter graph dump.sql --table orders --transitive  # Show all dependencies
sql-splitter graph dump.sql --table users --reverse      # Show all dependents

# Reorder SQL dump in topological FK order
sql-splitter order dump.sql -o ordered.sql        # Safe import order
sql-splitter order dump.sql --check               # Check for cycles
sql-splitter order dump.sql --reverse             # Reverse (for DROP operations)

# Query SQL dumps with DuckDB analytics engine
sql-splitter query dump.sql "SELECT COUNT(*) FROM users"
sql-splitter query dump.sql "SELECT * FROM orders WHERE total > 100" -f json
sql-splitter query dump.sql "SELECT * FROM users LIMIT 10" -o results.csv -f csv
sql-splitter query dump.sql --interactive         # Start REPL session
sql-splitter query huge.sql "SELECT ..." --disk   # Use disk mode for large files

# Query with caching for repeated queries
sql-splitter query dump.sql "SELECT ..." --cache  # Cache imported database
sql-splitter query --list-cache                   # Show cached databases
sql-splitter query --clear-cache                  # Clear all cached databases

# Generate shell completions (auto-installed with just install)
sql-splitter completions bash >> ~/.bashrc
sql-splitter completions zsh >> ~/.zshrc
sql-splitter completions fish >> ~/.config/fish/completions/sql-splitter.fish

Shell Completions

Shell completions are automatically installed when using just install. For manual installation:

# Install for current shell only
just install-completions

# Install for all shells (bash, zsh, fish)
just install-completions-all

Why sql-splitter?

sql-splitter is a dump-first, CLI-first tool designed for automation and CI/CD pipelines.

What it's optimized for

StrengthDescription
One tool for the workflowSplit → sample → shard → convert → merge in a single binary
Works on dump filesNo running database or JDBC connection needed (unlike mydumper, Jailer, Condenser)
Streaming architecture10GB+ dumps with bounded memory, ~1 GB/s split throughput
Multi-dialect + conversionMySQL, PostgreSQL, SQLite including COPY FROM stdin → INSERT
FK-aware operationsSampling and tenant sharding preserve referential integrity

When another tool might be better

  • mydumper — Parallel snapshots from live MySQL/MariaDB databases
  • Jailer — Rich GUI-based FK subsetting with JDBC across 12+ databases
  • sqlglot — Query-level transpilation and AST manipulation (31 dialects)
  • DuckDB — Complex analytical queries over SQL/CSV/JSON/Parquet

See docs/COMPETITIVE_ANALYSIS.md for detailed comparisons.

Options

Split Options

FlagDescriptionDefault
-o, --outputOutput directory, or an archive path (.tar.gz, .zip, …)output
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
-t, --tablesOnly split these tables (comma-separated)
--compressCompress each output file: gzip, zstd, bzip2, xznone
--io-strategyOutput device I/O strategy: auto, ssd, hdd, cheapauto
-p, --progressShow progress bar
--dry-runPreview without writing files
--schema-onlyOnly DDL statements (CREATE, ALTER, DROP)
--data-onlyOnly DML statements (INSERT, COPY)
--fail-fastStop on first error (for glob patterns)
--jsonOutput results as JSON

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql), plain or compressed (.gz/.bz2/.xz/.zst), or a .zip archive containing exactly one .sql member (deflated or stored; junk entries like __MACOSX/ and .DS_Store are ignored). This applies to every command that reads a dump, not just split.

Output formats. By default split writes one plain <table>.sql per table. --compress <fmt> instead writes one compressed file per table (<table>.sql.gz/.zst/.bz2/.xz) — useful for archival or transfer; each file compresses independently and in parallel. Alternatively, give -o an archive path (dump.tar.gz, .tgz, .tar.zst, .tar.bz2, .tar.xz, .tar, or .zip) to pack all tables into a single archive of <table>.sql entries. Archive output requires a single input file, and --compress applies only to directory output (for an archive the codec comes from the extension).

Slow output devices. The write path is tuned for SSDs by default. When the output lands on a spinning disk, USB stick, or network mount, --io-strategy auto (the default) detects it — a quick startup probe plus runtime backpressure monitoring — and adjusts how writes are scheduled. In our tests on a single USB 3.0 spinning drive (ExFAT), this ran roughly 1.5–2.5× faster than the SSD settings; treat that as indicative for similar hardware, not a guarantee — the gain depends on the device, filesystem, and workload.

The profiles aren't magic — they set three concrete knobs (writer thread count, write size, staging memory), and every profile produces byte-identical output. Pin one when you already know the target device:

ProfileWritersWrite sizeStagingBuilt for
ssdup to 4256 KB32 MBSSD/NVMe — high-IOPS devices that like many small parallel writes
hdd1up to 64 MB256 MBSpinning disks — one writer avoids competing seeks; big sequential writes amortize head movement
cheap1up to 64 MB512 MBCheap USB flash, network mounts — every write operation is expensive regardless of locality, so issue the fewest, largest ones. (Also answers to potato.)
autoadaptsadaptsadaptsUnknown targets — probes the output dir at startup, then switches between the profiles above based on observed backpressure

Merge Options

FlagDescriptionDefault
-o, --outputOutput SQL filestdout
-d, --dialectSQL dialect for headers/footersmysql
-t, --tablesOnly merge these tables (comma-separated)all
-e, --excludeExclude these tables (comma-separated)
--transactionWrap in BEGIN/COMMIT transaction
--no-headerSkip header comments
-p, --progressShow progress bar
--dry-runPreview without writing files
--jsonOutput results as JSON

Analyze Options

FlagDescriptionDefault
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
-p, --progressShow progress bar
--fail-fastStop on first error (for glob patterns)
--jsonOutput results as JSON

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Convert Options

FlagDescriptionDefault
-o, --outputOutput SQL file or directory (required for glob)stdout
--fromSource dialect: mysql, postgres, sqlite, mssqlauto-detect
--toTarget dialect: mysql, postgres, sqlite, mssqlrequired
--strictFail on any unsupported feature
-p, --progressShow progress bar
--dry-runPreview without writing files
--fail-fastStop on first error (for glob patterns)
--enum-namingMySQL→PostgreSQL enum type names: per-column or dedupeper-column
--jsonOutput results as JSON

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Supported conversions (12 pairs):

  • MySQL ↔ PostgreSQL (including COPY → INSERT)
  • MySQL ↔ SQLite
  • MySQL ↔ MSSQL
  • PostgreSQL ↔ SQLite
  • PostgreSQL ↔ MSSQL
  • SQLite ↔ MSSQL

Features:

  • 50+ data type mappings
  • AUTO_INCREMENT ↔ SERIAL ↔ INTEGER PRIMARY KEY ↔ IDENTITY
  • PostgreSQL COPY → INSERT with NULL and escape handling
  • Session command stripping
  • Semantic-preserving ENUM declaration conversion between MySQL and PostgreSQL (ENUM(...)CREATE TYPE ... AS ENUM); arrays and targets without native enums use a documented lossy fallback with a warning
  • Warnings for unsupported features (arrays, triggers, PostgreSQL-only DDL)

Validate Options

FlagDescriptionDefault
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--strictTreat warnings as errors (exit 1)
--jsonOutput results as JSON
--max-rows-per-tableMax rows per table for PK/FK checks (0 = no limit)1,000,000
--no-limitDisable row limit for PK/FK checks
--no-fk-checksSkip PK/FK data integrity checks
-p, --progressShow progress bar
--fail-fastStop on first error (for glob patterns)

Input can be a file path or glob pattern (e.g., *.sql, dumps/**/*.sql).

Validation checks:

  • SQL syntax validation (parser errors)
  • DDL/DML consistency (INSERTs reference existing tables)
  • Encoding validation (UTF-8)
  • Duplicate primary key detection (all dialects)
  • FK referential integrity (all dialects)

Sample Options

FlagDescriptionDefault
-o, --outputOutput SQL filestdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--percentSample percentage (1-100)
--rowsSample fixed number of rows per table
--preserve-relationsPreserve FK relationships
-t, --tablesOnly sample these tables (comma-separated)all
-e, --excludeExclude these tables (comma-separated)
--root-tablesExplicit root tables for sampling
--include-globalGlobal table handling: none, lookups, alllookups
--seedRandom seed for reproducibilityrandom
-c, --configYAML config file for per-table settings
--max-total-rowsMaximum total rows to sample (0 = no limit)
--no-limitDisable row limit
--strict-fkFail if any FK integrity issues detected
--no-schemaExclude CREATE TABLE statements from output
-p, --progressShow progress bar
--dry-runPreview without writing files
--jsonOutput results as JSON

Shard Options

FlagDescriptionDefault
-o, --outputOutput SQL file or directorystdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--tenant-columnColumn name for tenant identificationauto-detect
--tenant-valueSingle tenant value to extract
--tenant-valuesMultiple tenant values (comma-separated)
--root-tablesExplicit root tables with tenant column
--include-globalGlobal table handling: none, lookups, alllookups
-c, --configYAML config file for table classification
--max-selected-rowsMaximum rows to select (0 = no limit)
--no-limitDisable row limit
--strict-fkFail if any FK integrity issues detected
--no-schemaExclude CREATE TABLE statements from output
-p, --progressShow progress bar
--dry-runPreview without writing files
--jsonOutput results as JSON

Diff Options

FlagDescriptionDefault
-o, --outputOutput file (default: stdout)stdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--schema-onlyCompare schema only, skip data
--data-onlyCompare data only, skip schema
--formatOutput format: text, json, sqltext
-t, --tablesOnly compare these tables (comma-separated)all
-e, --excludeExclude these tables (comma-separated)
--max-pk-entriesMax PK entries to track (0 = no limit)10,000,000
-v, --verboseShow sample PK values for added/removed/modified rows
--primary-keyOverride PK for tables (format: table:col1+col2)auto-detect
--ignore-orderIgnore column order differences in schema comparison
--ignore-columnsIgnore columns matching glob patterns (e.g., *.updated_at)
--allow-no-pkCompare tables without PK using all columns as key
-p, --progressShow progress bar

What diff detects:

  • Tables added/removed/modified (columns, types, nullability)
  • Primary key changes
  • Foreign key changes
  • Index changes (CREATE INDEX, inline INDEX/KEY)
  • Rows added/removed/modified (via PK-based comparison)

Output formats:

  • text: Human-readable summary with optional PK samples
  • json: Structured data for automation (includes warnings)
  • sql: Migration script with ALTER/CREATE INDEX/DROP INDEX statements

Graph Options

FlagDescriptionDefault
-o, --outputOutput file (html, dot, mmd, json, png, svg, pdf)stdout
--formatOutput format: html, dot, mermaid, jsonauto-detect
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--layoutLayout direction: lr (left-right), tb (top-bottom)lr
-t, --tablesOnly include tables matching glob patternsall
-e, --excludeExclude tables matching glob patterns
--tableFocus on a specific table
--transitiveShow all dependencies of focused table
--reverseShow all tables that depend on focused table
--max-depthLimit traversal depthunlimited
--cycles-onlyOnly show tables in circular dependencies
--renderRender DOT to PNG/SVG/PDF using Graphviz
-p, --progressShow progress bar
--jsonOutput as JSON

Output formats:

  • html: Interactive diagram with dark/light theme, copy Mermaid button
  • dot: Graphviz DOT with ERD-style tables (columns, types, PK/FK markers)
  • mermaid: Mermaid erDiagram syntax
  • json: Full schema with tables, columns, relationships, and stats
  • png/svg/pdf: Rendered image (requires Graphviz dot command)

Order Options

FlagDescriptionDefault
-o, --outputOutput SQL filestdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
--checkCheck for cycles and report order (don't write)
--dry-runShow topological order without writing
--reverseReverse order (children before parents, for DROP)

Redact Options

FlagDescriptionDefault
-o, --outputOutput SQL filestdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
-c, --configYAML config file for redaction rules
--generate-configAnalyze input and generate annotated YAML config
--nullColumns to set to NULL (glob patterns, comma-separated)
--hashColumns to hash with SHA256 (glob patterns)
--fakeColumns to replace with fake data (glob patterns)
--maskColumns to partially mask (format: pattern=column)
--constantColumn=value pairs for constant replacement
--seedRandom seed for reproducible redactionrandom
--localeLocale for fake data (en, de_de, fr_fr, etc.)en
-t, --tablesOnly redact specific tables (comma-separated)all
-e, --excludeExclude specific tables (comma-separated)
--strictFail on warnings (e.g., unsupported locale)
-p, --progressShow progress bar
--dry-runPreview without writing files
--jsonOutput results as JSON
--validateValidate config only, don't process

Redaction strategies:

  • null: Replace value with NULL
  • constant: Replace with fixed value
  • hash: SHA256 hash (deterministic, preserves FK relationships)
  • mask: Partial masking with pattern (*=asterisk, X=keep, #=random digit)
  • fake: Generate realistic fake data (25+ generators)
  • skip: No redaction (passthrough)

shuffle rules are rejected. They require a two-pass rewrite to avoid emitting the original values, which is not implemented yet.

Fake data generators:

email, name, first_name, last_name, phone, address, city, state, zip, country, company, job_title, username, url, ip, ipv6, uuid, date, datetime, credit_card, iban, ssn, lorem, paragraph, sentence

Generate Options

generate produces synthetic, not anonymized, SQL data — from a hand-authored YAML model, a profiled dump, or both. See the complete generate documentation for the model language, generator/modifier/planner catalogs, inference, privacy, verification, diagnostics, and library API.

FlagDescriptionDefault
[INPUT]SQL dump or schema to profile (optional with a complete model)
-c, --configkind: model or kind: overrides YAML document
--emit-configWrite the resolved model as YAML instead of generating
-o, --outputOutput SQL file; - for stdoutstdout
--profile-depthProfiling depth: basic, fullbasic
--profile-sampleRetained sample capacity per column while profiling1000
--input-dialectDialect of [INPUT], if profilingauto-detect
--dialectOutput SQL dialectpreserve source/model
--scaleGlobal multiplicative row-count scale
--rowsGlobal absolute root row count
--table-rowsPer-table absolute row-count override (table=count, repeatable)
--table-scalePer-table row-count scale override (table=factor, repeatable)
--max-rowsCap applied last to every table, including family-planner child rows
--tablesOnly generate these tables (comma-separated globs)all
--excludeExclude these tables (comma-separated globs)none
--seedRun root seed, overriding the model's own seed:
--randomizeUse a fresh random seed instead of the model's
--schema-only / --data-onlyRender only DDL, or only row databoth
--batch-sizeRows per INSERT/COPY batch1000
--no-copyForce multi-row INSERT for PostgreSQL instead of COPY
--mssql-production-style[dbo]. names, named clustered PK, ON [PRIMARY], ANSI header (requires --dialect mssql)
--mssql-goEmit GO every N INSERT batches (requires --dialect mssql)
--checkValidate the model and exit; writes no SQL
--dry-runCompile and report resolved row counts; writes no SQL
--verifyGenerate to protected temp storage, audit, then publish atomically
--explainInclude inference reason/confidence/precedence in the report
--strictTreat model warnings as errors
--jsonOutput the report as JSON (owns stdout)
--quietSuppress the non-JSON summary report

Query Options

FlagDescriptionDefault
-f, --formatOutput format: table, json, jsonl, csv, tsvtable
-o, --outputWrite output to file instead of stdoutstdout
-d, --dialectSQL dialect: mysql, postgres, sqlite, mssqlauto-detect
-i, --interactiveStart interactive REPL session
--diskUse disk-based storage (for large dumps >2GB)auto
--cacheCache imported database for repeated queries
-t, --tablesOnly import specific tables (comma-separated)all
--memory-limitMemory limit for DuckDB (e.g., "4GB")
--timingShow query execution time
-p, --progressShow import progress
--list-cacheList cached databases
--clear-cacheClear all cached databases

REPL commands:

  • .tables — List all tables
  • .schema [table] — Show schema (all tables or specific table)
  • .describe <table> — Describe a specific table
  • .format <fmt> — Set output format (table, json, csv, tsv)
  • .count <table> — Count rows in a table
  • .sample <table> [n] — Show sample rows (default: 10)
  • .export <file> <query> — Export query results to file
  • .exit — Exit the REPL

Performance

See BENCHMARKS.md for detailed comparisons.

Testing

# Unit tests
cargo nextest run

# Verify against real-world SQL dumps (MySQL, PostgreSQL, WordPress, etc.)
just verify-realworld

AI Agent Integration

sql-splitter includes documentation optimized for AI agents:

  • llms.txt - Generated LLM-friendly documentation following the llmstxt.org specification
  • Agent Skill - Claude Code / Amp skill for automatic tool discovery

Install the skill in Claude Code / Amp:

amp skill add helgesverre/sql-splitter

License

MIT — see LICENSE.md