TPC-H Data Generator CLI

tpchgen-cli is a high-performance, parallel TPC-H data generator command line tool

This tool is more than 10x faster than the next fastest TPCH generator we know of (duckdb). On a 2023 Mac M3 Max laptop, it easily generates data faster than can be written to SSD. See BENCHMARKS.md for more details on performance and benchmarking.

• See the tpchgen README.md for project details
• Watch this awesome demo by @alamb to see tpchgen-cli in action
• Read the companion blog post in the Datafusion blog to learn about the project's history
• Try it yourself by following the instructions below

Try with uvx

uvx tpchgen-cli parquet -s 1 --output-dir /tmp/tpch

Install with pip

python -m pip install tpchgen-cli

Install via Rust

Install Rust and compile

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
RUSTFLAGS='-C target-cpu=native' cargo install tpchgen-cli

Examples

# Scale Factor 10, all tables, in Apache Parquet format in the current directory
# (3.6GB, 8 files, 60M lineitem rows, in 5 seconds on a modern laptop)
tpchgen-cli parquet -s 10

# Scale Factor 10, all tables, in `tbl`(csv like) format in the `sf10` directory
# (10GB, 8 files, 60M lineitem rows)
# Note: `tpchgen-cli tbl` also works explicitly
tpchgen-cli -s 10 --output-dir sf10

# Scale Factor 1000, lineitem table, in Apache Parquet format in sf1000 directory, 
# 20 part(itions), 100MB row groups
# (220GB, 20 files, 6B lineitem rows, 3.5 minutes on a modern laptop)
tpchgen-cli parquet -s 1000 --tables lineitem --parts 20 --row-group-bytes=100000000 --output-dir sf1000

# Scale Factor 10, partition 2 and 3 of 10 in sf10 directory
#
# partitioned/
# ├── lineitem
# │   ├── lineitem.2.tbl
# │   └── lineitem.3.tbl
# └── orders
#    ├── orders.2.tbl
#    └── orders.3.tbl
#     
for PART in `seq 2 3`; do
  tpchgen-cli --tables lineitem,orders --scale-factor=10 --output-dir partitioned --parts 10 --part $PART
done

By default tpchgen-cli shows a per-table progress bar on stderr while data is generated. Pass --no-progress to disable it (it is also disabled automatically when --quiet is set, when --stdout is used, or when stderr is not a terminal, e.g. in CI logs).

Performance

Scale Factor	tpchgen-cli	DuckDB	DuckDB (proprietary)
1	0:02.24	0:12.29	0:10.68
10	0:09.97	1:46.80	1:41.14
100	1:14.22	17:48.27	16:40.88
1000	10:26.26	N/A (OOM)	N/A (OOM)

• DuckDB (proprietary) is the time required to create TPCH data using the proprietary DuckDB format
• Creating Scale Factor 1000 data in DuckDB required 647 GB of memory, which is why it is not included in the table above.

Times to create TPCH tables in Parquet format using tpchgen-cli and duckdb for various scale factors.

Deprecation Notice

--format, --parquet-compression, and --parquet-row-group-bytes are deprecated as of v3.x and will be removed in v4.0.0. Use subcommands instead:

# Before
tpchgen-cli --format=parquet --parquet-compression=ZSTD(1) -s 10

# After
tpchgen-cli parquet --compression=ZSTD(1) -s 10