README.md

July 16, 2026 · View on GitHub

Fast OpenStreetMap PBF reader and writer for Rust

Rust library and CLI for reading, writing, and transforming OpenStreetMap PBF files. The full planet (87 GB, 11.6B elements) processes on a 32 GB machine - every command, bounded memory, no compromises.

Developed on Linux, untested elsewhere.

Built with LLMs. See LLM.md.

Features

Read .osm.pbf files sequentially, in parallel (par_map_reduce), or with a 3-stage pipelined decoder
Write valid .osm.pbf files with HeaderBuilder, BlockBuilder, and PbfWriter - dense node packing, delta encoding, configurable compression (none, zlib, zstd)
Blob passthrough for copying unmodified blobs during merge/cat - kernel-space copy eliminates userspace buffer overhead
Blob indexdata - embeds element type + ID range + spatial bbox in BlobHeader for O(1) blob classification without decompression
Blob tag index - per-blob tag key metadata enables skipping decompression of blobs that provably lack required tag keys
O_DIRECT I/O - optional linux-direct-io feature bypasses the page cache for planet-scale reads and writes
io_uring writes - optional linux-io-uring feature for maximum throughput when I/O-bound

Planet scale

Every command listed below runs on the full planet on normal hardware. Measured on an AMD Ryzen 9 5900X, 32 GB DDR4 RAM, NVMe SSD, with ~28 GB available RAM. Input: 87 GB indexed planet PBF, 11.6B elements.

Command	Wall	Peak anon RSS
`add-locations-to-ways --index-type external`	9m52s	12.0 GB
`apply-changes` (OSC-only daily diff, `zstd:1`)	4m29s	2.4 GB
`apply-changes --locations-on-ways` (daily diff)	2m15s	~3.3 GB
`build-geocode-index`	5m47s	~25 GB
`cat` (indexdata generation)	1m27s	~200 MB
`cat --clean version`	5m34s	750 MB
`cat --dedupe`	2h13m	1.4 GB
`cat --type way` (raw passthrough)	45s	10 MB
`check --ids` (streaming, default)	57s	504 MB
`check --ids --full`	1m10s	2.22 GB
`check --refs`	1m0s	2.17 GB
`diff -j 16` (two independent 47-day-apart planets, text)	3m48s	586 MB
`diff --format osc -j 16` (two independent 47-day-apart planets)	4m54s	634 MB
`extract --complete` (Europe bbox)	3m42s	4.7 GB
`extract --simple` (Europe bbox)	3m42s	3.0 GB
`extract --smart` (Europe bbox)	4m28s	11.17 GB
`getid`	6.1s	27 MB
`getid --invert`	1m31s	102 MB
`getparents`	22s	506 MB
`inspect`	6.5s	5 MB
`inspect --extended`	13m41s	34 MB
`inspect --nodes -j 16`	56.8s	410 MB
`inspect --tags -j 16`	2m50s	17.5 GB
`merge-changes --osc-seq N` (1-OSC daily)	44s	2 MB
`merge-changes --osc-range A..B` (7-OSC, ~1 week of dailies)	54s	2 MB
`multi-extract --simple -c` (5 regions, Europe bbox)	14m44s	9.4 GB
`multi-extract --smart -c` (5 regions, Europe bbox)	13m58s	22.9 GB
`renumber`	3m11s	3.3 GB
`sort` (already-sorted input)	2m27s	476 MB
`tags-filter` (default two-pass, `w/highway=primary`)	1m55s	2.6 GB
`tags-filter -i w/highway=primary` (invert-match)	7m57s	7.0 GB
`tags-filter -R highway=primary`	52s	688 MB
`time-filter` (snapshot, cutoff 2024-01-01)	4m24s	812 MB

Three commands write temp files to the output's parent directory: add-locations-to-ways --index-type external (~246 GB), diff (parallel by default; ~30 GB text shards at planet), diff --format osc (~45 GB XML shards). The others are scratch-free. diff -j 1 restores the scratch-free sequential path if temp disk is scarce.

History-shaped input (per-element version history and visibility, consumed by time-filter on multi-version files) is functional but deliberately outside this validation surface: no history dataset is configured or benchmarked, and the table above says nothing about it.

Per-command phase breakdowns are in reference/performance.md; per-command optimization arcs and retired phase breakdowns at older architectures are in reference/performance-history.md. Note that recorded results always track the latest git head and may not match the released version.

The goal for pbfhogg 1.0 is that every CLI command must be planet-scale safe on a 32GB RAM host (28-ish free GB.)

Usage

[dependencies]
pbfhogg = "0.5"

use pbfhogg::{ElementReader, Element};

let reader = ElementReader::from_path("input.osm.pbf")?;

if reader.header().is_sorted() {
    println!("PBF is sorted by type then ID");
}

reader.for_each(|element| {
    if let Element::Way(way) = element {
        // process way
    }
})?;
# Ok::<(), std::io::Error>(())

Writing

use pbfhogg::write::block_builder::{HeaderBuilder, BlockBuilder};
use pbfhogg::write::writer::{PbfWriter, Compression};

let header_bytes = HeaderBuilder::new()
    .bbox(9.0, 54.0, 13.0, 58.0)
    .sorted()
    .build()?;
let mut writer = PbfWriter::to_path("output.osm.pbf".as_ref(), Compression::default(), &header_bytes)?;

let mut bb = BlockBuilder::new();
bb.add_node(1, 556_761_000, 125_683_000, [("name", "Copenhagen")], None);
if let Some(bytes) = bb.take()? {
    writer.write_primitive_block(bytes)?;
}
writer.flush()?;
# Ok::<(), std::io::Error>(())

Read modes

Method	Order	Use case
`for_each`	File order	Sequential processing, order-dependent consumers
`for_each_pipelined`	File order	Fastest ordered read (parallel decompression)
`for_each_block_pipelined`	File order	Consumers that need owned `PrimitiveBlock` for parallel processing
`into_blocks_pipelined`	File order	Loop control: early exit, zipping two files
`par_map_reduce`	Arbitrary	Aggregation where order doesn't matter

CLI

pbfhogg inspect <file>                    File inspection (blocks, ordering, counts)
pbfhogg inspect --indexed <file>          Check if PBF has indexdata (exit code 0/1)
pbfhogg inspect tags <file>               Tag key=value frequencies
pbfhogg check <file> --ids                Validate ID uniqueness and ordering
pbfhogg check <file> --refs               Validate referential integrity
pbfhogg cat <files...> -o <out>           Concatenate PBFs (-t node,way,relation to filter)
pbfhogg sort <file> -o <out>              Sort into standard order (nodes, ways, relations by ID)
pbfhogg repack <file> -o <out>            Re-encode at a configurable --elements-per-blob N cap
pbfhogg degrade <file> -o <out>           Adversarial PBF generator
pbfhogg renumber <file> -o <out>          Renumber all IDs sequentially, remap cross-references
pbfhogg extract <file> -o <out> -b <bbox> Extract by bounding box
pbfhogg extract <file> -o <out> -p <geo>  Extract by GeoJSON polygon
pbfhogg extract <file> -c <config>        Multi-extract from JSON config
pbfhogg add-locations-to-ways <f> -o <o>  Embed node coordinates in ways
pbfhogg apply-changes <base> <osc> -o <o> Apply OSC diff (--locations-on-ways)
pbfhogg merge-changes <oscs...> -o <out>  Merge multiple OSC files (--simplify)
pbfhogg diff <old> <new>                  Compare two PBFs (-v verbose, --format osc)
pbfhogg tags-filter <file> -o <out> <exp> Filter by tag expressions (PBF or OSC input)
pbfhogg getid <file> -o <out> <ids>       Extract elements by ID (--invert to remove)
pbfhogg getparents <file> -o <out> <ids>  Find ways/relations referencing given IDs
pbfhogg time-filter <file> -o <out> <ts>  Filter history PBF to a timestamp
pbfhogg build-geocode-index <f> -d <dir>  Build reverse geocoding index

All write commands accept --compression (none, zlib, zstd, or with level: zlib:9). Default is zlib:6 for osmium interop. For internal pipelines that don't need osmium/JOSM compatibility, zstd:1 can be a substantial wall-time win where output compression is what limits the pipeline - measured ≈ −14 % on Europe add-locations-to-ways --index-type external (270.8 s zlib:6 at 0dc8ae1 → 233.3 s zstd:1 at 4fc8e35, UUID e2fba1bf) by relieving consumer/compression saturation in stage 4, at similar output size. This does not generalise by dataset size: the same command at planet scale is not compression-bound, and zstd:1 buys ≤5 % of the streaming phase there for ~80 % less compression CPU, at ~4 % larger output (2026-07-14, UUIDs 0e9d93cc vs ed5dd6c5). Measure your own pipeline before assuming the Europe number transfers. Commands that benefit from indexdata will error without it - pass --force to proceed (slower), or generate indexed PBFs with pbfhogg cat input.osm.pbf -o indexed.osm.pbf.

Applying accumulated dailies: when several OSC diffs have piled up (say a week of dailies), squash them with merge-changes first and run apply-changes once on the result, instead of applying each diff in turn:

pbfhogg merge-changes day1.osc.gz day2.osc.gz ... day7.osc.gz -o week.osc.gz
pbfhogg apply-changes planet.osm.pbf week.osc.gz -o updated.osm.pbf

One merge plus one apply is much cheaper than N applies: each apply-changes pass rewrites the whole base PBF, while the squash itself is cheap - 55 s at planet scale for 7 dailies (see the table above; down from 4m27s since the parallel merge drain, commit 99057fa). Adding --simplify keeps only the last change per object across the merged diffs, so the single apply can also touch fewer elements than the individual applies would in sequence.

See docs/cli/commands.md for detailed command documentation, docs/guide/advanced.md for O_DIRECT, io_uring, and index type details.

Performance

Read throughput - 59M elements in Denmark (461 MB), best of 3 (commit 90df51f):

Tool	Mode	Time
pbfhogg	parallel	0.31s
osmpbf 0.3	parallel	0.53s
pbfhogg	pipelined	1.3s
Planetiler 0.10	parallel	2.0s
pbfhogg	sequential	2.8s
osmpbf 0.3	sequential	5.6s
osmium 1.19	cat → opl	5.7s

CLI commands vs osmium - Denmark (487 MB, commit 6fc1283):

Command	pbfhogg	osmium	speedup
sort (sorted, indexdata)	0.7s	11.6s	17x
apply-changes (indexdata)	0.6s	7.2s	12x
cat --type way (raw passthrough)	0.24s	2.22s	9.3x
extract --smart (Tokyo bbox, Japan)	4.7s	13.4s	2.9x
tags-filter highway=primary -R	0.2s	0.56s	2.8x
add-locations-to-ways	9.7s	12.1s	1.2x

All CLI commands are cross-validated against osmium on Denmark (brokkr verify). See reference/osmium-parity.md for the full comparison matrix, DEVIATIONS.md for intentional behavioral differences, and CORRECTNESS.md for parser/encoder edge cases. Detailed per-command benchmarks and phase breakdowns are in reference/performance.md.

System: AMD Ryzen 9 5900X (12c/24t), 32 GB DDR4, NVMe SSD, Linux 6.18. Measured with brokkr bench.

Acknowledgements

pbfhogg started as a fork of osmpbf by Thomas Bruggemann. osmium-tool and libosmium by Jochen Topf are the reference implementation - pbfhogg's CLI covers the same use cases, cross-validated against osmium using brokkr.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.