orc-rust

A native Rust implementation of the Apache ORC file format, providing API's to read data into Apache Arrow in-memory arrays.

See the documentation for examples on how to use this crate.

Supported features

This crate currently only supports reading ORC files into Arrow arrays. Write support is planned (see Roadmap). The below features listed relate only to reading ORC files. At this time, we aim to support the ORCv1 specification only.

• Read synchronously & asynchronously (using Tokio)
• All compression types (Zlib, Snappy, Lzo, Lz4, Zstd)
• All ORC data types
• All encodings
• Rudimentary support for retrieving statistics
• Retrieving user metadata into Arrow schema metadata

Roadmap

The long term vision for this crate is to be feature complete enough to be donated to the arrow-rs project.

The following lists the rough roadmap for features to be implemented, from highest to lowest priority.

• Performance enhancements
• Predicate pushdown
• Row indices
• Bloom filters
• Write from Arrow arrays
• Encryption

A non-Arrow API interface is not planned at the moment. Feel free to raise an issue if there is such a use case.

Version compatibility

No guarantees are provided about stability across versions. We will endeavour to keep the top level API's (ArrowReader and ArrowStreamReader) as stable as we can, but other API's provided may change as we explore the interface we want the library to expose.

Versions will be released on an ad-hoc basis (with no fixed schedule).

Mapping ORC types to Arrow types

The following table lists how ORC data types are read into Arrow data types:

ORC Data Type	Arrow Data Type	Notes
Boolean	Boolean
TinyInt	Int8
SmallInt	Int16
Int	Int32
BigInt	Int64
Float	Float32
Double	Float64
String	Utf8
Char	Utf8
VarChar	Utf8
Binary	Binary
Decimal	Decimal128
Date	Date32
Timestamp	Timestamp(Nanosecond, None)	¹
Timestamp instant	Timestamp(Nanosecond, UTC)	¹
Struct	Struct
List	List
Map	Map
Union	Union(_, Sparse)	²

¹: ArrowReaderBuilder::with_schema allows configuring different time units or decoding to Decimal128(38, 9) (i128 of non-leap nanoseconds since UNIX epoch). Overflows may happen while decoding to a non-Seconds time unit, and results in OrcError. Loss of precision may happen while decoding to a non-Nanosecond time unit, and results in OrcError. Decimal128(38, 9) avoids both overflows and loss of precision.

²: Currently only supports a maximum of 127 variants

Contributing

All contributions are welcome! Feel free to raise an issue if you have a feature request, bug report, or a question. Feel free to raise a Pull Request without raising an issue first, as long as the Pull Request is descriptive enough.

Some tools we use in addition to the standard cargo that require installation are:

• taplo
• typos

cargo install typos-cli
cargo install taplo-cli

# Building the crate
cargo build

# Running the test suite
cargo test

# Simple benchmarks
cargo bench

# Formatting TOML files
taplo format

# Detect any typos in the codebase
typos

To regenerate/update the proto.rs file, execute the regen.sh script.

./regen.sh