API Reference

Complete auto-generated API documentation for all Datarax modules. Each page is built from source docstrings using mkdocstrings.

Module Groups

Section	Description
Core Components	Data structures, operators, samplers, and module base classes
Operators	Data transformation operators (image, composition strategies)
Sources	Data source implementations (HuggingFace, TFDS, ArrayRecord, Memory)
DAG	Directed Acyclic Graph pipeline execution engine
Distributed	Multi-device sharding, data parallelism, device mesh
Benchmarking	Timing, profiling, regression detection, monitoring
Monitoring	Runtime metrics collection, callbacks, reporters
Checkpoint	Pipeline state saving and restoration
Config	Configuration loading, schema, environment, registry
Samplers	Data sampling strategies (sequential, shuffle, range, epoch-aware)
Sharding	Array and per-process sharding modules
Batching	Batch assembly and collation
Control	Prefetching and iteration control
Memory	Shared-memory management for multi-process transforms
Performance	Roofline (calibrax), goodput, synchronization
Types & Protocols	Type aliases and checkpointable protocols
Utilities	PRNG, pytree utilities, external helpers
CLI	Command-line interface entry points

How It Works

API pages use the ::: directive to auto-generate documentation from Python docstrings:

::: datarax.core.operator

This pulls class signatures, method documentation, type annotations, and inheritance hierarchies directly from the source code. Google-style docstrings are parsed automatically.