s3dlio Python API Guide

May 13, 2026 · View on GitHub

Version: 0.9.100
Last Updated: May 12, 2026

Installation
Quick Start
S3 Connection Configuration
Architecture
Core Storage Operations
Zero-Copy Data Flow
Batch Operations
Multi-Backend Support
Multi-Endpoint Load Balancing
Streaming API
AI/ML Integration
Object Data Loader (General-Purpose) ✨ New in v0.9.100
Parquet DataLoader ✨ New in v0.9.98
Data Generation (NPZ / NPY)
s3torchconnector Compatibility
Checkpoint System
Performance & Threading
Advanced Features
API Reference
Migration Guide

Installation

From Wheel (Recommended)

# Build and install from source
cd s3dlio
./build_pyo3.sh
./install_pyo3_wheel.sh

# Verify installation
python -c "import s3dlio; print(s3dlio.__version__)"

From PyPI

pip install s3dlio

Requirements

Python 3.12+
Rust 1.90+ (for building from source)
Optional: PyTorch, JAX, or TensorFlow for ML integration

Quick Start

import s3dlio

# Initialize logging (optional)
s3dlio.init_logging("info")  # Options: trace, debug, info, warn, error

# Put data to storage — works with S3, Azure, GCS, local filesystem
s3dlio.put_bytes("s3://my-bucket/data.bin", b"Hello, World!")
s3dlio.put_bytes("file:///tmp/local.bin", b"Local data")

# Get data — returns BytesView (zero-copy from Rust memory)
data = s3dlio.get("s3://my-bucket/data.bin")
print(len(data))       # 13
print(bytes(data))     # b'Hello, World!'

# Range request — server-side, only fetches needed bytes
chunk = s3dlio.get_range("s3://my-bucket/data.bin", offset=0, length=5)

# List objects — returns full URIs
objects = s3dlio.list("s3://my-bucket/prefix/")

# Metadata
metadata = s3dlio.stat("s3://my-bucket/data.bin")
print(f"Size: {metadata['size']} bytes")

# Check existence
if s3dlio.exists("s3://my-bucket/data.bin"):
    s3dlio.delete("s3://my-bucket/data.bin")

Supported URI Schemes:

Scheme	Example	Description
`s3://`	`s3://bucket/key`	Amazon S3, MinIO, Ceph, VAST
`gs://`	`gs://bucket/key`	Google Cloud Storage
`az://`	`az://account/container/key`	Azure Blob Storage
`file://`	`file:///path/to/file`	Local filesystem
`direct://`	`direct:///path/to/file`	Direct I/O (O_DIRECT) — bypasses the OS page cache. v0.9.95: correctly routed through `ConfigurableFileSystemObjectStore::with_direct_io()` in `get_many()` (previously silently used buffered I/O).

configure_s3() is the Python equivalent of the CLI's --endpoint-url, --region, and --ca-bundle global flags. It sets the corresponding environment variables and clears the internal store cache so the next operation creates a fresh connection with the new settings.

Signature

s3dlio.configure_s3(
    endpoint_url: str | None = None,
    region:       str | None = None,
    ca_bundle:    str | None = None,
) -> None

Parameter	Env var set	CLI equivalent	Description
`endpoint_url`	`AWS_ENDPOINT_URL`	`--endpoint-url`	Full URL of the S3-compatible server, e.g. `"https://minio.corp:9000"`
`region`	`AWS_DEFAULT_REGION`	`--region`	AWS region name, e.g. `"us-east-1"`
`ca_bundle`	`AWS_CA_BUNDLE`	`--ca-bundle`	Filesystem path to a PEM CA certificate bundle for TLS

Example

import s3dlio

# Must be called BEFORE the first S3 operation (see critical note below)
s3dlio.configure_s3(
    endpoint_url="https://172.16.1.40:9000",
    region="us-east-1",
    ca_bundle="/etc/ssl/certs/my-ca.pem",
)

# All S3 operations now use the configured endpoint:
keys  = s3dlio.list("s3://my-bucket/", recursive=False)
data  = s3dlio.get("s3://my-bucket/file.bin")
s3dlio.put_bytes("s3://my-bucket/out.bin", data)
info  = s3dlio.stat("s3://my-bucket/file.bin")
s3dlio.delete("s3://my-bucket/file.bin")

Scope

configure_s3() applies to all S3 operations: list, get, get_range, get_many, put_bytes, put_many, stat, exists, delete, upload, download, mp_get, create_bucket, delete_bucket, and all async variants. It does not affect Azure or GCS backends (those use their own environment variables — see Authentication).

⚠️ Critical: Call Before the First S3 Operation

The underlying AWS SDK client is a process-global singleton (OnceCell) that is initialised exactly once — on the first S3 call — and cannot be reconfigured afterwards.

import s3dlio

# ✅ CORRECT — configure before any S3 operation
s3dlio.configure_s3(endpoint_url="https://minio:9000", region="us-east-1")
data = s3dlio.get("s3://bucket/key")   # uses the configured endpoint

# ❌ WRONG — SDK client already initialised; configure_s3() has no effect
data = s3dlio.get("s3://bucket/key")   # initialises client with default settings
s3dlio.configure_s3(endpoint_url="https://minio:9000")  # too late — ignored

configure_s3() does clear the store cache (STORE_CACHE), so if you only need to switch endpoints between operations (and the AWS SDK client settings are compatible with both), calling it mid-script will cause the next operation to build a new store against the new endpoint.

Relationship to Environment Variables

configure_s3() is syntactic sugar for os.environ assignments plus a cache flush. Both approaches are equivalent:

# These two blocks are identical in effect:

# Option A — configure_s3()
s3dlio.configure_s3(endpoint_url="https://minio:9000", region="us-east-1")

# Option B — environment variables directly
import os
os.environ["AWS_ENDPOINT_URL"]    = "https://minio:9000"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"
# (no cache flush needed if called before any S3 op)

Prefer configure_s3() when you want the cache flush guarantee.

Architecture

Runtime Model (v0.9.50)

s3dlio uses an io_uring-style submit pattern for all Python API calls:

Python Thread → spawn(async work) → channel.recv() → result

SUBMIT: The calling thread spawns the async future onto a dedicated global Tokio runtime
PROCESS: Runtime worker threads handle the async I/O
COMPLETE: Result flows back through std::sync::mpsc channel

This design is fully thread-safe. You can call any s3dlio function from:

Python ThreadPoolExecutor (16, 64, 128+ threads)
PyTorch DataLoader worker processes
Any plain OS thread

The calling thread blocks on channel recv (NOT on block_on), so there are no Tokio runtime conflicts.

Global Client Cache

s3dlio maintains a process-global DashMap<StoreKey, Arc<dyn ObjectStore>> cache:

Key: (scheme, endpoint, region) — NOT bucket-specific
Hit rate: >99% in typical workloads (<100ns lookup)
Thread-safe: Lock-free concurrent read/write (DashMap sharded locking)
Automatic: No manual client passing — first call creates, all subsequent calls reuse

# These all reuse the SAME underlying Rust ObjectStore client:
s3dlio.put_bytes("s3://bucket-a/key1", data1)  # Creates client
s3dlio.put_bytes("s3://bucket-b/key2", data2)  # Reuses client (same endpoint)
s3dlio.get("s3://bucket-c/key3")               # Reuses client

Configuration

# Control Tokio worker thread count (default: num_cpus)
export S3DLIO_WORKER_THREADS=16

Core Storage Operations

put_bytes() — Upload Data

# Upload bytes to any backend
s3dlio.put_bytes("s3://bucket/key", b"data")

# Upload from file
with open("local_file.bin", "rb") as f:
    s3dlio.put_bytes("s3://bucket/key", f.read())

# Async version (for use with asyncio)
await s3dlio.put_bytes_async("s3://bucket/key", b"data")

Data path: Python bytes → Bytes::copy_from_slice (one unavoidable copy from Python heap) → Bytes (Arc-counted, zero-copy through upload pipeline).

get() — Download Data

# Returns BytesView — zero-copy wrapper around Rust Bytes
data = s3dlio.get("s3://bucket/key")

# BytesView supports Python buffer protocol:
mv = memoryview(data)                          # Zero-copy memoryview
arr = numpy.frombuffer(data, dtype=np.uint8)   # Zero-copy NumPy array
raw = bytes(data)                              # Creates a copy (only if needed)
print(len(data))                               # Size in bytes

get_range() — Server-Side Range Request

# Fetch only bytes 1024-5119 from the server (saves bandwidth)
chunk = s3dlio.get_range("s3://bucket/key", offset=1024, length=4096)

# Fetch from offset to end of object
tail = s3dlio.get_range("s3://bucket/key", offset=1024)

list() — List Objects

# List all objects under prefix (returns full URIs)
uris = s3dlio.list("s3://bucket/prefix/")
# ['s3://bucket/prefix/file1.dat', 's3://bucket/prefix/file2.dat', ...]

# Recursive listing
uris = s3dlio.list("s3://bucket/prefix/", recursive=True)

# With glob pattern filter
uris = s3dlio.list("s3://bucket/prefix/", pattern="*.npz")

stat() — Get Metadata

meta = s3dlio.stat("s3://bucket/key")
print(meta['size'])           # int: object size in bytes
print(meta['last_modified'])  # str: timestamp
print(meta['etag'])           # str: ETag/hash

# Async version
meta = await s3dlio.stat_async("s3://bucket/key")

# Batch stat (async)
metas = await s3dlio.stat_many_async(["s3://bucket/a", "s3://bucket/b"])

exists() — Check Existence

if s3dlio.exists("s3://bucket/key"):
    print("Found")

# Async version
found = await s3dlio.exists_async("s3://bucket/key")

delete() — Remove Object

s3dlio.delete("s3://bucket/key")

mkdir() — Create Directory / Prefix

s3dlio.mkdir("file:///data/output/subdir")
await s3dlio.mkdir_async("s3://bucket/prefix/")

Zero-Copy Data Flow

s3dlio's get() returns BytesView, a Python object backed by Rust Bytes (Arc-counted reference). No data is copied when passing through the Rust runtime:

Rust async I/O → Bytes (Arc) → channel → BytesView (Python buffer protocol)

Using BytesView

data = s3dlio.get("s3://bucket/model_weights.bin")

# 1. Zero-copy memoryview (fastest — no allocation)
mv = memoryview(data)

# 2. Zero-copy NumPy array
import numpy as np
weights = np.frombuffer(data, dtype=np.float32)

# 3. Zero-copy PyTorch tensor
import torch
tensor = torch.frombuffer(data, dtype=torch.float32)

# 4. Convert to bytes (creates copy — only when needed)
raw = bytes(data)

Performance Impact

Operation	Copies	Notes
`s3dlio.get()`	0	Returns BytesView (Rust Bytes Arc)
`memoryview(data)`	0	Buffer protocol, no allocation
`np.frombuffer(data)`	0	Shares Rust memory
`torch.frombuffer(data)`	0	Shares Rust memory
`bytes(data)`	1	Explicit copy to Python heap
`s3dlio.put_bytes(uri, pydata)`	1	Unavoidable Python→Rust copy

Batch Operations

put_many() — Batch Upload (v0.9.50+)

Upload multiple objects in a single call with parallel execution:

# List of (uri, data) tuples
items = [
    ("s3://bucket/file1.bin", b"data1"),
    ("s3://bucket/file2.bin", b"data2"),
    ("s3://bucket/file3.bin", b"data3"),
]
s3dlio.put_many(items)

# Async version
await s3dlio.put_many_async(items)

get_many() — Batch Download

uris = ["s3://bucket/file1", "s3://bucket/file2", "s3://bucket/file3"]
results = s3dlio.get_many(uris, workers=64)

# Async version
results = await s3dlio.get_many_async(uris)

# O_DIRECT reads — bypass page cache (v0.9.95: now correctly uses O_DIRECT)
results = s3dlio.get_many(["direct:///data/file1.npz", "direct:///data/file2.npz"])

v0.9.95 fix: direct:// URIs in get_many() previously fell through to the standard buffered tokio::fs::read() path — O_DIRECT was never engaged. Fixed in v0.9.95: direct:// reads now go through ConfigurableFileSystemObjectStore::with_direct_io(), correctly bypassing the OS page cache.

upload() / download() — Bulk File Transfer

# Upload entire directory
s3dlio.upload(
    src_uri="file:///data/files/",
    dest_uri="s3://bucket/uploads/",
)

# Download prefix to local directory
s3dlio.download(
    src_uri="s3://bucket/downloads/",
    dest_dir="/data/output/"
)

mp_get() — Multi-Process GET

For maximum throughput with very large datasets:

result = s3dlio.mp_get(
    uri="s3://bucket/dataset/",
    procs=8,           # 8 worker processes
    jobs=128,          # 128 concurrent ops per process
    num=10000,         # 10,000 objects
    template="data_{}.bin"
)
print(f"Throughput: {result['throughput_mb_s']} MB/s")

Multi-Backend Support

Authentication

Amazon S3 / S3-Compatible:

import os
os.environ['AWS_ACCESS_KEY_ID'] = 'your_key'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'your_secret'
os.environ['AWS_REGION'] = 'us-east-1'
os.environ['AWS_ENDPOINT_URL'] = 'http://minio:9000'  # Optional: MinIO, Ceph, VAST

Google Cloud Storage:

gcloud auth application-default login
# Or: export GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json

Azure Blob Storage:

import os
os.environ['AZURE_STORAGE_ACCOUNT'] = 'myaccount'
os.environ['AZURE_STORAGE_KEY'] = 'mykey'
# Optional custom endpoint:
os.environ['AZURE_STORAGE_ENDPOINT'] = 'http://127.0.0.1:10000'  # Azurite

Backend	Endpoint Variable	Alternative
S3	`AWS_ENDPOINT_URL`	—
Azure	`AZURE_STORAGE_ENDPOINT`	`AZURE_BLOB_ENDPOINT_URL`
GCS	`GCS_ENDPOINT_URL`	`STORAGE_EMULATOR_HOST`

S3 Bucket Management

s3dlio.create_bucket("my-bucket")
s3dlio.delete_bucket("my-bucket")

Multi-Endpoint Load Balancing

Create stores that distribute operations across multiple endpoints (v0.9.14+):

import asyncio
import s3dlio

async def main():
    # From explicit URI list
    store = s3dlio.create_multi_endpoint_store(
        uris=["s3://bucket-1", "s3://bucket-2", "s3://bucket-3"],
        strategy="round_robin"  # or "least_connections"
    )
    
    # From template (expands {1...10})
    store = s3dlio.create_multi_endpoint_store_from_template(
        uri_template="s3://my-bucket-{1...10}",
        strategy="round_robin"
    )
    
    # From file (one URI per line)
    store = s3dlio.create_multi_endpoint_store_from_file(
        file_path="/path/to/endpoints.txt",
        strategy="least_connections"
    )
    
    # All operations are async
    await store.put("s3://bucket-1/data.bin", b"Hello")
    view = await store.get("s3://bucket-1/data.bin")      # BytesView
    view = await store.get_range("s3://bucket-1/x", 0, 1024)
    objects = await store.list("s3://bucket-1/", recursive=True)
    await store.delete("s3://bucket-1/data.bin")
    
    # Statistics
    print(store.endpoint_count())
    print(store.strategy())
    print(store.get_total_stats())
    print(store.get_endpoint_stats())

asyncio.run(main())

Strategies:

"round_robin" — Even distribution, lowest overhead
"least_connections" — Routes to endpoint with fewest active requests

Streaming API

For large uploads with compression and chunking:

options = s3dlio.PyWriterOptions()
options.compression = "zstd"       # none, zstd, gzip, lz4
options.compression_level = 3      # 1-22 for zstd
options.chunk_size = 4 * 1024 * 1024  # 4 MiB chunks

# Backend-specific writers
writer = s3dlio.create_s3_writer("s3://bucket/file.bin.zst", options)
writer = s3dlio.create_azure_writer("az://acct/ctr/file.bin.zst", options)
writer = s3dlio.create_filesystem_writer("file:///path/file.bin", options)
writer = s3dlio.create_direct_filesystem_writer("direct:///path/file.bin", options)

# Write in chunks, then finalize
writer.write(chunk1)
writer.write(chunk2)
stats = writer.finalize()
print(f"Wrote {stats['bytes_written']} bytes")

AI/ML Integration

PyTorch Datasets (Native)

from s3dlio import ObjectStoreMapDataset, ObjectStoreIterableDataset
from torch.utils.data import DataLoader

# Map-style (random access)
dataset = ObjectStoreMapDataset(uri="s3://bucket/train/", pattern="*.npz")
item = dataset[0]  # Fetch by index

# Iterable (streaming)
dataset = ObjectStoreIterableDataset(uri="s3://bucket/train/", shuffle=True)

# With DataLoader
loader = DataLoader(dataset, batch_size=32, num_workers=4)
for batch in loader:
    pass  # Train

JAX Integration

from s3dlio import JaxIterable

jax_iter = JaxIterable(uri="gs://bucket/train/", batch_size=32)
for batch in jax_iter:
    pass  # JAX training

TensorFlow Integration

from s3dlio import make_tf_dataset

ds = make_tf_dataset(uri="s3://bucket/train/", batch_size=32, shuffle=True)
for batch in ds:
    pass  # TF training

Object Data Loader (General-Purpose)

New in v0.9.100. Sliding-window, URI-carrying object streaming built on Tokio's buffer_unordered. Works with all s3dlio URI schemes without code changes.

Key APIs: PyDataset.from_uris() · PyBytesAsyncDataLoader.items() · PyObjectDataLoaderSyncIter.collect_batch(n) · PyObjectItem

📖 Python_Data-Loader.md — Object DataLoader section

Parquet DataLoader

New in v0.9.98. Epoch-aware, row-group-streaming Parquet loader for AI/ML training. Works with all s3dlio backends. Epoch-2+ zero-refetch fast path; raw and Arrow IPC decode modes.

Key API: create_async_loader(uri, {"format": "parquet", ...})

📖 Python_Data-Loader.md — Parquet section · Complete Parquet Guide

Data Generation (NPZ / NPY)

Added in v0.9.94. Generate complete NumPy NPZ archives in Rust — single allocation, Rayon parallel fill, hardware-accelerated CRC32 — without holding the Python GIL. ~5× faster than numpy.savez() for large files.

`generate_npz_bytes()` — Build NPZ in Rust

import s3dlio

# Build a 140 MiB unet3d-style NPZ archive (~20 ms vs ~178 ms for numpy.savez)
npz = s3dlio.generate_npz_bytes(
    shape=[6053, 6053, 1],   # x.npy array shape
    dtype="<f4",             # float32 little-endian (default)
    num_samples=1,           # number of labels in y.npy (default)
)
print(type(npz))   # <class 's3dlio.BytesView'>
print(len(npz))    # ~146,800,000 bytes

Arguments:

Parameter	Type	Default	Description
`shape`	`list[int]`	required	Array shape for `x.npy`
`dtype`	`str`	`"<f4"`	NumPy dtype string, e.g. `"<f4"` (float32), `"<f8"` (float64)
`num_samples`	`int`	`1`	Length of label array `y.npy` (int64 zeros)

Returns: BytesView — zero-copy buffer supporting Python buffer protocol. Pass directly to MultipartUploadWriter.write(), put_bytes(), or any buffer-accepting API.

Upload NPZ zero-copy

import s3dlio, concurrent.futures

# Pre-generate once (shape is fixed per benchmark run)
buf = s3dlio.generate_npz_bytes(shape=[6053, 6053, 1])

# Upload 48 files concurrently — no GIL contention, no memcpy
def upload(i):
    with s3dlio.MultipartUploadWriter.from_uri(
        f"s3://my-bucket/train/sample_{i:06d}.npz"
    ) as w:
        w.write(buf)  # BytesView fast path: Arc clone only, GIL released

with concurrent.futures.ThreadPoolExecutor(max_workers=48) as pool:
    list(pool.map(upload, range(1000)))

Performance (28-core machine, loopback fake S3):

Method	write() latency (140 MiB)	Throughput N=48
`numpy.savez()` + `bytes()` + `put_bytes()`	~178 ms gen + ~109 ms write	~1,250 MiB/s
`generate_npz_bytes()` + `write(BytesView)`	~20 ms gen + ~6 ms write	~2,440 MiB/s

NPZ file structure

The generated archive matches NumPy's format exactly:

file.npz
├── x.npy   — float32 (or dtype) array, shape as specified, Rayon-filled random data
└── y.npy   — int64 array of shape (num_samples,), all zeros (class labels)

Load with standard NumPy:

import numpy as np
arrays = np.load("file.npz")
x = arrays["x"]   # float32 ndarray
y = arrays["y"]   # int64 ndarray

s3torchconnector Compatibility

s3dlio provides a drop-in replacement for AWS s3torchconnector. Change one import line:

# Before (s3torchconnector):
from s3torchconnector import S3IterableDataset, S3MapDataset, S3Checkpoint

# After (s3dlio — zero code changes needed):
from s3dlio.compat.s3torchconnector import S3IterableDataset, S3MapDataset, S3Checkpoint

S3IterableDataset

from s3dlio.compat.s3torchconnector import S3IterableDataset
from torch.utils.data import DataLoader

dataset = S3IterableDataset.from_prefix("s3://bucket/train/", region="us-east-1")

for item in dataset:
    print(item.bucket, item.key)
    data = item.read()  # Returns BytesView (zero-copy)
    # Use with torch.frombuffer(data, dtype=torch.uint8)

S3MapDataset

from s3dlio.compat.s3torchconnector import S3MapDataset

dataset = S3MapDataset.from_prefix("s3://bucket/train/", region="us-east-1")
item = dataset[0]       # Random access
item = dataset[-1]      # Negative indexing supported
print(len(dataset))     # Number of objects

S3Checkpoint

from s3dlio.compat.s3torchconnector import S3Checkpoint
import torch

checkpoint = S3Checkpoint(region="us-east-1")

# Save — torch.save() writes to in-memory buffer, then uploads via put_bytes()
with checkpoint.writer("s3://bucket/model.pt") as writer:
    torch.save(model.state_dict(), writer)

# Load — downloads to BytesView, wrapped in seekable BufferedReader
with checkpoint.reader("s3://bucket/model.pt") as reader:
    state_dict = torch.load(reader, weights_only=True)

S3Client (Low-Level)

from s3dlio.compat.s3torchconnector import S3Client, S3ClientConfig

client = S3Client(
    region="us-east-1",
    endpoint="http://minio:9000",
    s3client_config=S3ClientConfig(force_path_style=True)
)

# Upload
writer = client.put_object("my-bucket", "key.bin")
writer.write(b"data")
writer.close()

# Download (full object)
reader = client.get_object("my-bucket", "key.bin")
data = reader.read()  # BytesView (zero-copy)

# Download (range — server-side, saves bandwidth)
reader = client.get_object("my-bucket", "key.bin", start=0, end=1023)
chunk = reader.read()  # 1024 bytes

# List
for result in client.list_objects("my-bucket", "prefix/"):
    for info in result.object_info:
        print(info.key)

Advantages over s3torchconnector

Feature	s3torchconnector	s3dlio compat
S3	Yes	Yes
Azure / GCS / Local	No	Yes
Zero-copy reads	No	Yes (BytesView)
Server-side range requests	No	Yes (get_range)
Global client cache	No	Yes (DashMap)
Multi-endpoint load balancing	No	Yes

For complete migration instructions, see S3TORCHCONNECTOR_MIGRATION.md.

Checkpoint System

Save / Load

# Save checkpoint to any backend
s3dlio.save_checkpoint(
    uri="s3://bucket/checkpoints/epoch_10.bin",
    data={"epoch": 10, "model": model.state_dict()},
    compress=True
)

# Load checkpoint
ckpt = s3dlio.load_checkpoint("s3://bucket/checkpoints/epoch_10.bin")

# Load with validation
ckpt = s3dlio.load_checkpoint_with_validation(
    uri="s3://bucket/checkpoints/epoch_10.bin",
    expected_keys=["epoch", "model"]
)
model.load_state_dict(ckpt["model"])

Distributed Checkpointing

# Each rank saves its shard
s3dlio.save_distributed_shard(
    uri=f"s3://bucket/ckpt/shard_{rank}.bin",
    shard_data=local_state,
    rank=rank,
    world_size=world_size
)

# Rank 0 finalizes
if rank == 0:
    s3dlio.finalize_distributed_checkpoint(
        base_uri="s3://bucket/ckpt/",
        world_size=world_size
    )

Performance & Threading

Thread Safety (v0.9.50)

All s3dlio functions are fully thread-safe. Use ThreadPoolExecutor freely:

from concurrent.futures import ThreadPoolExecutor

def upload_one(i):
    s3dlio.put_bytes(f"s3://bucket/obj_{i}.bin", data)

# 16 threads uploading concurrently — no runtime conflicts
with ThreadPoolExecutor(max_workers=16) as pool:
    list(pool.map(upload_one, range(1000)))

Fixed in v0.9.50:

v0.9.27: Per-call Runtime::new() caused "dispatch failure" after ~40 objects
v0.9.40: GLOBAL_RUNTIME.block_on() caused "Cannot start a runtime from within a runtime"
v0.9.50: io_uring-style submit pattern — works from ANY thread context

Pre-Stat Size Caching

uris = s3dlio.list("s3://bucket/dataset/")
s3dlio.get_many_stats(uris, concurrency=100)

# Subsequent gets use cached sizes (2.5x faster)
for uri in uris:
    data = s3dlio.get(uri)

Performance Tips

Use put_many() for batch uploads — single round-trip, parallel execution
Use get_range() instead of get() + slicing — server-side range saves bandwidth
Use memoryview(data) not bytes(data) — avoid unnecessary copies
Pre-stat with get_many_stats() — eliminates per-get stat overhead
Use ThreadPoolExecutor — s3dlio handles concurrency internally via global runtime
Set S3DLIO_WORKER_THREADS — tune Tokio worker count for your workload
Enable range optimization for large objects — see below

Large Object Download Optimization

For workloads with large S3 objects (> 100 MB), enable parallel range downloads:

import os
os.environ['S3DLIO_ENABLE_RANGE_OPTIMIZATION'] = '1'
os.environ['S3DLIO_RANGE_THRESHOLD_MB'] = '64'  # Objects ≥ 64 MB use parallel ranges

import s3dlio

# Large objects automatically use parallel range downloads
data = s3dlio.get("s3://bucket/large-checkpoint.bin")  # 148 MB → 25-50% faster

How it works:

Disabled by default to avoid HEAD request overhead on small objects
When enabled, objects ≥ threshold are downloaded using parallel range requests
HEAD request determines size, then concurrent GET ranges fetch chunks in parallel
Best for: ML checkpoints, large datasets, multi-GB files
Not recommended for: Many small objects, latency-sensitive workloads

Tuning parameters:

import os

# Conservative (recommended for mixed workloads)
os.environ['S3DLIO_ENABLE_RANGE_OPTIMIZATION'] = '1'
os.environ['S3DLIO_RANGE_THRESHOLD_MB'] = '64'  # Default

# Aggressive (for very large objects > 500 MB)
os.environ['S3DLIO_ENABLE_RANGE_OPTIMIZATION'] = '1'
os.environ['S3DLIO_RANGE_THRESHOLD_MB'] = '128'
os.environ['S3DLIO_RANGE_CONCURRENCY'] = '32'  # More parallel requests
os.environ['S3DLIO_CHUNK_SIZE'] = '16777216'  # 16 MB chunks

Actual benchmark results (16x 148 MB objects, MinIO):

Threshold	Time	Throughput	Speedup
Disabled (baseline)	5.52s	429 MB/s (0.42 GB/s)	1.00x
8 MB	3.50s	676 MB/s (0.66 GB/s)	1.58x (58% faster)
16 MB	3.27s	725 MB/s (0.71 GB/s)	1.69x (69% faster)
32 MB	3.23s	732 MB/s (0.71 GB/s)	1.71x (71% faster)
64 MB (default)	3.14s	755 MB/s (0.74 GB/s)	1.76x (76% faster) 🏆
128 MB	3.22s	735 MB/s (0.72 GB/s)	1.71x (71% faster)

Summary:

64 MB threshold (default) is optimal, achieving 76% improvement
16-64 MB range all perform excellently (69-76% faster)
Even aggressive 8 MB threshold delivers 58% improvement with minimal downside
HEAD overhead: ~10-20ms (well amortized over parallel download)

Advanced Features

NPZ Files

# Read NumPy .npz files directly from any backend
data = s3dlio.get("s3://bucket/data.npz")
import numpy as np, io
arrays = np.load(io.BytesIO(bytes(data)))

TFRecord Index

s3dlio.create_tfrecord_index(
    input_path="s3://bucket/dataset.tfrecord",
    output_path="file:///tmp/dataset.idx"
)

Operation Logging

s3dlio.init_op_log("file:///tmp/operations.log")
# ... operations are automatically logged ...
s3dlio.finalize_op_log()

Python Convenience Helpers

# list_keys — returns relative keys (not full URIs)
keys = s3dlio.list_keys("s3://bucket/prefix/")  # ['file1.bin', 'file2.bin']

# list_full_uris — returns full URIs (alias for list())
uris = s3dlio.list_full_uris("s3://bucket/prefix/")

# get_object — same as get()
data = s3dlio.get_object("s3://bucket/key")

# stat_object — same as stat()
meta = s3dlio.stat_object("s3://bucket/key")

API Reference

Core Functions

Function	Description	Returns
`configure_s3(endpoint_url, region, ca_bundle)`	Configure S3 endpoint, region, and CA bundle; clears store cache — call before first S3 op	None
`put_bytes(uri, data)`	Upload bytes	None
`put_bytes_async(uri, data)`	Async upload	Coroutine
`put_many(items)`	Batch upload `[(uri, data), ...]`	None
`put_many_async(items)`	Async batch upload	Coroutine
`get(uri)`	Download object	BytesView
`get_range(uri, offset, length=None)`	Server-side range request	BytesView
`get_many(uris, workers)`	Parallel download	List[BytesView]
`get_many_async(uris)`	Async parallel download	Coroutine
`list(uri, recursive=False, pattern=None)`	List objects	List[str]
`stat(uri)`	Get metadata	dict
`stat_async(uri)`	Async metadata	Coroutine
`stat_many_async(uris)`	Batch metadata	Coroutine
`exists(uri)`	Check existence	bool
`exists_async(uri)`	Async existence check	Coroutine
`delete(uri)`	Delete object	None
`mkdir(uri)`	Create directory/prefix	None
`mkdir_async(uri)`	Async mkdir	Coroutine
`upload(src_uri, dest_uri)`	Bulk upload files	None
`download(src_uri, dest_dir)`	Bulk download files	None
`mp_get(uri, procs, jobs, num, template)`	Multi-process GET	dict
`create_bucket(name)`	Create S3 bucket	None
`delete_bucket(name)`	Delete S3 bucket	None

Data Generation Functions (v0.9.94+)

Function	Description	Returns
`generate_npz_bytes(shape, dtype="<f4", num_samples=1)`	Build NPZ archive in Rust (GIL-free, Rayon parallel fill)	BytesView

BytesView

Property/Method	Description	Zero-Copy
`len(view)`	Size in bytes	Yes
`memoryview(view)`	Python memoryview	Yes
`bytes(view)`	Convert to bytes	No (copy)
`np.frombuffer(view)`	NumPy array	Yes
`torch.frombuffer(view)`	PyTorch tensor	Yes

Multi-Endpoint API

Function	Description
`create_multi_endpoint_store(uris, strategy)`	Create from URI list
`create_multi_endpoint_store_from_template(template, strategy)`	Create from template
`create_multi_endpoint_store_from_file(path, strategy)`	Create from file

MultiEndpointStore Methods (all async):

Method	Returns
`put(uri, data)`	Coroutine[None]
`get(uri)`	Coroutine[BytesView]
`get_range(uri, offset, length)`	Coroutine[BytesView]
`list(uri, recursive)`	Coroutine[List[Dict]]
`delete(uri)`	Coroutine[None]
`get_endpoint_stats()`	List[Dict]
`get_total_stats()`	Dict
`endpoint_count()`	int
`strategy()`	str

Streaming Writers

Function	Description
`create_s3_writer(uri, opts)`	S3 streaming writer
`create_azure_writer(uri, opts)`	Azure streaming writer
`create_filesystem_writer(uri, opts)`	Local filesystem writer
`create_direct_filesystem_writer(uri, opts)`	Direct I/O writer

s3torchconnector Compat Classes

Class	Description
`S3IterableDataset.from_prefix(uri, region)`	Streaming dataset
`S3MapDataset.from_prefix(uri, region)`	Random access dataset
`S3Checkpoint(region)`	Checkpoint save/load
`S3Client(region, endpoint, config)`	Low-level client
`S3ClientConfig(force_path_style, max_attempts)`	Client configuration
`S3Item`	Item with `.bucket`, `.key`, `.read()`

AI/ML Integration

Class/Function	Framework
`ObjectStoreMapDataset(uri, pattern)`	PyTorch
`ObjectStoreIterableDataset(uri, shuffle)`	PyTorch
`JaxIterable(uri, batch_size)`	JAX
`make_tf_dataset(uri, batch_size)`	TensorFlow

Checkpoint System

Function	Description
`save_checkpoint(uri, data, compress)`	Save checkpoint
`load_checkpoint(uri)`	Load checkpoint
`load_checkpoint_with_validation(uri, expected_keys)`	Load with validation
`save_distributed_shard(uri, data, rank, world_size)`	Save distributed shard
`finalize_distributed_checkpoint(base_uri, world_size)`	Finalize distributed checkpoint

Logging

Function	Description
`init_logging(level)`	Set log level: trace, debug, info, warn, error
`init_op_log(path)`	Enable operation logging to file
`finalize_op_log()`	Flush and close operation log
`is_op_log_active()`	Check if operation logging is active

Migration Guide

From v0.9.40

v0.9.50 is a non-breaking upgrade. The only change is the internal runtime architecture:

v0.9.40	v0.9.50
`GLOBAL_RUNTIME.block_on()`	io_uring submit via `run_on_global_rt()`
Panics in multi-threaded Python	Fully thread-safe from any context
No `put_many()`	`put_many()` and `put_many_async()` added

From v0.8.x

Removed (v0.8.x)	Replacement (v0.9.x+)
`list_objects(bucket, prefix)`	`list(uri)`
`get_object(bucket, key)`	`get(uri)` or `get_range(uri, offset, length)`

From s3torchconnector

See s3torchconnector Migration Guide.

Support

GitHub: https://github.com/russfellows/s3dlio
Issues: https://github.com/russfellows/s3dlio/issues
License: Apache-2.0