Memory

July 4, 2026 ยท View on GitHub

Memory management utilities for efficient data handling. These tools help manage shared memory for multi-process data loading.

Components

ComponentPurposeUse Case
SharedMemoryManagerCross-process sharingMulti-worker dataloading

!!! note "Key points"

- Shared memory avoids copying data between processes
- Arrays of at least 1 MiB are automatically placed in shared memory
- Used by `datarax.workers` for cross-process transforms
- Use for custom multi-process pipelines

Quick Start

import numpy as np
from datarax.memory import SharedMemoryManager

# The manager sizes segments automatically
manager = SharedMemoryManager()

# Publish an array to shared memory (arrays >= 1 MiB are shared)
shared = manager.make_shared("batch_0", np.zeros((1024, 1024), dtype=np.float32))

# Retrieve it from another reference
batch = manager.get_shared("batch_0")

# Release all segments
manager.cleanup()

Modules

Multi-Worker Pattern

Use the context manager so segments are always released:

import numpy as np
from datarax.memory import SharedMemoryManager

with SharedMemoryManager() as manager:
    for i in range(100):
        batch = load_batch(i)
        manager.make_shared(f"batch_{i}", np.asarray(batch))

    # Retrieve by name
    batch = manager.get_shared("batch_0")
    process(batch)
# Segments are cleaned up on exit

See Also