RetinaNet Training Benchmark Results
May 8, 2026 · View on GitHub
Date: 2026-04-26
Host: loki-russ
s3dlio version: 0.9.95
dlio_benchmark: editable install (/home/eval/Documents/Code/dlio_benchmark/)
Model: retinanet (b200 accelerator profile)
Dataset: 250,000 × ~323 KB JPEG files (fake/random data)
MPI ranks: 4
Batch size: 24
Epochs: 3
Steps/epoch/rank: 2,602 (= (250000 / 24 / 4) - warmup)
Compute time/step: 0.04755 s (simulated)
Background
A bug was fixed in s3dlio (prior session) where the direct:// URI scheme was not
actually using O_DIRECT — it silently fell back to buffered tokio::fs::read().
The fix routes Scheme::Direct through
ConfigurableFileSystemObjectStore::with_direct_io().
These 4 tests verify the fix and establish a performance baseline across all storage modes supported by mlp-storage.
A companion bug was also fixed in dlio_benchmark: _uri_for_obj_key() was
hardcoding s3:// instead of reading uri_scheme from storage options.
AU Formula
AU (Accelerator Utilization) = total_compute_time / epoch_wall_time
= (num_steps × compute_time_per_step) / epoch_wall_time
= (2602 × 0.04755 s) / epoch_wall_time
≈ 123.7 s / epoch_wall_time
Relationship between throughput and AU:
| Throughput (total s/s) | Per-rank s/s | Epoch time | AU |
|---|---|---|---|
| ~900 | ~225 | ~277 s | ~44% |
| ~1860 | ~465 | ~134 s | ~92% |
| ~1910 | ~478 | ~130 s | ~95% |
| ~1925 | ~481 | ~130 s | ~95% |
AU is a direct function of epoch wall time. Two runs with different throughputs cannot have the same AU unless they have the same epoch duration. Any result claiming otherwise is a documentation error.
Run Index
All result directories under /mnt/nvme_data/mlperf_storage_results/training/retinanet/run/.
| Run timestamp | Label | Status |
|---|---|---|
| 20260426_105648 | direct:// attempt (wrong storage_root) | Failed |
| 20260426_105745 | (early aborted run) | Failed |
| 20260426_110031 | (early aborted run) | Failed |
| 20260426_110211 | direct:// pre-fix wheel | Completed |
| 20260426_113500 | direct:// post-fix — T1 | Completed ✓ |
| 20260426_114955 | file:// s3dlio — T2 | Completed ✓ |
| 20260426_120232 | --file POSIX (wrong data path) | Failed |
| 20260426_120346 | --file POSIX, flush — T3 attempt | Completed ✓ |
| 20260426_121232 | --file POSIX, flush — T3 | Completed ✓ |
| 20260426_122554 | datagen attempt (double-prefixed params) | Failed |
| 20260426_122809 | datagen only (250,000 objects → s3-ultra) | Completed ✓ |
| 20260426_122934 | --object s3dlio → s3-ultra — T4 | Completed ✓ |
Full Result Data
Pre-fix baseline: direct:// without O_DIRECT (run 20260426_110211)
This run used the wheel before the O_DIRECT fix was installed. direct:// silently
fell back to buffered I/O, producing the same throughput as file://. This confirms
the original bug.
| Epoch | Throughput (s/s) | AU% | Wall time |
|---|---|---|---|
| 1 | 1909.3 | 94.95% | 151.7 s |
| 2 | 1916.1 | 95.28% | 130.6 s |
| 3 | 1910.0 | 94.98% | 131.0 s |
| Avg | 1911.8 | 95.07% |
E1 is longer than E2/E3 because the page cache was cold on first epoch, then warmed. This cache-warmup pattern is the signature of buffered I/O — it would not appear with true O_DIRECT.
T1 — direct:// via s3dlio, O_DIRECT active, no page cache flush (run 20260426_113500)
Storage mode: uri_scheme=direct, storage_root=/mnt/nvme_data
Page cache flush: None
s3dlio wheel: 0.9.95 (post-fix)
| Epoch | Throughput (s/s) | AU% | Wall time |
|---|---|---|---|
| 1 | 895.9 | 44.50% | 300.3 s |
| 2 | 895.4 | 44.47% | 279.9 s |
| 3 | 903.1 | 44.85% | 277.5 s |
| Avg | 898.1 | 44.61% |
train_au_meet_expectation: fail (< 85% target)
Interpretation: O_DIRECT is confirmed active. Throughput is capped at ~900 s/s (~225 MB/s per rank) because O_DIRECT bypasses the page cache and forces direct disk reads, exposing the raw NVMe bandwidth limit at this concurrency level. E1 is notably slower (300 s vs 280 s) due to inode/metadata lookup overhead on first access, not page cache (O_DIRECT skips page cache entirely).
T2 — file:// via s3dlio, buffered I/O, no page cache flush (run 20260426_114955)
Storage mode: uri_scheme=file, storage_root=/mnt/nvme_data
Page cache flush: None
s3dlio wheel: 0.9.95
| Epoch | Throughput (s/s) | AU% | Wall time |
|---|---|---|---|
| 1 | 1910.3 | 94.99% | 151.4 s |
| 2 | 1921.2 | 95.53% | 130.2 s |
| 3 | 1914.1 | 95.18% | 130.7 s |
| Avg | 1915.2 | 95.23% |
train_au_meet_expectation: success (> 85% target)
Interpretation: Buffered I/O with page cache. E1 is slower (151 s vs 130 s) because the page cache was cold — T1 used O_DIRECT and did not populate the page cache, so T2 starts cold. E2/E3 are fast because the cache is now warm.
NOTE — Session notes error: An earlier session summary incorrectly recorded T2 as having AU=44.5% with throughput E1:1652/E2:1919/E3:1913. That data was wrong. The 44.5% AU belongs exclusively to T1 (O_DIRECT). At 1915 s/s, the math gives AU = 123.7 s / 130 s ≈ 95%. It is mathematically impossible to have ~1900 s/s throughput and 44.5% AU simultaneously.
T3 — --file native POSIX, page cache flush before each epoch (run 20260426_121232)
Storage mode: native POSIX --file, data_folder=/mnt/nvme_data/retinanet
Page cache flush: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' before each epoch
s3dlio: not used
| Epoch | Throughput (s/s) | AU% | Wall time |
|---|---|---|---|
| 1 | 1880.6 | 93.52% | 156.7 s |
| 2 | 1860.1 | 92.53% | 134.4 s |
| 3 | 1856.5 | 92.36% | 134.6 s |
| Avg | 1865.7 | 92.80% |
train_au_meet_expectation: success (> 85% target)
Interpretation: Each epoch starts from a cold page cache (flush before every epoch). E1 is longer because of additional startup overhead (DLIO initialization) on top of the cold cache. E2/E3 are consistent at ~134 s. POSIX with cold cache is ~3% slower than s3dlio buffered with warm cache (130 s), which makes sense.
An earlier attempt (T3a, 20260426_120346) produced nearly identical results: E1:1873/E2:1859/E3:1859, avg AU=92.71%.
T4 — --object s3dlio → s3-ultra (loopback), page cache flush active (run 20260426_122934)
Storage mode: uri_scheme=s3, bucket mlp-retinanet, endpoint http://127.0.0.1:9101
Server: s3-ultra v0.1.6, --access-key testkey --secret-key testsecret
Page cache flush: active (benign for object storage — data never in local page cache)
s3dlio wheel: 0.9.95
| Epoch | Throughput (s/s) | AU% | Wall time |
|---|---|---|---|
| 1 | 1925.3 | 95.73% | 153.6 s |
| 2 | 1914.1 | 95.19% | 130.6 s |
| 3 | 1918.7 | 95.41% | 130.3 s |
| Avg | 1919.4 | 95.44% |
train_au_meet_expectation: success (> 85% target)
Interpretation: s3-ultra returns pseudo-random data over loopback HTTP/1.1. Object bytes are never stored or cached on disk. Despite this, throughput and AU match or exceed buffered NVMe file reads — the loopback network is not a bottleneck. E1 is slightly longer (153 s vs 130 s) due to connection setup and metadata initialization on first epoch.
Comparison Summary
| Test | Storage mode | Avg s/s | Avg AU% | Pass? |
|---|---|---|---|---|
| Pre-fix | direct:// (O_DIRECT NOT active) | 1911.8 | 95.07% | ✓ |
| T1 | direct:// O_DIRECT active, no flush | 898.1 | 44.61% | ✗ |
| T2 | file:// s3dlio, no flush | 1915.2 | 95.23% | ✓ |
| T3 | POSIX --file, flush/epoch | 1865.7 | 92.80% | ✓ |
| T4 | --object s3dlio → s3-ultra | 1919.4 | 95.44% | ✓ |
Target: AU ≥ 85% (b200 profile)
Key Findings
1. O_DIRECT fix confirmed
The pre-fix run (110211) shows direct:// at 95% AU — indistinguishable from
file://. The post-fix run (T1, 113500) shows direct:// at 44.6% AU and
~900 s/s, confirming O_DIRECT is now active and bypassing the page cache.
2. T2 session notes were incorrect
The session summary prior to this document incorrectly stated T2 had AU=44.5%. The actual value is 95.23%. The 44.5% was T1's value, apparently copied incorrectly. The AU calculation in dlio_benchmark is correct. No code change required.
3. Page cache flush effect
Without flush (T2): page cache warms after E1, E2/E3 at ~130 s/epoch.
With flush (T3): every epoch starts cold, all epochs at ~134-157 s/epoch.
The flush costs ~4 s/epoch (~3% throughput penalty) but ensures repeatable results.
4. s3-ultra loopback is not a bottleneck
T4 (s3-ultra over loopback) matches buffered NVMe at ~1919 s/s and 95.4% AU. The fake S3 server is suitable for functional testing and storage-library benchmarking without requiring real object storage infrastructure.
Configuration Reference
.env for T4 (object mode)
AWS_ACCESS_KEY_ID=testkey
AWS_SECRET_ACCESS_KEY=testsecret
AWS_ENDPOINT_URL=http://127.0.0.1:9101
AWS_REGION=us-east-1
STORAGE_LIBRARY=s3dlio
STORAGE_URI_SCHEME=s3
BUCKET=mlp-retinanet
Page cache flush in dlio_benchmark/main.py
import subprocess
# ...
if self.my_rank == 0:
try:
subprocess.run(
["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"],
check=True, timeout=30
)
except Exception:
pass
self.comm.barrier()
T1 / T2 run command
cd /home/eval/Documents/Code/mlp-storage
time uv run mlpstorage training run \
--model retinanet --num-accelerators 4 --accelerator-type b200 \
--client-host-memory-in-gb 47 --open --object \
--data-dir /mnt/nvme_data --allow-run-as-root --skip-validation \
--params dataset.num_files_train=250000 \
storage.storage_options.uri_scheme=direct # or: uri_scheme=file
Verify object count (fast)
# -c flag returns count only — much faster than full listing
AWS_ACCESS_KEY_ID=testkey AWS_SECRET_ACCESS_KEY=testsecret \
AWS_ENDPOINT_URL=http://127.0.0.1:9101 AWS_REGION=us-east-1 \
s3-cli list -c s3://mlp-retinanet/retinanet/train/
# Output: Total objects: 250000 (0.957s, rate: 261,259 objects/s)
T4 run command
cd /home/eval/Documents/Code/mlp-storage
time uv run mlpstorage training run \
--model retinanet --num-accelerators 4 --accelerator-type b200 \
--client-host-memory-in-gb 47 --open --object \
--data-dir retinanet --allow-run-as-root --skip-validation \
--params dataset.num_files_train=250000
# (storage params injected automatically from .env)
Bugs Fixed This Session Pair (Apr 25–26, 2026)
| Component | Bug | Fix |
|---|---|---|
s3dlio/src/python_api/python_core_api.rs | Scheme::Direct used buffered tokio::fs::read() instead of O_DIRECT | Split Scheme::File | Scheme::Direct arm; route Direct through ConfigurableFileSystemObjectStore::with_direct_io() |
dlio_benchmark/reader/_s3_iterable_mixin.py | _uri_for_obj_key() hardcoded s3:// prefix | Use self._opts.get("uri_scheme", "s3") |
dlio_benchmark/main.py | Page cache flush used open("/proc/sys/vm/drop_caches", "w") which fails without root | Replace with subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"]) |
s3-ultra/examples/start-s3-ultra.sh | Started without --access-key/--secret-key; health check used unauthenticated curl | Add auth key args; use aws s3api list-buckets (signed) for health check |