MLPerf® Storage is a benchmark suite to characterize the performance of storage systems that support machine learning workloads.

During LLM inference, models store intermediate attention data in a structure called the KV (K-Vector / V-Vector) cache to reduce duplicate prefill computation. This cache grows with conversation length and can consume enormous amounts of memory. Production systems offload this cache from expensive GPU VRAM to cheaper CPU RAM or NVMe storage.

This benchmark simulates this offloading behavior. It generates realistic multi-user inference workloads and measures how your storage performs under pressure. Detailed design documentation can be found in the DESIGN.md file.

Usage

The KV cache benchmark runs three distinct workloads using two model sizes:

WORKLOAD_PARAMS = {
    1: {
        'model': 'llama3.1-8b',
        'num-users': 200,
        'duration': 300,
        'gpu-mem-gb': 0,
        'cpu-mem-gb': 0,
        'max-concurrent-allocs': 16,
        'generation-mode': 'none',
    },
    2: {
        'model': 'llama3.1-8b',
        'num-users': 100,
        'duration': 300,
        'gpu-mem-gb': 0,
        'cpu-mem-gb': 4,
        'max-concurrent-allocs': 16,
        'generation-mode': 'none',
    },
    3: {
        'model': 'llama3.1-70b-instruct',
        'num-users': 70,
        'duration': 300,
        'gpu-mem-gb': 0,
        'cpu-mem-gb': 0,
        'max-concurrent-allocs': 4,
        'generation-mode': 'none',
    },
}

For closed submissions, the paramaters that define these workloads are fixed.

The benchmark execution process requires two steps:

1. Datasize - Calculate required client memory and storage space required per workload instance
2. Run - Execute the benchmark (runs all three workloads)

[root@localhost ]# ./mlpstorage closed kvcache run --help
usage: mlpstorage closed kvcache run --results-dir RESULTS_DIR [--allow-run-as-root] [--cache-dir CACHE_DIR]
                                     [--config-file CONFIG_FILE] [--debug] [--dry-run] [--exec-type {mpi,docker}] [-h]
                                     [--hosts HOSTS [HOSTS ...]] [--kvcache-bin-path KVCACHE_BIN_PATH]
                                     [--max-timeseries-samples MAX_TIMESERIES_SAMPLES] [--mpi-bin {mpirun,mpiexec}]
                                     [--mpi-btl {auto,vader,tcp}] [--mpi-params MPI_PARAMS] [--num-processes NUM_PROCESSES]
                                     [--oversubscribe] [--quiet] [--skip-timeseries] [--skip-validation]
                                     [--stream-log-level STREAM_LOG_LEVEL] [--timeseries-interval TIMESERIES_INTERVAL] [--verbose]
                                     [--verify-lockfile PATH]

MLPerf Submission Guidelines

Official MLPerf KV cache storage submissions do not require the specification of test paramaters, as standardized test parameters will be used when the KV cache benchmark is run using the mlpstorage command with the closed flag. These parameters have been validated through extensive discovery testing (1,411 Fast system tests, 268 Slow system tests comparing 14,000 MB/s vs 3,000 MB/s storage).

All storage devices must be preconditioned before running the benchmark. This stop is critical to ensure reproducability of results and consistency of results over time. Please set aside sufficient time for preconditioning, as this process may take multiple days for large storage systems.

Running the KV Cache Benchmark

The minimal command required to run the KV cache benchmark suite is:

[root@localhost ]# ./mlpstorage closed kvcache run --cache-dir /path/to/storage/ --results-dir /path/to/output/

This runs a single instance of the KV cache benchmark suite on the local host.

In order to scale the I/O load generated by the KV cache benchmark, multiple instances of the benchmark are launched using MPI. For example, to run the benchmark across two hosts, the minimal command required is:

[root@localhost ]# ./mlpstorage closed kvcache run --hosts 10.117.61.121,10.117.61.165 --cache-dir /path/to/storage/ --results-dir /path/to/output/

Additional parameters may be required to allow this benchmark to run in your MPI environment:

MPI:
  --allow-run-as-root
  --mpi-bin {mpirun,mpiexec}
                        Execution type for MPI commands. Supported options: ['mpirun', 'mpiexec']
  --mpi-btl {auto,vader,tcp}
                        MPI Byte Transport Layer for single-host runs. 'auto' lets OpenMPI select automatically (default; works on
                        most systems). 'vader' forces POSIX shared-memory transport (fast; may fail in containers or as root).
                        'tcp' forces TCP loopback transport (universally compatible; recommended for containers and root
                        environments). Has no effect on multi-host runs.
  --mpi-params MPI_PARAMS
                        Additional parameters passed verbatim to the MPI launcher. Pass them as a single quoted string, e.g. --mpi-
                        params="-genv PMI_VERSION=2 -genv FI_PROVIDER=tcp". Because MPI flags begin with '-', use the '--mpi-
                        params=...' (equals) form so argparse does not mistake them for options. May be supplied multiple times;
                        values are concatenated.

MLPerf KV Cache Benchmark Results

KV cache benchmark results are calculated for each workload:

• "Throughput (Tokens/sec)" - Sum of "storage_throughput_tokens_per_sec" from all instances during a run
• "Read Bandwidth (GiB/s)" - Sum of "tier_storage_read_bandwidth_gbps" from all instances during a run
• "Write Bandwidth (GiB/s)" - Sum of "tier_storage_write_bandwidth_gbps" from all instances during a run
• "P95 Read Latency (ms)" - Highest of "storage_read_p95_ms" from all instance during a run

Each workload is run three times. The final submission results that will be displayed in the results table is the mean of the three runs for throughput and bandwidth, and the max of the three runs for latency. These results are reported in the generated summary.json file at the root of the run's result directory.