Benchmarking

Full Benchmark Harness

The examples below replay the arxiv length trace. Generate it first (see scripts/traces/arxiv):

uv run python scripts/traces/arxiv/prepare_trace.py

This writes data/traces/arxiv_summarization_llada2_tokenizer_filtered_4k.csv.

Colocated Benchmark Example

uv run python -m sangam.benchmark.main \
    --model GSAI-ML/LLaDA-8B-Instruct \
    --mode colocated \
    --gpus 0,1 \
    --num-requests 200 \
    --interval-type poisson --qps 2.0 \
    --length-type trace \
    --length-trace-file data/traces/arxiv_summarization_llada2_tokenizer_filtered_4k.csv \
    --output-dir benchmark_output

Hybrid Benchmark Exmaple (requires atleast 2 GPUs)

uv run python -m sangam.benchmark.main \
    --model GSAI-ML/LLaDA-8B-Instruct \
    --mode hybrid \
    --prefill-gpus 0 --hybrid-colocated-gpus 1 \
    --num-requests 200 \
    --interval-type poisson --qps 2.0 \
    --length-type trace \
    --length-trace-file data/traces/arxiv_summarization_llada2_tokenizer_filtered_4k.csv \
    --output-dir benchmark_output

Auto-launched benchmark servers share the same launch surface as sangam.entrypoints.launch, including scheduler policies, distributed settings, and prefill/decode batching limits.

Trace Inputs

Full Request Trace

Use --request-generator-type trace --trace-file ... to replay requests with arrival times and token counts. The CSV is expected to contain:

• Date: used with --trace-date to select a single day
• Time: arrival time in seconds within that date; scaled by --time-scale-factor
• PromptTokenCount: prompt length before scaling/clipping
• CompletionTokenCount: completion length before scaling/clipping

PromptTokenCount and CompletionTokenCount are scaled by the configured prefill/decode scale factors, clipped to at least 1 token each, and reduced as needed to satisfy the configured max-token cap.

Interval Trace

Use --interval-type trace --interval-trace-file ... to replay only arrival spacing. The CSV is expected to contain:

• arrival_time: parseable timestamp column

Rows are filtered by --interval-trace-start-time and --interval-trace-end-time, normalized to seconds from the first retained request, then rescaled by --interval-trace-time-scale-factor.

Length Trace

Use --length-type trace --length-trace-file ... to replay only prompt/completion lengths. This is how the data/traces/arxiv_summarization_llada2_tokenizer_filtered_4k.csv trace used in the examples above (generated by scripts/traces/arxiv/prepare_trace.py) is consumed. The CSV is expected to contain:

• prompt_len: required
• gen_len: required
• optional messages: a JSON list of chat messages, e.g. [{"role": "user", "content": "..."}]
• optional session_id and request_id_in_session: combined into a per-request external_id of the form "{session_id}-{request_id_in_session}"

prompt_len and gen_len are scaled, clipped to at least 1 token each, and capped to the configured max-token limit. When messages is present, the runner tokenizes and submits the real chat payload instead of placeholder token ids; this requires --model to point at an instruct model with a chat template. Any additional columns (such as total_len) are ignored.

The number of rows replayed is bounded by --num-requests / --duration, so the full CSV is only replayed when --num-requests is at least the row count.

Example length-trace skeleton:

session_id,request_id_in_session,messages,prompt_len,gen_len
2,0,"[{""role"": ""user"", ""content"": ""Summarize the following arxiv paper...""}]",1024,256

Prompt-Bearing Trace Behavior

When a length trace includes messages, the harness tokenizes and submits that exact chat payload. Prompt-bearing trace requests also default to unmasking_strategy=conf_threshold with confidence_threshold=0.9 when sampling flags are otherwise left at implicit defaults.

Benchmark requests now carry an explicit stable request_seed that is preserved through request normalization and prompt-text alignment. That keeps nonzero-temperature sampling reproducible across scheduler modes and repeated benchmark runs.

Benchmark Outputs

Each run writes a timestamped directory under benchmark_output/ containing the most important artifacts below:

• config.yaml: resolved benchmark configuration
• benchmark_results.json: aggregate latency/throughput summary plus per-request records, including prompt_text and generated_text when present
• request_metrics.csv: per-request metrics such as latency, total scheduling delay, request_scheduling_delay_prefill, request_scheduling_delay_decode, and forward-pass counts
• worker_timeline.csv: worker queue-state visibility
• worker_batch_metrics.csv: only when --enable-individual-batch-metrics is enabled; includes batch_sampling_time and batch_phase, plus batch_op_attn_time and batch_op_mlp_time when --enable-operation-metrics is enabled
• worker_plots/worker_batch_execution_time.{csv,png}: aggregate worker batch execution-time CDF across all batch phases
• worker_plots/worker_decode_length_sum_time_series.{csv,png}: decode-batch timeseries of summed per-request sequence lengths per worker
• worker_plots/worker_batch_phase_time_totals.{csv,png}: total execution time each worker spent in prefill, decode, and mixed batch phases

Timing values in request_metrics.csv, block_metrics.csv, and worker_batch_metrics.csv are exported with 4 decimal places.

Interpretation notes:

• gen_tokens means generated/completion tokens only; prompt tokens are excluded
• summary tokens_per_sec is total successful completion tokens divided by benchmark wall-clock time
• failed requests are excluded from throughput summaries
• the harness runs one warmup request, then resets scheduler-side metrics so exported artifacts reflect only the measured run

Benchmark Timeout

Every benchmark run is bounded by --benchmark-timeout (default 900 seconds / 15 minutes), covering server startup, warmup, the main request loop, and teardown. On expiry the harness logs a warning, tears down the server, and exits with code 2. Pair with --export-partial-metrics to flush any metrics collected before the timeout fires. Pass --benchmark-timeout 0 to disable the cap.