SALAD-Bench (multiple-choice) for Inspect AI

An Inspect AI implementation of the multiple-choice half of SALAD-Bench, a hierarchical safety benchmark for LLMs.

• Paper: SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models (Li et al., ACL 2024 Findings) — arXiv:2402.05044
• Dataset: OpenSafetyLab/Salad-Data, config mcq_set — 3,840 questions, Apache-2.0, pinned by commit
• Taxonomy: 6 top-level domains → 16 task categories → 65 fine-grained categories

Method

Each sample presents a potentially harmful query with three candidate answers, and the model must select the safe option(s). One or two of the three options can be safe, so the gold label is a set ("A", "AC", …) and scoring is an exact set match — picking one safe option when two are safe is wrong, as is including an unsafe option.

Scoring is deterministic and needs no judge model. The answer is extracted from the <ans>...<eoa> envelope the prompt requests; this is SALAD-Bench's MC-Judge (a regex extract-and-match checker), as distinct from the model-based MD-Judge used for the free-form QA sets.

Reported metrics: accuracy and stderr overall, plus accuracy grouped by top-level safety domain. A small number of rows carry a null domain; they remain scored and are bucketed as Unknown rather than dropped.

Usage

pip install -e .
inspect eval src/inspect_salad_bench/salad_bench.py@salad_bench_mcq \
  --model openai/gpt-4o-mini --limit 100

No grader/judge model or API key beyond the model under test is required.