CASTLE Benchmark

An Inspect AI implementation of CASTLE, a benchmark for evaluating language models' ability to identify vulnerabilities in C code and classify them with Common Weakness Enumeration (CWE) identifiers.

The evaluation uses the CASTLE-C250 dataset of 250 C code samples and reports the CASTLE score, true-positive rate, and false-positive rate.

Installation

Clone the repository and install the project with uv:

git clone https://github.com/AI-Sec-dev/inspect-eval-castle.git
cd inspect-eval-castle
uv sync

Usage

uv run inspect eval castle/castle --model <provider/model>

# Limit the evaluation to 10 samples
uv run inspect eval castle/castle --model <provider/model> --limit 10

# Evaluate a specific sample
uv run inspect eval castle/castle --model <provider/model> --sample-id <id>

For example, with a mock model:

uv run inspect eval castle/castle --model mockllm/model

Development

uv sync
uv run pytest
uv run ruff check
uv run ruff format --check

Citation

If you use this evaluation, cite the original CASTLE paper:

@article{castle2025,
  title = {{CASTLE}: Benchmarking Dataset for Static Code Analyzers and
           {LLMs} towards {CWE} Detection},
  author = {Dubniczky, Richard A. and Horv\'at, Krisztofer Zolt\'an and
            Bisztray, Tam\'as and Ferrag, Mohamed Amine and
            Cordeiro, Lucas C. and Tihanyi, Norbert},
  year = {2025},
  eprint = {2503.09433},
  archiveprefix = {arXiv},
  primaryclass = {cs.CR}
}

License

This implementation is licensed under the MIT License. The bundled CASTLE dataset may be subject to separate upstream terms.