AraTrust (Inspect AI)

An Inspect AI implementation of AraTrust, a human-written multiple-choice benchmark measuring the trustworthiness of large language models in Arabic.

• Paper: AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic (Alghamdi et al., COLING 2025)
• Dataset: asas-ai/AraTrust (MIT)

The benchmark has 522 questions across eight trustworthiness categories: truthfulness, ethics, physical health, mental health, unfairness, illegal activities, privacy, and offensive language. Each question has three options and one correct answer; the metric is accuracy.

Installation

uv sync            # or: pip install -e .

This installs inspect_ai and the eval package.

Usage

Run against every category:

inspect eval src/aratrust/aratrust.py --model openai/gpt-4o

Restrict to a single category (-T category=...):

inspect eval src/aratrust/aratrust.py -T category="Physical Health" --model openai/gpt-4o

Valid categories: Trustfulness, Ethics, Physical Health, Mental Health, Unfairness, Illegal, Privacy, Offensive (as spelled in the dataset's Category field).

Dataset handling

The dataset is pinned to a specific commit (d4dd124ed5b90aeb65a7dda7d88e34fb464a31ec) for reproducibility. Three source quirks are handled explicitly in record_to_sample:

1. Answer glyphs. The correct answer is stored as an Arabic letter. Both the hamza'd alef (أ) and, in two rows, the bare alef (ا) map to option A; ب maps to B and ج to C. Whitespace is trimmed.
2. Embedded option labels. Each option text is prefixed with its own Arabic label (e.g. أ) نعم). These printed labels are unreliable on a few rows, so they are stripped and the answer is resolved by column position.
3. Malformed rows. Rows with an empty option or an unmappable answer are dropped. At the pinned revision this removes exactly one row (an empty option A with shifted labels), leaving 521 usable samples. The count is asserted in the test suite so any upstream change surfaces loudly.

Evaluation report

Reference results reported in the paper (zero-shot, accuracy):

Model	AraTrust accuracy
GPT-4	81%
GPT-3.5 Turbo	79%
AceGPT 13B	65%
Jais 13B	65%
AceGPT 7B	54%

Reproduction with this implementation (521 samples, temperature 0):

Model	Accuracy	Std error
openai/gpt-4o	93.5%	1.1%
openai/gpt-4o-mini	90.0%	1.3%

Both scores are above the paper's GPT-4 (81%), which is expected for newer models, and well above the 33% chance level for three options, confirming the option to answer alignment is correct.

To reproduce against a model of your choice:

inspect eval src/aratrust/aratrust.py --model openai/gpt-4o --temperature 0

Development

uv sync --extra dev
ruff check . && ruff format --check .
mypy src
pytest                              # unit + dataset-download tests
RUN_DATASET_DOWNLOAD_TESTS=0 pytest # unit tests only (no network)

License and attribution

This implementation is released under the MIT License (see LICENSE). The AraTrust dataset is authored by Alghamdi et al. and distributed under the MIT License; see NOTICE for attribution.

Citation

@inproceedings{alghamdi-etal-2025-aratrust,
    title = "{A}ra{T}rust: An Evaluation of Trustworthiness for {LLM}s in {A}rabic",
    author = "Alghamdi, Emad A. and Masoud, Reem I. and Alnuhait, Deema and Alomairi, Afnan Y. and Ashraf, Ahmed and Zaytoon, Mohamed",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025)",
    year = "2025",
    url = "https://arxiv.org/abs/2403.09017",
}