Concordia Evals

This repository contains Inspect-native evaluations maintained by Concordia AI. Evaluations here are designed to be run with the Inspect AI framework.

Relationship to UKGovernmentBEIS/inspect_evals

This repository is not part of the official inspect_evals monorepo. Following the Inspect Evals Register model, each evaluation is maintained independently here and registered with the official project via a small metadata file under register/<eval_name>/eval.yaml in UKGovernmentBEIS/inspect_evals. This keeps dependencies isolated while still making the evaluations discoverable through the official Inspect Evals documentation.

Evaluations

Evaluation	Description
BixBench	Evaluates language model agents on bioinformatics data analysis tasks. Agents analyze biological datasets using Jupyter notebooks and answer open-ended research questions.

Installation

git clone https://github.com/concordia-ai/concordia_evals.git
cd concordia_evals
uv sync

Usage

Each evaluation can be run directly from its task file. For example:

uv run inspect eval src/concordia_evals/bixbench/bixbench.py@bixbench --model openai/gpt-4o

See the individual evaluation directories under src/concordia_evals/ for detailed usage, parameters, and environment setup.