airunner-eval

Benchmark and evaluation harness for the LLM agents in Capsize-Games/airunner (AI Runner). This is not shipped to customers; it's benchmark tooling used to evaluate agent quality (tool selection, mood, document handling, response quality) against real running daemons.

Provenance

Extracted from Capsize-Games/airunner at commit b9cfbef52 (2026-09-17), as issue #2194, part of the repository-split tracker #2185. History for the moved files is preserved (git filter-repo); it was the first extraction because nothing in airunner/airunner-services depends on it -- the dependency arrow only ever pointed one way, eval -> services.

Install

pip install -e ".[dev]"

Depends on the published airunner-common and airunner-services PyPI packages (~=6.1, tracking docs/architecture/versioning-and-compatibility-policy.md in the airunner repository, issue #2191) rather than a path dependency, since this repository no longer lives in the same tree.

Known limitation: duplicated test infrastructure

tests/llm_functional_support.py (daemon start/stop, environment setup, log-waiting helpers for functional tests that need a real running daemon) is a copy, not a shared dependency. The original still lives in airunner's services/tests/llm_functional_support.py, where it also backs several non-eval functional test files (test_llm_functional.py, test_llm_tts_functional.py, test_stt_transcribe_functional.py, test_gui_llm_tts_functional.py, test_gui_stt_llm_tts_functional.py).

This wasn't discovered until the extraction was already in progress, and resolving it properly -- publishing a shared daemon-testing support package both repositories import, or some other shared home -- is a real, separate piece of design work, not something to force through as part of a history-preserving move. Filed as a known gap rather than silently duplicating the file and moving on; see the extraction issue's tracking comment for the follow-up.

Until that's resolved, a behavioral fix or feature addition to the shared daemon-testing helpers needs to land in both repositories by hand.

Running the tests

pytest tests -v -m eval --timeout=300

LLM quality evals need a GPU and locally served models (see docs/agent-eval-tests.md in the airunner repository) and cannot pass on a stock CI runner; CI keeps the job for its uploaded report but does not gate on it, matching the continue-on-error: true behaviour the original eval-tests CI job had in the source repository.

Licence

GPL-3.0-only. See LICENSE and NOTICE.