Automated Checks

July 13, 2026 · View on GitHub

Note for template users: this document is synced from inspect_evals where these checks are required for registry submission. In this template they are recommended, not required — autolint is advisory by default. See Checks and enforcement to opt in.

We have a number of automatic checks that can be run using tools/run_autolint.py:

uv run python tools/run_autolint.py <eval_name>

To suppress a check, use:

  • Line-level: # noautolint: <check_name>
  • File-level: # noautolint-file: <check_name> (at top of file)
  • Directory-level: Add check name to src/<eval_name>/<subdir>/.noautolint
  • Eval-level: Add check name to src/<eval_name>/.noautolint

You don't need to read these checks - they are presented here as a reference in case of linting errors.

File Structure (Automated)

  • The evaluation is located in a sub-directory of src/ (eval_location)
  • __init__.py exports task and related functions (init_exports)
  • @task functions are contained within src/<eval_name>/<eval_name>.py (main_file)
  • Task registered in pyproject.toml under [project.entry-points.inspect_ai] (registry)
  • eval.yaml exists in the evaluation directory with all required fields (eval_yaml)
  • README.md exists and has no TODO markers (readme)

Code Quality (Automated)

  • No imports from private inspect_ai modules (those starting with _) (private_api_imports)
  • Score() calls use CORRECT/INCORRECT constants instead of literal strings (score_constants)
  • External eval-specific dependencies declared in pyproject.toml (external_dependencies)

Tests (Automated)

  • Test directory exists at tests/<eval_name> (tests_exist)
  • Test directory and subdirectories have __init__.py (tests_init)
  • At least one E2E test uses mockllm/model (e2e_test)
  • record_to_sample is tested with a real sample (if used) (record_to_sample_test)
  • Custom @solver decorated functions have tests (custom_solver_tests)
  • Custom @scorer decorated functions have tests (custom_scorer_tests)
  • Custom @tool decorated functions have tests (custom_tool_tests)

Best Practices (Automated)

  • get_model() only called inside @solver/@scorer decorated functions (get_model_location)
  • Sample() calls include an id= parameter for stable IDs (sample_ids)
  • @task functions provide defaults for overridable parameters (solver, scorer, etc.) (task_overridable_defaults)
  • Dataset pinning is enforced at runtime: hf_dataset(), load_dataset(), snapshot_download(), and hf_hub_download() wrappers require a revision= keyword argument