Contributing to VeraBench

July 26, 2026 · View on GitHub

Thank you for your interest in contributing to VeraBench. This document provides guidelines and information for contributors.

How to Contribute

Reporting Issues

If you find a bug, incorrect test case, or have a suggestion:

Check the existing issues to see if it has already been reported.
If not, open a new issue.
Provide as much context as possible, including problem IDs and error output where relevant.

Adding New Problems

New benchmark problems are welcome. For each new problem, you must produce:

A .vera canonical solution in solutions/vera/ that passes vera check and vera verify.
A JSON problem definition in problems/tier{N}/ with all required fields (including description_neutral).
A Python baseline in solutions/python/.
A TypeScript baseline in solutions/typescript/.
An Aver baseline in solutions/aver/.
An AILANG baseline in solutions/ailang/.

Every comparison language needs a baseline, or vera-bench baselines --language {lang} covers fewer problems than its siblings and the cross-language numbers stop being like-for-like. (Baselines only execute the problems that carry test_cases — all 60 as of v0.0.18.)

Adding or changing test_cases on a problem carries three obligations, and vera-bench baselines across all four languages is the check that you met them:

The Aver and AILANG canonical mains print one line per test case, in test-case order — they are data that must track the problem JSON. Extend them with the new cases.
The GRADEABLE_ADDED table in scripts/plot_results.py records, for an existing problem gaining its first test cases, the bench version whose sweeps first grade it. Without an entry, regenerating an older version's charts silently deflates its published numbers.
All five languages must actually grade the new cases. If one cannot (the aver/ailang line protocol cannot host multi-line stdout, for instance), the problem stays ungraded everywhere rather than scored unevenly.

See CLAUDE.md for problem structure, tier definitions, and Vera gotchas.

Adding a New Comparison Language

VeraBench supports cross-language comparison. Currently: Vera, Python, TypeScript, Aver, and AILANG. To add a new language:

Canonical solutions — Create solutions/{lang}/ with one solution per problem (60 files). Each must produce correct output for all test cases.
Prompt builder — Add build_{lang}_prompt() to vera_bench/prompts.py. Use description_neutral (not description, which is Vera-specific). If the language has a reference doc (like SKILL.md or llms.txt), fetch it at runtime.
Code evaluator — Add _evaluate_{lang}_code() to vera_bench/runner.py. This writes generated code to a temp file, runs the compiler/interpreter, and compares output to expected values.
Baseline runner — Add run_{lang}_baseline() to vera_bench/baseline_runner.py. This runs canonical solutions against test cases.
CLI integration — Add the language name to the --language choice list in vera_bench/cli.py.
Problem JSON — If your language needs special descriptions, add them to the problem JSONs. For most languages, description_neutral is sufficient.
Tests — Add unit tests for the new evaluator, prompt builder, and any helper functions.
CodeRabbit exclusions — Add !**/*.{ext} and !solutions/{lang}/** to .coderabbit.yaml path_filters (CodeRabbit doesn't understand novel languages).

Reference implementations: PR #48 (Aver) and PR #70 (AILANG). The AILANG one is the better model if your language ships its teaching prompt inside its own CLI rather than at a URL — it shells out to ailang prompt --source embedded instead of fetching a doc, which keeps the prompt version-locked to the installed compiler.

Code Contributions

For contributions to the benchmark harness:

Fork the repository.

Create a feature branch from main:

git checkout -b feature/your-feature-name

Make your changes, following the coding standards below.
Add or update tests as appropriate.

Ensure all checks pass:

ruff check .
ruff format --check .
pytest -v
vera-bench validate

Commit your changes with a clear commit message.
Push to your fork and open a pull request.

Development Setup

Prerequisites

Python 3.11+
Git
Vera compiler on PATH

Installation

git clone https://github.com/aallan/vera-bench.git
cd vera-bench
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,llm]"

Pre-commit Hooks

pip install pre-commit
pre-commit install

This installs hooks that run on every commit: trailing whitespace, YAML/JSON validation, ruff lint and format, problem validation (when problem files change), and pytest.

Running Tests

pytest                          # run all tests
pytest -v                       # verbose output
pytest --cov=vera_bench         # with coverage
vera-bench validate             # validate all 60 problems

Linting

ruff check .                    # lint
ruff format --check .           # format check
ruff check --select S vera_bench/  # security lint

Coding Standards

Python Code

Python 3.11+, type hints everywhere.
ruff for linting and formatting (line length 88).
click for CLI, rich for terminal output.
JSONL for results files.
Subprocess calls to vera must have timeouts.

Problem Definitions

Follow the existing JSON schema (see any file in problems/).
De Bruijn index notes are critical — document slot ordering in the notes field.
Every canonical .vera solution must pass vera check and vera verify.
Test cases must have correct expected values verified against the canonical solution.

Commit Messages

Use the imperative mood ("Add feature" not "Added feature").
Keep the first line under 72 characters.
Reference related issues with #issue-number.

Pull Requests

Keep pull requests focused on a single change.
Update relevant documentation and tests.
Fill in the pull request template.
Ensure CI passes before requesting review.

Branch Protection

The main branch is protected:

Pull request required for all changes.
CI must pass (validate, test, lint, security, dependency-audit).
No direct pushes.