AutoNumerics: Code and Results

This repository contains the AutoNumerics implementation, frozen experimental results used for reporting, independent evaluator sidecars, and standalone CodePDE numerical checks. It intentionally does not include the paper source.

For manuscript-to-result verification, start with RESULTS_GUIDE.md. It identifies the authoritative JSON files, metric definitions, runtime scopes, and reporting constraints without duplicating the result tables.

Repository Layout

• implementation/main_pipeline/: formulation, planning, selection, code generation, repair, execution, evaluation, and reporting.
• implementation/ablations/: stability-awareness and paired coarse-to-fine ablation entrypoints.
• implementation/blind_evaluator/: isolated oracle-free Blind Evaluator and downstream comparison tools.
• implementation/residual_evaluation/: read-only tools used to reproduce the archived experiment tables under the final metric policy.
• benchmarks/: frozen Benchmark-200 v1.0.0 catalog, including the exact Original-24 and Hard-25 evaluation specifications.
• results/paper_result_0711/: curated experiment artifacts. Its RESULTS_MANIFEST.json identifies the authoritative experiment directories and the single final table bundle.
• validation/codepde_standalone_checks/: standalone numerical checks for the five CodePDE/PDEBench-derived cases.
• scripts/: portable command wrappers.

No absolute local path is included in the release. Historical relative provenance fields inside frozen result metadata are preserved as recorded; they are not runtime dependencies. Plans, generated solver source, candidate outputs, and recorded execution outcomes were not regenerated. The authoritative evaluator-side diagnostics and table aggregates are identified in RESULTS_GUIDE.md.

Quick Start

All commands below must be run from the repository root: the directory that contains this README.md, requirements.txt, implementation/, and scripts/.

Option 1: Git clone

Clone the repository and enter it:

git clone <repository-url> Autonumerics
cd Autonumerics

Option 2: GitHub ZIP

Use Code > Download ZIP on GitHub, extract the archive, and open a terminal in the extracted directory. GitHub commonly names that directory Autonumerics-main or Autonumerics-<branch>. Confirm that it is the correct directory before continuing:

test -f README.md
test -f requirements.txt
test -x scripts/run_tests.sh

Install and verify

Create an isolated Python environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Run the offline test suite and portability checks:

./scripts/run_tests.sh

The offline tests do not require an API key. For LLM-backed pipeline, ablation, or agent-based Blind Evaluator runs, configure the API through environment variables:

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://us.api.openai.com/v1"

Confirm that the benchmark catalogs and command wrappers are available:

./scripts/run_main.sh --which original24 --list-problems
./scripts/run_main.sh --which hard --list-problems
./scripts/run_main.sh --which benchmark200 --list-problems

Run one small pipeline job:

./scripts/run_main.sh \
  --which original24 \
  --problem-id advection_2d_periodic_sin \
  --output-root outputs/main

All newly generated files are written under the requested output root. The frozen artifacts under results/ are treated as read-only.

Installation

Python 3.10 or newer is recommended.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

LLM-backed runs require an API key:

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://us.api.openai.com/v1"

The API key is read only from the environment and is not stored in this repository.

Main Pipeline

List the Original-24 problems:

./scripts/run_main.sh --which original24 --list-problems

Run one problem:

./scripts/run_main.sh \
  --which original24 \
  --problem-id advection_2d_periodic_sin \
  --output-root outputs/main

Run a catalog or explicit subset:

./scripts/run_batch.sh \
  --which original24 \
  --max-problems 1 \
  --output-root outputs/main

Ablation Studies

Run the stability-awareness ablation:

./scripts/run_stability_ablation.sh \
  --which original24 \
  --conditions baseline no_pipeline_stability \
  --evaluation-mode full \
  --num-plans 10 \
  --top-k 5 \
  --output-root outputs/stability

Run paired coarse-to-fine Design B:

./scripts/run_coarse_to_fine_ablation.sh \
  --which original24 \
  --design B \
  --num-plans 10 \
  --top-k 5 \
  --output-root outputs/coarse_to_fine

The full Original-24 and Hard-25 sequential scripts are also available under implementation/ablations/.

Blind Evaluation

The deterministic Blind Evaluator smoke test does not require an API key:

./scripts/run_blind_evaluator.sh \
  --existing-results-root results/paper_result_0711/original24_main_pipeline \
  --mode execution_metadata_baseline \
  --output-root blind_evaluator_outputs

The included root-level hash manifest is used automatically. The agent evaluator additionally requires OPENAI_API_KEY and --model.

The current main pipeline computes evaluator-owned term-balanced residuals directly. New pipeline runs do not require any residual post-processing. The tools under implementation/residual_evaluation/ exist only to reproduce the metric aggregation and tables for the included archived experiments.

Standalone CodePDE Checks

The scripts 1.py through 5.py use data files stored in the same directory. Run them from that directory:

cd validation/codepde_standalone_checks
python 1.py

These are early standalone numerical checks, not AutoNumerics-generated candidate solvers. The reported AutoNumerics CodePDE results are under results/paper_result_0711/codepde_data_derived_5_main_pipeline/.

Verification

Run local tests and portability checks:

./scripts/run_tests.sh

The test command does not call an external model API.