AReaL-SWEAgent

July 1, 2026 · View on GitHub

An RL agent adapter that runs SWE-bench coding tasks inside sandboxed AEnvironment containers and returns a reward — built to plug into the AReaL RL training framework.

For each task, AReaL hands a SWE-bench instance to one of two agents — a built-in tool-use agent (swe) or a delegated Claude Code CLI (cc) — which edits code inside an isolated sandbox. The adapter then extracts the resulting patch, grades it, and returns (reward, stats) for the RL loop.

flowchart TD
    AReaL["AReaL RL trainer<br/>serves the model being trained"]
    RUN["run_agent_return_reward()<br/>aweagent.runner entry"]
    LIFE["lifecycle<br/>env → agent → reward → (reward, stats)"]

    subgraph AG["Agent Layer"]
        SWE["SWEAgent<br/>LLM loop + bash tools"]
        CC["CCAgent<br/>Claude Code delegation"]
    end

    subgraph EN["Environment Layer"]
        AENVSWE["AenvSWE"]
        AENVCC["AenvCC"]
    end

    SANDBOX["AEnvironment sandbox (k8s)<br/>persistent repo workspace + shell<br/>+ in-sandbox grading (eval_script)"]

    AReaL -->|"data + override_base_url / api_key"| RUN --> LIFE --> AG
    AG -->|"run commands / edit files"| EN -->|"AENV_SYSTEM_URL"| SANDBOX
    SWE -. "ask: next action?" .-> AReaL
    CC -. "(via gateway)" .-> AReaL
    SANDBOX -->|"patch graded → reward"| LIFE
    LIFE -->|"(reward, stats)"| AReaL

    classDef a fill:#e3f2fd,stroke:#1565c0,color:#0d47a1;
    classDef b fill:#f3e5f5,stroke:#7b1fa2,color:#4a148c;
    classDef c fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20;
    classDef d fill:#fff3e0,stroke:#ef6c00,color:#e65100;
    class AReaL a;
    class RUN,LIFE b;
    class SWE,CC,AENVSWE,AENVCC c;
    class SANDBOX d;

How it integrates with AReaL

The single entry point is aweagent.runner.run_agent_return_reward(data). AReaL calls it once per rollout:

from aweagent.runner import run_agent_return_reward

reward, stats = await run_agent_return_reward(instance)

It reads its configuration from environment variables and from the per-task data record:

Source	Key	Purpose
env	`SWEAgent_AGENT_TYPE`	`swe` or `cc` (default `swe`)
env	`SWEAgent_CONFIG`	config name from `aweagent/configs/` (default `train`)
env	`LOG_DIR`	per-task log/trace output dir
env	`AENV_SYSTEM_URL`	AEnvironment api-service (the sandbox cluster)
data	`override_base_url` / `override_api_key`	LLM endpoint AReaL injects per rollout
data	`instance_id`, `workdir`, `f2p_script`, …	SWE-bench task fields

Two independent connections per rollout:

LLM endpoint (override_base_url/override_api_key) → the model AReaL is training. The agent asks it "what command next?".
Sandbox (AENV_SYSTEM_URL) → the AEnvironment container where the agent actually runs commands, edits files, and is graded.

Quick start in AReaL

This project lives under the AReaL project; AReaL's SWE RL training entry (examples/swe/train_swe_rl.py) drives it. You point AReaL at this checkout and at your AEnvironment cluster through the training config.

1. Make the adapter importable. Put this checkout on PYTHONPATH and point the agent-root env vars at it:

# in the actor scheduling spec's env_vars (AReaL training config)
PYTHONPATH: /path/to/AReaL-SWEAgent:/path/to/AReaL
AWEAGENT_ROOT: /path/to/AReaL-SWEAgent
SWE_AGENT_ROOT: /path/to/AReaL-SWEAgent
AENV_SYSTEM_URL: http://<your-aenv-host>:<port>     # the sandbox cluster
# cc agent only — proxy that fronts the LLM gateway (omit for the swe agent):
REMOTE_PROXY_SERVICE_URL: http://<your-proxy-host>:<port>
REMOTE_PROXY_API_KEY: <your-proxy-key>

2. Install the SDK in the worker (the training image usually ships everything except the AEnvironment SDK). Add to the actor's additional_bash_cmds:

additional_bash_cmds:
- export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt PIP_CERT=/etc/ssl/certs/ca-certificates.crt
- <python> -m pip install --no-deps aenvironment==<version> || echo 'WARN: aenvironment install failed'
- <python> -m pip install 'fastmcp<3' || echo 'WARN: fastmcp install failed'
- <python> -c 'import aenv, aweagent' || echo 'WARN: aenv/aweagent import failed'

3. Select the agent and config under econfig:

econfig:
  agent_type: swe                              # swe | cc
  swe_agent_config: 1_0_0/min-swe-agent-train-top1
  cc_agent_config: <cc config>                 # only if agent_type: cc
  agent_root: /path/to/AReaL-SWEAgent
  swe_agent_root: /path/to/AReaL-SWEAgent
  step_limit: 100
  timeout: 3600.0

AReaL's workflow then calls run_agent_return_reward(data) per rollout, injecting override_base_url/override_api_key (the model being trained) into each data record. On exit it runs aweagent.maintenance.clean_instances (via the config's post_exit_hook) to release any leaked sandboxes.

Reference: the swe agent path is the lightest (no proxy needed) and is exercised by AReaL's SWE RL training entry. The cc agent additionally needs the proxy env vars above and the cc_golang_bash gateway image on the sandbox cluster.

Agents

Agent	`SWEAgent_AGENT_TYPE`	How it solves a task
SWE	`swe`	Built-in agent: an LLM conversation loop with `bash` / `search_replace` tools. Drives the LLM directly.
Claude Code	`cc`	Delegates the whole task to a Claude Code CLI running inside the sandbox, via the `cc_golang_bash` gateway.

Sandbox backend

Tasks run in AEnvironment containers (a k8s-backed sandbox platform). The environment wrapper:

creates a persistent-bash-env instance per task at AENV_SYSTEM_URL,
hides the repo's .git so the agent sees a clean checkout, then restores it to compute the diff,
supports both the SWE-bench /testbed layout and the scaleswe /workspace/<repo> layout via repo_dir,
grades the patch (resolve / no-resolve) to produce the reward.

The run-tool payload key depends on the sandbox image: persistent-bash-env@1.1.5 requires cmds, other versions use command. This is handled automatically in envs/swe.py.

Installation

Requirements: Python ≥ 3.10; the aenvironment SDK; access to an AEnvironment cluster.

pip install -e .            # runtime
pip install -e ".[dev]"     # + pytest

The aenvironment SDK is the client for the sandbox runtime. Install it from your AEnvironment deployment if it is not on your package index. The unit tests mock it, so pytest runs without a cluster.

Project structure

aweagent/
├── runner.py        # run_agent_return_reward — the AReaL entry point
├── lifecycle.py     # env create → agent run → classify → reward
├── pipeline.py      # reward (patch grading) + agent registry
├── agents/          # base, swe, cc
├── envs/            # AenvSWE, AenvCC (AEnvironment SDK wrappers)
├── models/          # OpenAI-compatible async LLM client (swe agent)
├── tools/           # search_replace, file_edit (swe agent)
├── common/          # io, logging
├── maintenance/     # clean_instances (release leaked sandboxes)
└── configs/         # train, eval_cc, 1_0_0/{cc,min-swe-agent,*}
cc_golang_bash/      # Go gateway for the Claude Code sandbox
tests/               # Python test suite (aenv SDK mocked)

Development & testing

pip install -e ".[dev]"
pytest                                   # Python suite (aenv mocked)
cd cc_golang_bash && go build ./... && go test ./...

License

Apache-2.0.