Harness-RL trains language-model agents inside a central-agent multi-agent harness without flattening the execution into a single token sequence. It constructs black-box trajectories from interface calls and session prefix trees, then applies Conflict-Aware Policy Optimization (CAPO) to route gradients from action tokens and argument tokens to their corresponding parameter subspaces.

The implementation supports central-only training and joint multi-agent training. StackPlanner provides the production agent harness, while slime provides distributed rollout, SGLang serving, Megatron training, checkpointing, and weight synchronization.

Repository layout

.
├── StackPlanner2.0/                  # Multi-agent harness and runtime
├── slime/                            # Distributed RL training framework
├── slime/slime_plugins/stackplanner/ # Harness-RL integration and CAPO
├── configs/                          # Portable example configurations
├── scripts/                          # Data, probing, service, and evaluation tools
└── train_stackplanner.py             # Training entry point

Installation

The supported setup is Linux x86_64 with NVIDIA GPUs, a compatible CUDA driver, Git, Make, and Python 3.12.

git clone --branch Harness-RL https://github.com/jiangxinke/Harness-RL.git
cd Harness-RL

make doctor
make setup
source scripts/training_env.sh

The setup is stored under $HOME/.cache/harness-rl by default. To use another location, set SP_ENV_PREFIX, SP_SOURCE_ROOT, or SP_BUILD_CACHE when running make setup.

Prepare data

Download and convert the pinned ASearcher training split:

make prepare-data

Prepare the evaluation manifest:

make prepare-eval

Generated datasets are written to data/ and are not tracked by Git.

Prepare models

Place the Hugging Face checkpoint and its Megatron torch-distributed conversion at locations of your choice. slime includes the conversion utility:

python slime/tools/convert_hf_to_torch_dist.py --help

Update hf_checkpoint, reference_checkpoint, tokenizer, the distributed topology, and the model architecture arguments in the selected file under configs/.

Configure retrieval

Harness-RL uses the Search-R1 retriever through the standard StackPlanner web_search interface.

make search-setup
python scripts/prepare_search_r1_assets.py \
  --asset-dir "$HOME/.cache/harness-rl/search-r1"
make search-start
make configure-runtime

Set SP_SEARCH_R1_ROOT, SP_SEARCH_R1_CUDA_DEVICES, or SP_SERVICE_HOST to match your deployment. The committed configuration contains no credentials and uses loopback service addresses by default.

Central-only runs also require a frozen OpenAI-compatible specialist endpoint:

SP_FROZEN_MODEL_PATH=/path/to/model make frozen-start

Train

Three portable templates are provided:

Configuration	Training policy	CAPO scope
configs/experiment_a_central.yaml	Central agent only	Disabled
configs/experiment_b_all_agents.yaml	Central and specialist policies	Action/argument routes across actor roles
configs/experiment_c_capability_decoupled.yaml	Central agent only	Action/argument routes for the central policy

The templates use relative repository paths and start in dry-run mode. Edit the model paths and hardware topology, inspect the resolved command, then set dry_run to false.

python train_stackplanner.py \
  --config configs/experiment_a_central.yaml \
  --print-command

python train_stackplanner.py \
  --config configs/experiment_a_central.yaml

Resume from a slime checkpoint with:

python train_stackplanner.py \
  --config configs/experiment_a_central.yaml \
  --resume-from /path/to/torch-dist-checkpoint

Build CAPO routing artifacts

Experiments B and C require routing metadata generated from successful rollout records. These files are runtime inputs and remain outside version control.

torchrun --nproc-per-node 1 scripts/probe_stackplanner_capabilities.py \
  --model /path/to/hugging-face-checkpoint \
  --model-digest <sha256> \
  --rollout-glob "outputs/<run-id>/trajectories/*.jsonl" \
  --output-dir outputs/capability/central \
  --taxonomy central-capability

Use --taxonomy all-agent-action-output for joint multi-agent training, then point the capability fields in the corresponding configuration to the generated directory.

Evaluate

Generate predictions with the production StackPlanner graph:

python scripts/generate_evaluation_predictions.py \
  --manifest data/evaluation_630.jsonl \
  --output-dir outputs/evaluation \
  --checkpoint /path/to/checkpoint \
  --checkpoint-digest <sha256> \
  --actor-base-url http://127.0.0.1:18001/v1 \
  --actor-model Qwen2.5-3B-Instruct \
  --frozen-base-url http://127.0.0.1:18002/v1 \
  --frozen-model Qwen2.5-3B-Instruct

Score the generated predictions with an OpenAI-compatible judge:

python scripts/evaluate_predictions.py \
  --manifest data/evaluation_630.jsonl \
  --predictions outputs/evaluation/predictions.jsonl \
  --output-dir outputs/evaluation/report \
  --judge-base-url http://127.0.0.1:18002/v1

Citation

@article{jiang2026harnessrl,
  title   = {Harness-RL: Black-Box Reinforcement Learning with Action-Args
             Decoupling for Central-Agent Multi-Agent Harnesses},
  author  = {Jiang, Xinke and Zhang, Zhixin and Yang, Zhibang and Gao, Jiaran
             and Qiu, Rihong and Chen, Shijin and Chu, Xu and Zhao, Junfeng
             and Wang, Yasha},
  journal = {arXiv preprint arXiv:2608.29641},
  year    = {2026}
}