Steer, Don't Solve: Training Small Critic Models for Large Code Agents

Code, configs, and launch scripts for training and running a small critic model next to a large coding agent. The critic reads the agent's trajectory every k steps and returns a short structured critique: detected error categories, evidence, a recovery action, task status, and one line of guidance. It steers the agent and never writes the patch. A 4B or 8B critic trained this way improves six different coding agents on SWE-bench Verified, at a small fraction of the cost of using a frontier model as the critic.

How the critics are trained

SFT. A frozen coding agent runs on 500 R2E-Gym tasks from matplotlib, moto, and sympy (disjoint from SWE-bench Verified). Every 5 steps, Claude Opus 4.6 writes a critique of the trajectory so far under a high-level prompt that forbids code and asks for one or two sentences of guidance. The (trajectory prefix, critique) pairs are the SFT data for a Qwen3-4B or Qwen3-8B critic.

DPO. The SFT critic samples ten critiques at each critique point, an LLM judge picks the best and the worst, and the resulting preference pairs are used for DPO on top of the 4B SFT critic.

Released models and data

Everything the paper reports is under the code-critic-model organization on Hugging Face. <base>-Critic-SFT is the critic trained on the full mixed corpus; a suffix names the one thing an ablation changes.

Model	Base	Trained on	Paper
Qwen3-8B-Critic-SFT	Qwen3-8B	critic-sft-cwm-qwen	main 8B critic (Tables 1 to 4)
Qwen3-4B-Critic-SFT	Qwen3-4B-Instruct-2507	critic-sft-cwm-qwen	4B SFT critic (Tables 1, 3)
Qwen3-4B-Critic-SFT-DPO	Qwen3-4B-Critic-SFT	1,409 preference pairs	4B SFT + DPO critic (Table 1)
Qwen3-8B-Critic-SFT-CWM-only	Qwen3-8B	critic-sft-cwm-only	corpus ablation (Table 3)
Qwen3-8B-Critic-SFT-Qwen-only	Qwen3-8B	critic-sft-qwen-only	corpus ablation (Table 3)
Qwen3-4B-Critic-SFT-Qwen-only	Qwen3-4B-Instruct-2507	critic-sft-qwen-only	corpus ablation (Table 3)
Qwen3-8B-Critic-SFT-Detailed-Prompt	Qwen3-8B	critic-sft-cwm-only-detailed-prompt	prompt ablation (Table 4)

Dataset	Records	Agent trajectories	Teacher prompt
critic-sft-cwm-qwen	6,447	CWM-32B + Qwen3-Next-80B-A3B	high-level
critic-sft-cwm-only	4,532	CWM-32B	high-level
critic-sft-qwen-only	1,915	Qwen3-Next-80B-A3B	high-level
critic-sft-cwm-only-detailed-prompt	3,135	CWM-32B	detailed

Each model card lists the exact paper rows it backs, its training data and hyperparameters, and the serving command. The local copies of the SFT corpora live in finetuning/prm_sft_*/.

Main results

Resolve rate, localization rate, non-empty patch rate, and non-loop rate on SWE-bench Verified (500 instances). Each configuration reports the better of k=5 and k=10. Qwen3-4B and Qwen3-8B alone are the untrained base models used as critics. Bold marks the best non-Opus value in each column of an agent block.

Coding agent	Critic	Resolved %	Localized %	Non-Empty %	Non-Loop %
Qwen3-32B	No critic	8.8	49.2	74.8	92.0
	Qwen3-4B	10.2	55.4	90.0	95.2
	Qwen3-4B + SFT	11.4	52.4	74.6	90.4
	Qwen3-4B + SFT + DPO	14.4	52.2	69.2	95.0
	Qwen3-8B	10.0	49.4	73.0	93.0
	Qwen3-8B + SFT	13.8	49.8	65.8	93.0
	Opus 4.6 (upper bound)	26.8	67.8	82.4	92.4
Qwen3-Next-80B-A3B	No critic	20.0	69.8	92.6	95.0
	Qwen3-4B	20.2	70.4	99.8	97.8
	Qwen3-4B + SFT	24.2	76.8	99.2	98.2
	Qwen3-4B + SFT + DPO	26.2	78.4	99.2	98.4
	Qwen3-8B	20.6	69.4	97.6	94.8
	Qwen3-8B + SFT	25.2	75.6	99.8	96.2
	Opus 4.6 (upper bound)	38.2	81.4	100.0	99.4
GPT-OSS-20B	No critic	3.0	11.8	58.4	1.4
	Qwen3-4B	6.8	36.6	98.8	3.0
	Qwen3-4B + SFT	9.8	52.6	99.6	5.0
	Qwen3-4B + SFT + DPO	14.8	55.0	99.6	5.6
	Qwen3-8B + SFT	13.0	52.0	99.4	5.4
GLM-4.7-Flash-30B-A3B	No critic	21.6	34.8	36.0	54.8
	Qwen3-4B	29.8	62.2	70.4	67.2
	Qwen3-4B + SFT	35.2	69.6	79.6	74.2
	Qwen3-4B + SFT + DPO	35.8	71.0	79.2	75.0
	Qwen3-8B + SFT	37.6	73.4	83.4	70.6
GPT-OSS-120B (medium reasoning)	No critic	20.4	57.4	95.2	94.2
	Qwen3-4B	16.8	46.6	100.0	99.2
	Qwen3-4B + SFT	31.2	75.2	99.8	98.0
	Qwen3-4B + SFT + DPO	34.8	75.8	99.2	97.2
	Qwen3-8B + SFT	31.4	74.8	99.2	96.8
o3-mini	No critic	19.0	69.4	99.4	100.0
	Qwen3-4B	20.4	64.4	100.0	100.0
	Qwen3-4B + SFT	27.6	73.2	100.0	99.6
	Qwen3-4B + SFT + DPO	28.2	75.2	100.0	99.4
	Qwen3-8B + SFT	29.4	71.0	100.0	98.8

Cost against resolve rate for the same six agents. Circles are the agents alone, stars add the 8B SFT critic, and hollow circles add the Opus 4.6 teacher as critic. Cost covers the agent plus the critic.

Multi-SWE-bench. Resolve rate on 300 Multi-SWE-bench instances with the 8B critic at k=5. The critic saw only Python during training.

Coding agent	No critic	+ Qwen3-8B-Critic-SFT
Qwen3-Next-80B-A3B	9.7	11.7
Qwen3-32B	1.3	3.3

Training corpus. Resolve rate for critics trained on the three SFT corpora.

Critic size	Corpus	Qwen3-Next-80B-A3B	Qwen3-32B
none		20.0	8.8
4B	Qwen-only	24.6	10.8
4B	Qwen + CWM	24.2	11.4
8B	CWM-only	23.6	11.4
8B	Qwen-only	26.2	10.6
8B	Qwen + CWM	25.2	13.8

Teacher prompt. Resolve rate for 8B critics trained on detailed versus high-level teacher critiques. Both use the same high-level prompt at inference.

Critic	Qwen3-Next-80B-A3B	Qwen3-32B
No critic	20.0	8.8
8B + SFT (detailed-prompt critiques)	24.6	12.6
8B + SFT (high-level critiques, ours)	25.2	13.8

Using this repository

• QUICKSTART.md: clone, install, serve a critic, run an agent on SWE-bench Verified, and retrain the critic, in a handful of commands.
• HANDOVER.md: directory map, where data lives on the cluster, naming conventions, the LiteLLM path for running without AWS, and end-to-end commands for training and inference.

The agent loop is a fork of mini-swe-agent with hooks that call the critic every k steps (mini-swe-agent/src/minisweagent/agents/default_prm.py). Critic training uses LLaMA-Factory with the configs in finetuning/.

Citation

@misc{gandhi2026steerdontsolvetraining,
  title={Steer, Don't Solve: Training Small Critic Models for Large Code Agents},
  author={Shubham Gandhi and Yiqing Xie and Atharva Naik and Ruichen Zhu and Carolyn Rose},
  year={2026},
  eprint={2606.21811},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2606.21811}
}