README.md

June 17, 2026 · View on GitHub

From Question Answering to Task Completion:
A Survey on Agent System and Harness Design

This repo is used for recording, tracking, and organizing papers, systems, benchmarks, and other resources on LLM-based agent system and harness design, as a supplement to our 📄 survey (PDF).

If you find any work missing or have any suggestions (papers, implementations, and other resources), feel free to open pull requests. We will add the missing entries to this repo ASAP.

✉️ Corrections & suggestions: jianyuan_guo@outlook.com (Jianyuan Guo)

📰 News

[2026.06] Repository and Survey released.

LLM-based agents are moving from passive question answering to active task completion. They perceive environments, invoke tools, maintain state, recover from errors, and act over extended horizons.

The central question of our survey is:

Where does the bottleneck in agent performance reside, in the foundation model, in the execution harness, or in the coupling between them?

We argue that agent quality emerges from the interaction between:

Factor	What it governs
Model capability	Reasoning, perception, planning, instruction following
Runtime infrastructure	Tools, context, memory, state, sandboxing, verification, recovery
Task structure	Horizon, environment, observability, oracle strength, risk
Evaluation design	Success criteria, cost, latency, safety, trace quality

Instead of treating agents as models with auxiliary tools, we view an LLM-based agent as:

Agent = Foundation Model + Execution Harness

The execution harness is the runtime system that determines what the model observes, which actions it can take, how state persists, how failures are detected, and how outcomes are verified.

Functional view of an agent as a goal-directed closed-loop system
Functional view of an agent: perception, state, reasoning, action, and feedback adaptation.

🗺️ Survey Structure

Survey structure overview
Structure of our survey: from background and paradigm shifts to harness anatomy, tasks, evaluation, and future directions.

📄 Read the full survey (PDF)

🔄 Four Paradigms of Agent Engineering

Our survey organizes the field along four engineering paradigms — from eliciting behavior to internalizing it:

Paradigm	Core Question	Main Lever	Typical Limitation
Prompt Engineering	How should we ask the model?	Instructions, exemplars, reasoning prompts	Does not manage state, tools, or recovery
Workflow & Context Engineering	What information should enter the model context?	Retrieval, memory, tool definitions, context compression	Mostly feedforward; weak recovery and verification
Harness Engineering	How do we keep the whole system on track?	Runtime loop, sandbox, state, tools, verifiers, rollback	Requires task-specific engineering and evaluation
Agent-Native Training	What agentic behavior should be internalized into the model?	RL, trajectory learning, tool-use training, self-evolution	Still requires harnesses for data, verification, and governance

Four paradigms of agent engineering: prompt → context → harness → agent-native training.

🏗️ Execution Harness Anatomy

The harness is decomposed into six runtime responsibilities that jointly determine whether model capability becomes reliable task completion:

Component	Role in Agent Systems
Observation Interface	Converts environment state into model-usable observations
Context Manager	Selects, compresses, retrieves, and formats information
Control Loop	Decides when to plan, act, verify, retry, stop, or escalate
Action Interface	Exposes tools, APIs, browser/GUI controls, shell commands, or robot actions
State and Artifact Store	Persists memory, logs, files, checkpoints, traces, and intermediate outputs
Verification and Governance	Checks correctness, safety, permissions, policy compliance, and recovery

Implementation view of an LLM-based agent with execution harness
Implementation view: foundation model coupled with a six-component execution harness.

Task-to-Harness Pressure Mapping

Task Property	Failure Pressure	Harness Response
Long horizon	State drift and context loss	Checkpoints, summaries, artifacts, memory
Partial observability	Hidden or indirect state	Structured observations and grounding
Strong oracle	Checkable outcomes	Verifier loops and repair cycles
Weak or delayed oracle	Uncertain success	Provenance, review, approval, conservative stopping
Irreversible actions	Persistent side effects	Sandbox, gates, rollback, permissions
High autonomy / low latency	Limited human correction	Budgets, controllers, logging, escalation

📖 Paper List

Papers are formatted as: Title, Author et al., venue badge, optional GitHub badge. Badge colors: red = arXiv, blue = conference/journal, lightgrey = blog/report.

Foundational Papers (Survey Mainline)

Language Models are Few-Shot Learners, Brown et al.,
Training Language Models to Follow Instructions with Human Feedback, Ouyang et al.,
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Wei et al.,
ReAct: Synergizing Reasoning and Acting in Language Models, Yao et al.,
Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al.,
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al.,
Toolformer: Language Models Can Teach Themselves to Use Tools, Schick et al.,
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, Yang et al.,
OpenHands: An Open Platform for AI Software Developers as Generalist Agents, Wang et al.,
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Jimenez et al.,
WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al.,
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, Xie et al.,
Building Effective Agents, Anthropic Engineering,
Effective Harnesses for Long-Running Agents, Anthropic Engineering,
Harness Engineering: Leveraging Codex in an Agent-First World, OpenAI,
Model Context Protocol, Anthropic,

Surveys and Meta-analyses

A comprehensive survey of agents for computer use: Foundations, challenges, and future directions, Sager et al.,
Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents, Tang et al.,
A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?, Chen et al.,
GUI-360: A Comprehensive Dataset and Benchmark for Computer-Using Agents, Mu et al.,
Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills, Jiang et al.,
Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents, Naser et al.,
Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI, Sang et al.,
Gui-ceval: A hierarchical and comprehensive chinese benchmark for mobile gui agents, Li et al.,
From static templates to dynamic runtime graphs: a survey of workflow optimization for llm agents, Yue et al.,
Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness, Nguyen et al.,
ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents, Jeon et al.,
How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses, Watanabe et al.,
The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey, Kim et al.,

Prompting, Reasoning, and Planning

Beyond Local Code Optimization: Multi-Agent Reasoning for Software System Optimization, Peng et al.,
TRACE: A Multi-Agent System for Autonomous Physical Reasoning for Seismology, Liu et al.,
Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chain, Liu et al.,
SAGE: Multi-Agent Self-Evolution for LLM Reasoning, Peng et al.,
Brain-Inspired Graph Multi-Agent Systems for LLM Reasoning, Hao et al.,
Efficient Failure Management for Multi-Agent Systems with Reasoning Trace Representation, Zhang et al.,
Improving Clinical Diagnosis with Counterfactual Multi-Agent Reasoning, You et al.,
WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback, Hu et al.,
Throttling Web Agents Using Reasoning Gates, Kumar et al.,
From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents, Seo et al.,
SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly, Zhu et al.,
From Prompt-Response to Goal-Directed Systems: The Evolution of Agentic AI Software Architecture, Alenezi et al.,
IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents, Lee et al.,

Workflow, Tool Use, and Agent Frameworks

SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair, Zhang et al.,
Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing, Li et al.,
When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems, Huang et al.,
Configuring Agentic AI Coding Tools: An Exploratory Study, Galster et al.,
TurboAgent: An LLM-Driven Autonomous Multi-Agent Framework for Turbomachinery Aerodynamic Design, Du et al.,
VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning, Li et al.,
Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery, Yu et al.,
The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data, Vallabhaneni et al.,
ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration, Chen et al.,
How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use, Lin et al.,
ASTER: Agentic Scaling with Tool-integrated Extended Reasoning, Zhang et al.,
iML: Executable, Problem-Grounded, and Broadly Exploratory Code-Driven AutoML, Le et al.,
Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use, Agarwal et al.,
Environment-Grounded Multi-Agent Workflow for Autonomous Penetration Testing, Somma et al.,
Profile-Then-Reason: Bounded Semantic Complexity for Tool-Augmented Language Agents, Enabe et al.,
Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents, Agashe et al.,
Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Search, Chen et al.,
EFT-CoT: A Multi-Agent Chain-of-Thought Framework for Emotion-Focused Therapy, Du et al.,
ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents, Wang et al.,
SolAgent: A Specialized Multi-Agent Framework for Solidity Code Generation, Chen et al.,
TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code, Huang et al.,
SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation, Chen et al.,

Harness, Runtime, Memory, and Protocols

DOVA: Deliberation-First Multi-Agent Orchestration for Autonomous Research Automation, Shen et al.,
MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web, Steiner et al.,
MemMA: Coordinating the Memory Cycle through Multi-Agent Reasoning and In-Situ Self-Evolution, Lin et al.,
Utility-Guided Agent Orchestration for Efficient LLM Tool Use, Liu et al.,
Differentiable Modal Logic for Multi-Agent Diagnosis, Orchestration and Communication, Sulc et al.,
Structurally Aligned Subtask-Level Memory for Software Engineering Agents, Shen et al.,
CodeScout: Contextual Problem Statement Enhancement for Software Agents, Suri et al.,
LDP: An identity-aware protocol for multi-agent LLM systems, Prakash et al.,
Context Engineering: From Prompts to Corporate Multi-Agent Architecture, Vishnyakova et al.,

Benchmarks, Evaluation, and Safety

MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents, Yan et al.,
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents, Kuntz et al.,
SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents, Zhang et al.,
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web, Kapoor et al.,
EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments, Liu et al.,
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use, Liu et al.,
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol, Zhu et al.,
SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents, Ying et al.,
AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems, Ferrag et al.,
Persona2web: Benchmarking personalized web agents for contextual reasoning with user history, Kim et al.,
CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation, Fu et al.,
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents, Ye et al.,
OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents, Jia et al.,
UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability, Cristescu et al.,
ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments, Ai et al.,
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development, Yang et al.,
Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents, Tang et al.,
Featurebench: Benchmarking agentic coding for complex feature development, Zhou et al.,
LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations, Pham et al.,
Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI, Qi et al.,
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies, Kulkarni et al.,
TurkingBench: A Challenge Benchmark for Web Agents, Xu et al.,
Bearcubs: A benchmark for computer-using web agents, Song et al.,
macOSWorld: A Multilingual Interactive Benchmark for GUI Agents, Yang et al.,
Towards adaptive ml benchmarks: Web-agent-driven construction, domain expansion, and metric optimization, Jia et al.,
Promoting Sustainable Web Agents: Benchmarking and Estimating Energy Consumption through Empirical and Theoretical Analysis, Krupp et al.,
Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents, Shen et al.,
Agencybench: Benchmarking the frontiers of autonomous agents in 1m-token real-world contexts, Li et al.,
LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios, Chen et al.,
Agentnoisebench: Benchmarking robustness of tool-using llm agents under noisy condition, Wang et al.,
Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark, Pulavarthy et al.,
COOP^2: Defining, Observing, and Repairing Cooperation in LLM Multi-Agent Systems, Yang et al.,
Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification, He et al.,
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering, Li et al.,
From laboratory to real-world applications: Benchmarking agentic code reasoning at the repository level, Li et al.,
Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration, Wang et al.,
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers, Bandi et al.,
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning, Yoa et al.,
Memory for autonomous llm agents: Mechanisms, evaluation, and emerging frontiers, Du et al.,
Cuaaudit: Meta-evaluation of vision-language models as auditors of autonomous computer-use agents, Sumyk et al.,
Enterpriseops-gym: Environments and evaluations for stateful agentic planning and tool use in enterprise settings, Malay et al.,
Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems, Cai et al.,
A Safety-Aware Role-Orchestrated Multi-Agent LLM Framework for Behavioral Health Communication Simulation, Cho et al.,
HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization, Chen et al.,
VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seeking, Liu et al.,
OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows, Sun et al.,
LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied Environments with Tool Augmentation, Hwangbo et al.,
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents, Liu et al.,
AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets, Fan et al.,
Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation, Qiao et al.,
ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges, Kenia et al.,
RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic, Wang et al.,
It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents, Korgul et al.,
Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents, Cartagena et al.,
ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints, Choi et al.,
Exploring the Reasoning Depth of Small Language Models in Software Architecture: A Multidimensional Evaluation Framework Towards Software Engineering 2.0, Vo et al.,
LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories, Sun et al.,
CoMAI: A Collaborative Multi-Agent Framework for Robust and Equitable Interview Evaluation, Sun et al.,
WebPII: Benchmarking Visual PII Detection for Computer-Use Agents, Zhao et al.,
Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety, Chen et al.,
The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents, Yu et al.,
GUIDE: Resolving domain bias in GUI agents through real-time web video retrieval and plug-and-play annotation, Xie et al.,
Full-duplex-bench-v3: Benchmarking tool use for full-duplex voice agents under real-world disfluency, Lin et al.,
Medbrowsecomp: Benchmarking medical deep research and computer use, Chen et al.,
RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents, Yang et al.,
OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents, Abhyankar et al.,
SCUBA: Salesforce Computer Use Benchmark, Dai et al.,
WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions, Srivastava et al.,
CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent, Lin et al.,
From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production, Shlomov et al.,
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory, Wei et al.,
KGCE: Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models, Liu et al.,
Agentlongbench: A controllable long benchmark for long-contexts agents via environment rollouts, Fang et al.,
Swe context bench: A benchmark for context learning in coding, Zhu et al.,
EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems, Zhang et al.,
BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents, Zhang et al.,
GraphSkill: Documentation-Guided Hierarchical Retrieval-Augmented Coding for Complex Graph Reasoning, Wang et al.,
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios, Song et al.,
Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems, Wang et al.,
ArchBench: Benchmarking Generative-AI for Software Architecture Tasks, Adnan et al.,
Can LLM Agents Be CFOs? A Benchmark for Resource Allocation in Dynamic Enterprise Environments, Han et al.,
Efficient benchmarking of ai agents, Ndzomga et al.,
Beliefshift: Benchmarking temporal belief consistency and opinion drift in llm agents, Myakala et al.,
Memorycd: Benchmarking long-context user memory of llm agents for lifelong cross-domain personalization, Zhang et al.,
SciVisAgentBench: A benchmark for evaluating scientific data analysis and visualization agents, Ai et al.,
ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities, Zanoli et al.,
YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution, He et al.,
HippoCamp: Benchmarking Contextual Agents on Personal Computers, Yang et al.,
PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools, Feng et al.,
Agenthazard: A benchmark for evaluating harmful behavior in computer-use agents, Feng et al.,
How well do agentic skills work in the wild: Benchmarking llm skill usage in realistic settings, Liu et al.,
Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation, Li et al.,
Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks, Xu et al.,
AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning, Zha et al.,
AppSelectBench: Application-Level Tool Selection Benchmark, Chen et al.,
Arabic Prompts with English Tools: A Benchmark, Kubrak et al.,
Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents, V et al.,
IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks, Mateega et al.,
Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generation, Kica et al.,
Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations, Honarvar et al.,
NAAMSE: Framework for Evolutionary Security Evaluation of Agents, Pai et al.,

Domain Agents

Coding & Software Engineering

Towards autonomous normative multi-agent systems for Human-AI software engineering teams, Dam et al.,
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development, Agarwal et al.,
Agyn: A Multi-Agent System for Team-Based Autonomous Software Engineering, Benkovich et al.,
CoAct-1: Computer-using Multi-agent System with Coding Actions, Song et al.,
Self-Organizing Multi-Agent Systems for Continuous Software Development, Lyu et al.,
Fault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach to Safe Autonomous Execution, Yan et al.,
ESAA: Event Sourcing for Autonomous Agents in LLM-Based Software Engineering, Brito dos Santos Filho et al.,
FLARE: Agentic Coverage-Guided Fuzzing for LLM-Based Multi-Agent Systems, Hui et al.,
CCD-CBT: Multi-Agent Therapeutic Interaction for CBT Guided by Cognitive Conceptualization Diagram, Liu et al.,
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks, Xie et al.,
``I'm Not Reading All of That'': Understanding Software Engineers' Level of Cognitive Engagement with Agentic Coding Assistants, Catalan et al.,
Budget-aware agentic routing via boundary-guided training, Zhang et al.,
Programming with Pixels: Can Computer-Use Agents do Software Engineering?, Aggarwal et al.,
SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents, Ding et al.,
MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering, Guo et al.,
AgentStepper: Interactive Debugging of Software Development Agents, Hutter et al.,
Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, Chen et al.,
Aidev: studying ai coding agents on github, Li et al.,
Wink: Recovering from Misbehaviors in Coding Agents, Nanda et al.,
AgenticTyper: Automated Typing of Legacy Software Projects Using Agentic AI, Pohle et al.,
SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents, Kon et al.,
XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights, Joshi et al.,

Web, GUI & Computer-Use

The dawn of gui agent: A preliminary case study with claude 3.5 computer use, Hu et al.,
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience, Sun et al.,
``Are We Done Yet?'': A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents, Sumyk et al.,
Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, Xu et al.,
Are Autonomous Web Agents Good Testers?, Chevrot et al.,
Efficient agent training for computer use, He et al.,
VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos, Lu et al.,
OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution, Zhang et al.,
WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements, Teoh et al.,
Go-Browse: Training Web Agents with Structured Exploration, Gandhi et al.,
WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents, Liu et al.,
WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering, He et al.,
DPO Learning with LLMs-Judge Signal for Computer Use Agents, Luo et al.,
Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent, Cheng et al.,
Scaling Agents for Computer Use, Gonzalez-Pumariega et al.,
Surfer 2: The Next Generation of Cross-Platform Computer Use Agents, Andreux et al.,
Fara-7B: An Efficient Agentic Model for Computer Use, Awadallah et al.,
Camels can use computers too: System-level security for computer use agents, Foerster et al.,
Evocua: Evolving computer use agents via learning from scalable synthetic experience, Xue et al.,

Embodied & Robotics

Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration, Sun et al.,
CGoT: A Novel Inference Mechanism for Embodied Multi-Agent Systems Using Composable Graphs of Thoughts, Nie et al.,
Advances and Innovations in the Multi-Agent Robotic System (MARS) Challenge, Kang et al.,
Can a Robot Walk the Robotic Dog: Triple-Zero Collaborative Navigation for Heterogeneous Multi-Agent Systems, Wang et al.,
From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?, Shaji et al.,

Agent-Native Training and RL

Can We Trust a Black-box LLM? LLM Untrustworthy Boundary Detection via Bias-Diffusion and Multi-Agent Reinforcement Learning, Zhou et al.,
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning, Qi et al.,
Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning, Wei et al.,
WorkForceAgent-R1: Incentivizing reasoning capability in llm-based web agents via reinforcement learning, Zhuang et al.,
Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning, Urgun et al.,
ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents, Lai et al.,
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning, Chen et al.,
Reinforcement Learning Integrated Agentic RAG for Software Test Cases Authoring, Hariharan et al.,
WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning, Xu et al.,
Adaptive Robust Estimator for Multi-Agent Reinforcement Learning, Li et al.,
Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning, Jiang et al.,
LangMARL: Natural Language Multi-Agent Reinforcement Learning, Yao et al.,
MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learning, Nesterova et al.,
ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning, Chen et al.,
MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools, Wang et al.,
ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning, Zhou et al.,
DynaWeb: Model-Based Reinforcement Learning of Web Agents, Ding et al.,
Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization, Wang et al.,
SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training, Wen et al.,
Content Caching-Assisted Vehicular Edge Computing Using Multi-Agent Graph Attention Reinforcement Learning, Shen et al.,
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning, Sun et al.,
LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems, Duan et al.,
Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning, Li et al.,
AgriWorld: A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents, Zhang et al.,
When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making, Liu et al.,
LATS: Large Language Model Assisted Teacher-Student Framework for Multi-Agent Reinforcement Learning in Traffic Signal Control, Zhang et al.,
Multi-Agent LLM Governance for Safe Two-Timescale Reinforcement Learning in SDN-IoT Defense, Jamshidi et al.,
ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning, Gao et al.,
PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy, Duy et al.,
Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL, Li et al.,
Guided by trajectories: Repairing and rewarding tool-use trajectories for tool-integrated reasoning, Gong et al.,
SWE-World: Building Software Engineering Agents in Docker-Free Environments, Sun et al.,
SuperLocalMemory: Privacy-Preserving Multi-Agent Memory with Bayesian Trust Defense Against Memory Poisoning, Bhardwaj et al.,

Other High-Relevance Systems and Reports

Coral: Towards autonomous multi-agent evolution for open-ended discovery, Qu et al.,
Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems, Hosseini et al.,
ProductResearch: Training E-Commerce Deep Research Agents via Multi-Agent Synthetic Trajectory Distillation, Wang et al.,
Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusion, Liu et al.,
AIVV: Neuro-Symbolic LLM Agent-Integrated Verification and Validation for Trustworthy Autonomous Systems, Kwon et al.,
SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis, Chen et al.,
Deception and Communication in Autonomous Multi-Agent Systems: An Experimental Study with Among Us, Milkowski et al.,
From Perception to Autonomous Computational Modeling: A Multi-Agent Approach, Wilke et al.,
Advancing language multi-agent learning with credit re-assignment for interactive environment generalization, He et al.,
Unleashing Diverse Thinking Modes in LLMs through Multi-Agent Collaboration, He et al.,
Towards Ethical Multi-Agent Systems of Large Language Models: A Mechanistic Interpretability Perspective, Lee et al.,
Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montage, Hu et al.,
Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generation, Zhang et al.,
The PBSAI Governance Ecosystem: A Multi-Agent AI Reference Architecture for Securing Enterprise AI Estates, Willis et al.,
OMGs: A multi-agent system supporting MDT decision-making across the ovarian tumour care continuum, Zhang et al.,
Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants, Herrera et al.,
EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair, Chen et al.,
Evoscientist: Towards multi-agent evolving ai scientists for end-to-end scientific discovery, Lyu et al.,
Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QA, Abbas et al.,
Multi-Agent Collaboration for Automated Design Exploration on High Performance Computing Systems, Menon et al.,
ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamics, Zhao et al.,

🛠️ Representative Systems

System	Domain	Harness Style	Key Design
SWE-agent	Software engineering	ReAct-style loop	Shell/editor interface, iterative inspect-edit-test
Agentless	Software engineering	Fixed pipeline	Localization, repair generation, patch selection
AutoCodeRover	Software engineering	Search-guided repair	Repository-aware code search and validation
OpenHands	Software engineering	General runtime agent	Shell, editor, browser/tools, iterative execution
PatchPilot	Software engineering	Structured repair workflow	Reproduction, localization, validation, refinement
Codex	Software engineering	Managed coding agent	Code execution, editing, task management
AppAgent	Mobile GUI	Exploration-driven	Smartphone interaction as agent task
Mobile-Agent	Mobile GUI	Vision-grounded loop	OCR, icon detection, visual perception
OpenSquilla	General agents	Token-efficient runtime	Higher intelligence density under fixed budget
Sibyl-AutoResearch	Research agents	Self-evolving harness	Trial-and-error memory, evidence gates, repair loops

📊 Benchmarks at a Glance

Benchmark	Focus	Environment	Primary Signal
SWE-bench	Coding	Real GitHub issues	Resolution rate
WebArena	Web	Realistic websites	Task success
VisualWebArena	Multimodal web	Visual web tasks	Task success
OSWorld	Desktop	Real OS	Multi-app success
Terminal-Bench	Terminal / coding	Command line	Task success
TheAgentCompany	Enterprise tasks	Simulated company	Task success
AgentBench	General agents	Interactive environments	Task completion
GAIA	General assistants	Multi-step tasks	Accuracy
MLAgentBench	ML engineering	Experimentation tasks	Performance improvement
OS-Harm	Safety	Desktop computer use	Harmful action rate
LoCoMo	Long-term memory	Multi-session dialogue	QA / consistency
MCPWorld	API + GUI	Hybrid tool environments	Task success / tool use
MCPAgentBench	MCP tool use	Real MCP servers	Tool-use success

Beyond headline task success, benchmark leaderboards also reveal model–harness interaction effects. On Terminal-Bench 2.0, the same foundation model can span a wide accuracy range depending on the execution harness, suggesting that reported scores reflect paired model–harness systems rather than model capability alone.

Terminal-Bench 2.0 model-harness effects
(a) Terminal-Bench 2.0 accuracy across model–harness pairings; dashed lines connect the same model under different harnesses. (b) Within-model variation across harnesses.

💡 Value-Aware Evaluation

Agent evaluation should not only report task success. Practical deployment also depends on:

Cost: tokens, API calls, tool calls, compute, infrastructure
Latency: wall-clock time, step count, P95/P99 tail latency
Reliability: repeated-run consistency, stress robustness, recovery
Safety: policy violations, permission boundaries, side effects
Process quality: trace inspectability, provenance, verifier use, rollback

Two systems with the same success rate may differ dramatically in cost, risk, auditability, and deployment value.

🤝 Contributing

Contributions are welcome! You can help by:

Adding missing papers, systems, or benchmarks
Fixing broken links or metadata
Suggesting better taxonomy categories
Adding concise summaries for important papers
Opening issues for correction, clarification, or discussion

Paper Formatting Guidelines

* **_Paper Title_**, Author et al., [![arXiv](https://img.shields.io/badge/arXiv-YY.MM-red?labelColor=grey)](PAPER_URL) [![GitHub](https://img.shields.io/badge/GitHub-Code-white?logo=github)](CODE_URL)

Badge Colors

red — arXiv preprints
blue — conference / journal papers
lightgrey — blogs, reports, product pages

📑 Citation

If this repository or the survey is useful for your work, please cite:

@misc{guo2026agentengineering,
  title = {From Question Answering to Task Completion: A Survey on Agent System and Harness Design},
  author = {Guo, Jianyuan and Hao, Zhiwei and Wang, Chengcheng and Fan, Cheng and Luo, Tingzhang and Li, Hongguang and Gao, Ying and Mei, Hefei and Peng, Jiankun and Xu, Rongjian and Dong, Minjing and Wu, Han and Zheng, Mengyu and Han, Kai and Wang, Shiqi and Xu, Chang and Wang, Yunhe},
  year = {2026},
  howpublished = {Preprint},
}

📄 License

This repository is released under the MIT License unless otherwise specified. Paper copyrights belong to their respective authors.

From Question Answering to Task Completion:A Survey on Agent System and Harness Design

From Question Answering to Task Completion:
A Survey on Agent System and Harness Design