Awesome LLM Agent Trajectory Analysis: Failure Attribution, Enhancement, Self-Evolving, Optimization, Repair, Debugging
August 25, 2026 · View on GitHub
A curated repository for LLM Agent Trajectory Analysis, based on the survey: A Survey for LLM Agent Trajectory Analysis: From Failure Attribution to Enhancement (IEEE Transactions on Software Engineering (TSE), 2026).
🔥🔥 Our paper is now published in IEEE Transactions on Software Engineering (TSE 2026)! Check out "A Survey for LLM Agent Trajectory Analysis: From Failure Attribution to Enhancement" on IEEE Xplore, ResearchGate, GitHub, or Hugging Face.
This repository targets key search topics in this area, including LLM agent failure attribution, trajectory-based debugging, root cause analysis, agent repair, and enhancement/optimization of LLM agent systems.
This repo organizes the field along five dimensions:
- Failure Taxonomy
- Failure Attribution
- System Enhancement and Optimization
- Trajectory Monitoring and Analysis Tools
- Datasets and Benchmarks
Contact
For questions, suggestions, or collaboration opportunities, please feel free to reach out:
Junjie Wang
📧 Email: junjie@iscas.ac.cn
Mengzhuo Chen
📧 Email: chenmengzhuo2023@iscas.ac.cn
Table of Contents
- Why This Topic
- Keywords / Search Terms
- Survey at a Glance
- Paper Collection Protocol
- Awesome Paper List
- Open Challenges
- How to Contribute
- Citation
- Contact
- License
Why This Topic
LLM-based agent systems increasingly behave like a new software paradigm. Their failures are often not deterministic code bugs, but are embedded in long, language-heavy, multi-step trajectories (reasoning, communication, tool use, environment feedback). This makes trajectory analysis the core path for:
- Diagnosing failures
- Attributing root causes
- Guiding system-level fixes and optimization
Keywords / Search Terms
LLM agent failure attribution, failure attribution, agent failure analysis, trajectory analysis, trajectory-based debugging, root cause analysis, agent repair, enhancement optimization, LLM agent optimization, agent debugging, AgentOps, LLM multi-agent systems
Survey at a Glance
- Time window covered: early 2025 to April 2026
- Total collected papers: 55
- Core technical focus:
- Failure attribution (22 papers)
- Enhancement/optimization (14 papers)
- Supporting foundations:
- Failure taxonomy (9 papers)
- Monitoring/analysis tools (7 papers)
- Benchmarks/datasets (10 papers)
Key observations from the survey:
- Attribution methods have evolved from prompting-based inspection to causal inference, specialized tracer models, and intervention-driven runtime analysis.
- Step-level attribution remains challenging (around 40% in common benchmark settings).
- Benchmark diversity and observability are still bottlenecks.
Paper Collection Protocol
The survey uses:
- Automated search on ACM DL, IEEE Xplore, arXiv, DBLP
- Manual search in major SE/AI venues
- Backward snowballing
- Inclusion/exclusion criteria + quality checklist
From 1,652 initially retrieved papers, 55 were retained for final analysis.
Awesome Paper List
Failure Taxonomy
- TRAIL: Trace Reasoning and Agentic Issue Localization
- Aegis: Taxonomy and Optimizations for Overcoming Agent-environment Failures in LLM Agents
- Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
- Where LLM Agents Fail and How They Can Learn From Failures
- Why Do Multi-Agent LLM Systems Fail?
- How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations
- AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economy
Failure Attribution Methods for LLM Agents
Pattern Analysis-Based
- Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysis (FAMAS)
- Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
- CORRECT: Condensed eRror Recognition via Knowledge Transfer in Multi-Agent Systems
- Scope Delineation Before Localization (SDBL)
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamics
LLM Reasoning-Based
- Which Agent Causes Task Failures and When?
- Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution (ECHO)
- RAFFLES: Reasoning-based Attribution of Faults for LLM Systems
- Automatic Failure Attribution and Critical Step Prediction based on Causal Inference (CDC-MAS)
- Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems
- From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems
- AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- CodeTracer: Towards Traceable Agent States for Failure Attribution
- ERRORPROBE: Towards Self-Improving Error Diagnosis in Multi-Agent Systems
- LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures
- SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
- TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
- TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Model Fine-Tuning-Based
- AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
- GraphTracer: Graph-Guided Failure Tracing in LLM Agents
- Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
Dynamic Runtime-Based
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
- AgentDebug (Where LLM Agents Fail and How They can Learn From Failures)
- TraceElephant: Seeing the Whole Elephant for Failure Attribution in LLM-based Multi-Agent Systems
- Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
Enhancement, Optimization, and Repair
Structural and Workflow Optimization
- Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents
- Maestro: Joint Graph & Config Optimization for Reliable AI Agents
- Failure-Driven Workflow Refinement (CE-Graph)
- Instruction-Level Weight Shaping (ILWS)
Agent Internal Optimization
- SCOPE: Prompt Evolution for Enhancing Agent Effectiveness
- AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering
- ReCreate: Reasoning and Creating Domain Agents Driven by Experience
- SEAlign: Alignment Training for Software Engineering Agent
- Trajectory-Informed Memory Generation for Self-Improving Agent Systems
- Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills
Runtime and Supervisory Optimization
- Improving the Efficiency of LLM Agent Systems through Trajectory Reduction (AgentDiet)
- Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systems (SUPERVISOR AGENT)
- Wink: Recovering from Misbehaviors in Coding Agents
- Process-Centric Analysis of Agentic Software Systems
Trajectory Monitoring, Debugging, and Analysis Tools
System-Level Monitoring and Passive Diagnosis
- AgentSight: System-Level Observability for AI Agents using eBPF
- Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems
- AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories
- Agent Trajectory Explorer: Visualizing and Providing Feedback on Agent Trajectories
Interactive Analysis and Active Debugging
- Interactive Debugging and Steering of Multi-Agent AI Systems (AGDebugger)
- XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflows
- DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviors
Datasets and Benchmarks for Failure Attribution and Repair
Production Web-Agent Trajectory Benchmarks
- ClawBench: A Benchmark for Real-World Web Agent Tasks
: 283 real-world web tasks across 163 websites, with replay video, screenshots, HTTP traffic, browser actions, and agent messages. This provides complementary multi-layer trajectories for studying web-agent failures and process quality; it is not a failure-attribution method itself.
Process and Span-Level Error Localization
-
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
: 1,000 tool-augmented agent trajectories with 8,509 human-labeled step annotations for step-level process quality diagnosis.
-
CodeTracer / CodeTraceBench: Towards Traceable Agent States
: 4,316 coding-agent trajectories with human-verified stage- and step-level annotations for failure localization.
-
TELBench / DRIFT: Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories
Context Retrieval Benchmarks
- ContextBench: A Benchmark for Context Retrieval in Coding Agents
Real-World Failure Collection
- Who&When
- TRAIL
- AgentErrorBench
- TraceElephant
- ClawBench: Can AI Agents Complete Everyday Online Tasks?
- Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows
: 307 trajectories, lifecycle-level annotation + repair strategy
- AgentRx
: first unrecoverable failure step annotation
- CodeTraceBench
- MP-Bench
Synthetic Data via Error Injection
- Aegis
- CORRECT-Error
Others and Empirical Studies
- Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
- MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability
- Trajectory Guard — A Lightweight, Sequence-Aware Model for Real-Time Anomaly Detection in Agentic AI
- From Features to Actions: Explainability in Traditional and Agentic AI Systems
- Signals: Trajectory Sampling and Triage for Agentic Interactions
- Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios
- Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
- The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
Open Challenges
- Improve attribution accuracy through stronger causal modeling of trajectories.
- Move from attribution accuracy metrics to repair-utility and enhancement-oriented metrics.
- Build richer, larger, and more diverse full-observability benchmarks.
- Shift evaluation from binary success/failure to multi-dimensional capability assessment.
- Evolve from fragmented tools toward integrated AgentOps governance.
- Expand from controlled benchmarks to production and domain-specific settings (coding, GUI, web, embodied agents).
- Extend failure definitions beyond task completion to constraint fulfillment, alignment, and safety.
How to Contribute
Contributions are welcome. You can help by:
- Adding new papers (2026+)
- Improving taxonomy/category mapping
- Adding benchmark metadata and reproducibility resources
- Fixing links and metadata errors
Please open a PR with:
- Paper title
- Link (arXiv/DOI/project)
- Venue and date
- Suggested category (one of the five dimensions)
- Short note (optional)
Citation
@article{wang2026surveytrajectory,
author={Wang, Junjie and Wang, Yawen and Chen, Mengzhuo and Xie, Xiaofei and Chen, Chunyang and Mu, Fangwen and Liu, Zhe and Wang, Qing},
journal={IEEE Transactions on Software Engineering},
title={A Survey for LLM Agent Trajectory Analysis: From Failure Attribution to Enhancement},
year={2026},
pages={1-23},
doi={10.1109/TSE.2026.3717765},
url={https://ieeexplore.ieee.org/abstract/document/11626967}
}
License
This repository is released under the MIT License.