README.md
July 8, 2026 Ā· View on GitHub
Awesome LLM Agent Security
A curated list of resources about LLM agent security, vulnerabilities, attacks, and their applications in cybersecurity.
š Contents
| š Research | šÆ Threat Model | š ļø Build Agent | š Results | š¼ Solutions |
|---|---|---|---|---|
| Papers & Publications | Threats & Vulnerabilities | Development Guides for cybersecurity | Successful cases | Tools Overview |
š° Fundamentals
Basic Concepts
LLM Agent - is an intelligent system used by a large language model to perform tasks.
Note
MultiAgent: When there is an ensemble of many agents that perform one large task - this is called a multi-agent system. It is based on a large language model that is responsible for task planning and decision making.
Memory
Memory in autonomous agents can be categorized into two main types:
ā¢ Short-Term Memory: This refers to the agent's ability to utilize in-context learning, where it retains information temporarily during a single interaction. This is often limited by the model's context window, which restricts the amount of information it can process at once.
ā¢ Long-Term Memory: This allows the agent to store and recall information over extended periods. Long-term memory is typically managed through external vector stores, enabling the agent to retrieve relevant information quickly. This capability is crucial for tasks that require knowledge accumulation and recall over time.
Memory plays a vital role in how agents learn from past experiences, refine their actions, and improve their performance in future tasks.
Planning
Planning involves the agent's ability to break down complex tasks into manageable subgoals. This process can be enhanced through various techniques:
Task Decomposition: The agent can decompose a large task into smaller, more manageable steps. Techniques like Chain of Thought (CoT) prompting encourage the model to think step-by-step, making it easier to tackle complex problems.
Self-Reflection: Agents can evaluate their past actions, learn from mistakes, and refine their strategies. This iterative process helps improve decision-making and planning over time.
External Planning Tools: Some agents may utilize classical planning methods, such as the Planning Domain Definition Language (PDDL), to generate structured plans based on predefined domains.
Effective planning is essential for agents to navigate complex tasks and adapt to changing circumstances.
Action
Action refers to the execution of tasks based on the agent's planning and memory. This involves:
Tool Use: Agents can interact with external APIs and tools to gather information, execute code, or perform specific functions that extend their capabilities beyond what is encoded in their model weights.
ReAct Framework: This framework integrates reasoning and action, allowing agents to generate reasoning traces alongside their actions. This helps in understanding the rationale behind decisions and improving future actions.
Dynamic Interaction: Agents can adapt their actions based on real-time observations and feedback from their environment, allowing for more responsive and intelligent behavior.
š Framework Comparison
š Framework Comparison
| Framework | Key Features | Focus Area | License | Language Support | Distributed Systems |
|---|---|---|---|---|---|
| LangChain | ā¢ Chain-based architecture ā¢ Memory management ā¢ Tool integration | General purpose LLM apps | MIT | Python, JavaScript | Limited |
| AutoGPT | ā¢ Autonomous goal pursuit ā¢ Long-term memory ā¢ Self-prompting | Autonomous agents | MIT | Python | No |
| AgentGPT | - Web-based interface - Task decomposition - Visual workflow | Task automation | MIT | TypeScript | Yes |
| BabyAGI | - Task prioritization - Simple architecture - Learning focus | Research & Education | MIT | Python | No |
| Lyzr | - Enterprise security - Scalable architecture - Pre-built agents | Enterprise solutions | Commercial | Python | Yes |
| CrewAI | - Multi-agent collaboration - Role-based agents - Team coordination | Complex workflows | Apache 2.0 | Python | Yes |
Key Differences:
| Category | Features | Examples/Details |
|---|---|---|
| Architecture Focus | ā¢ Task-oriented ā¢ Chain-based ā¢ Multi-agent ā¢ Enterprise-grade | ā¢ BabyAGI, AgentGPT ā¢ LangChain ā¢ CrewAI ā¢ Lyzr |
| Use Case Optimization | ā¢ Research & Experimentation ā¢ Enterprise Applications ā¢ Personal Automation ā¢ Educational Purposes | ā¢ Academic projects ā¢ Business solutions ā¢ Individual tools ā¢ Learning platforms |
| Development Approach | ā¢ Low-code solutions ā¢ Programming-intensive ā¢ Visual builders ā¢ API-first design | ā¢ No-code platforms ā¢ Custom development ā¢ Drag-and-drop interfaces ā¢ API integration |
| Deployment Options | ā¢ Cloud-native ā¢ Self-hosted ā¢ Hybrid deployment ā¢ Edge computing support | ā¢ Cloud platforms ā¢ On-premise solutions ā¢ Mixed environments ā¢ Edge devices |
| Integration Capabilities | ā¢ API connectivity ā¢ Database support ā¢ Third-party tools ā¢ Custom extensions | ā¢ REST/GraphQL APIs ā¢ Various DB systems ā¢ External services ā¢ Custom plugins |
š¤ LLM Agents in Cybersecurity
Agent Core Properties
|
|
Security Applications & Benefits
|
|
Advantages in Security Context
|
|
Creation Methods
| Method | Description |
|---|---|
| Human-designed | Agents with specific security roles |
| Auto-generated | Agents created by LLMs for specialized tasks |
| Hybrid | Approaches combining human expertise and AI capabilities |
Security Landscape
š OWASP Top 10 for AI Agents (Non official)
| Category | Risk | Description |
|---|---|---|
| AAC-01 | Authorization & Control Hijacking | Unauthorized control of agent actions |
| ACS-02 | Critical Systems Interaction | Unsafe interaction with critical systems |
| AGI-03 | Goal & Instruction Manipulation | Malicious modification of agent objectives |
| AHE-04 | Hallucination Exploitation | Exploitation of agent's false assumptions |
| AIC-05 | Impact Chain & Blast Radius | Cascading effects of agent actions |
| AMC-06 | Memory & Context Manipulation | Tampering with agent's memory systems |
| AOR-07 | Orchestration Exploitation | Multi-agent system vulnerabilities |
| ARE-08 | Resource Exhaustion | DoS and resource depletion attacks |
| ASC-09 | Supply Chain Attacks | Compromised dependencies and components |
| AKP-10 | Knowledge Base Poisoning | Contamination of agent's knowledge base |
Practice Labs & Applications
| Project | Type | Features | Purpose |
|---|---|---|---|
| Damn Vulnerable LLM Agent | Educational Lab | - ReAct agent testing - Prompt injection scenarios - SQL injection practice | Learning prompt & ReAct injection techniques |
| Medusa | Testing Platform | - Agent vulnerability testing - Security assessment - Attack simulation | Practical security testing |
| PromptTrace | Training Platform | - 10 attack labs - 15-level CTF (The Gauntlet) - Context Trace prompt stack visibility | Free hands-on AI security training aligned with OWASP Top 10 for LLMs |
š Research & Publications
Academic Papers
| Title | Authors | Year | Key Findings |
|---|---|---|---|
| Internal Safety Collapse in Frontier LLMs | Wu et al. | 2026 | - Novel failure: task completion overrides safety - Agents generate exploits/toxins autonomously - Jailbreaks any frontier LLM in pass@3 - Code |
| Security Concerns with AI Agents | VPNRanks | 2024 | - 52.5% data leakage predicted by 2025 - Market growth to $7.41B - PII exposure risks |
| Key Challenges in AI Agent Security | Tal Eliyahu | 2024 | - Confidentiality risks - Integrity concerns - Availability threats |
| Beyond RCE: Autonomous Code Execution | Security Runners | 2024 | - Code execution risks - Agent autonomy threats - Security implications |
| Exploiting Huggingface's Assistants | Lasso Security | 2023 | - Data extraction vulnerabilities - Assistant exploitation - Security measures |
Security Tools & Frameworks
| Project | Type | Description | Features |
|---|---|---|---|
| HackSynth | Framework | AI security testing framework | - Vulnerability assessment - Attack simulation - Security validation |
| OsintAGI | Tool | OSINT automation framework | - Intelligence gathering - Data analysis - Automated research |
| Agent-Smith | Security Tool | Agent security testing | - Behavior analysis - Vulnerability detection - Security assessment |
| AI-OPS | Platform | Security operations for AI | - Threat detection - Response automation - Security monitoring |
| Darkmoon | Platform | Autonomous AI pentest platform & MCP host | - Per-tech offensive sub-agents - Active Directory & Kubernetes - Evidence trail per finding - Orchestrates 80+ offensive tools |
| PentAGI | Security Tool | Automated penetration testing | - Autonomous AI agents - Professional security tools - Comprehensive monitoring |
| Tenuo | Authorization Framework | Capability-based authorization for AI agents | - Cryptographic warrants with task-scoped TTLs - Offline verification - Proof-of-possession binding - LangChain/LangGraph/MCP integrations |
| APort | Runtime Policy & Verification | Runtime policy and verification layer for AI agents and MCP-connected tools | - Guardrails around tool use - Policy enforcement - Auditable runtime verification |
| Agent Threat Rules (ATR) | Detection Standard | Open-source AI agent security detection rules | - 108 detection rules for MCP/agent threats - 62.7% MCP recall, 99.7% precision. 96.9% SKILL.md recall. Shipped in Cisco AI Defense - OWASP Agentic Top 10 full coverage - npm install & one-command scan |
| Viridis MCP | MCP Services | Aristotle-verified attribution-enforcement MCP services for AI agents | - /v1/injection/detect (T-IB-02) - /v1/canon/scan (T-IB-05) - /v1/maxwell/challenge (T-IB-09) - Free tier, 7/7 corpus theorems formally proven in Lean 4 by Aristotle (Harmonic) |
| Agent Memory Guard | Security Library | OWASP ASI06 memory poisoning defense for AI agents | - Memory integrity validation
- Poisoned memory detection
- LangChain/LlamaIndex middleware
- Audit logging & tenant isolation |
Benchmarks & Evaluations
| Project | Focus | Metrics | Key Features |
|---|---|---|---|
| Agent-Attack | Attack Testing | Security vulnerabilities | - Attack vectors - Defense evaluation - Risk assessment |
| Auto-Pen-Bench | Penetration Testing | Security benchmarks | - Automated testing - Performance metrics - Security scoring |
| ASB | Security Benchmark | Agent security | - Security metrics - Performance analysis - Vulnerability testing |
| LLM-Agent-Benchmark | Comprehensive | Agent evaluation | - Security testing - Performance metrics - Benchmark collection |
Security Projects & Implementations
| Project | Type | Purpose | Features |
|---|---|---|---|
| Multi-Agent-SecOps | Security Operations | LLM-based security | - Threat detection - Response automation - Security monitoring |
| Cyber-Security-LLM-Agents | Security Framework | Agent-based security | - Security automation - Threat analysis - Response coordination |
| Ridge Security | Platform | AI security solution | - Vulnerability assessment - Security testing - Risk management |
Technical Presentations & Whitepapers
| Title | Organization | Year | Key Topics |
|---|---|---|---|
| The Double AI Agent | Black Hat EU | 2024 | - Agent manipulation techniques - Double agent scenarios - Defense strategies |
| Mind the Data Gap | Black Hat EU | 2024 | - Privacy in AI agents - Multi-agent systems - Security controls |
| Agentic AI: New Frontier | AI Security Union | 2024 | - Security implications - Future trends - Risk analysis |
| Agent Security Analysis | arXiv | 2023 | - Security frameworks - Vulnerability assessment - Protection measures |
| Agent Behavior Study | arXiv | 2023 | - Behavioral analysis - Attack patterns - Security recommendations |
Community Resources
- OWASP AI Agent Security Project - Official repository
- OWASP Slack #team-llm-autonomus-agents - Community discussions