Awesome GenAI Security

June 27, 2026 · View on GitHub

A curated list of links, references, books, videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices, etc., which are related to GenAI, LLM, RAG, MCP, Agents, and Agentic AI security.

Table of Contents

GenAI security banner

GenAI Security Papers & Standards

Important papers, standards, and checklists from organizations like OWASP, NIST, and others.

  1. OWASP Top 10 for LLM Applications 2025
  2. OWASP LLM AI Security and Governance Checklist
  3. OWASP Agentic AI Top 10
  4. NIST AI RMF Playbook
  5. NIST AI Risk Management Framework (AI RMF)
  6. NIST Adversarial Machine Learning
  7. Microsoft Failure Models in Machine Learning
  8. Microsoft Threat Modeling AI/ML
  9. OWASP GenAI Security Project
  10. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
  11. Google Secure AI Framework (SAIF)
  12. Anthropic Responsible Scaling Policy
  13. ENISA Multilayer Framework for Good Cybersecurity Practices for AI
  14. Databricks AI Security Framework (DASF) 2.0 - Practical controls mapped to AI system components and risks.
  15. OWASP Securing Agentic Applications Guide 1.0 - Reference architecture and controls for building secure agentic apps.
  16. CSA MAESTRO - Agentic AI Threat Modeling Framework - Seven-layer threat modeling for agentic AI systems.
  17. Microsoft: Zero Trust for AI (Tools & Guidance) - Applying Zero Trust principles to AI agents and workloads.

AI Security Books

  1. AI Value Creators
  2. AI Engineering by Chip Huyen
  3. Designing Machine Learning Systems
  4. Hands-On Large Language Models
  5. Nexus by Yuval Noah Harari
  6. The Developer's Playbook for Large Language Model Security: Building Secure AI Applications by Steve Wilson
  7. 10 Best AI Security Books (Practical DevSecOps)
  8. AI Security E-book 101 (Practical DevSecOps, PDF)
  9. Red Teaming AI: A Field Manual for Attacking Intelligent Systems - Philip A. Dursey (No Starch, early access)
  10. Not with a Bug, But with a Sticker - Ram Shankar Siva Kumar & Hyrum Anderson
  11. Generative AI Security - Ken Huang, Yang Wang (Springer)
  12. Adversarial AI Attacks, Mitigations, and Defense Strategies - John Sotiropoulos (Packt)

AI Security Videos

  1. Intro to LLM Security - WhyLabs
  2. OWASP Top 10 for LLM Applications Explained - OWASP
  3. Hacking LLMs and Prompt Injection - LiveOverflow
  4. AI Red Teaming - DEFCON AI Village
  5. Securing LLM Applications - SANS Institute

Online Tutorials / Blogs / Presentations

Articles and guides covering LLM, RAG, and general GenAI security.

  1. LLM Security
  2. What are foundational models?
  3. A quick check on the AI Threat Model
  4. Security Incident Response using LLM
  5. OWASP: CheatSheet – A Practical Guide for Securely Using Third-Party MCP Servers 1.0
  6. AI Security Interview Questions (Practical DevSecOps)
  7. Emerging AI Security Roles (Practical DevSecOps)
  8. AI Security Engineer Roadmap (Practical DevSecOps)
  9. Prompt Injection Attacks and Defenses in LLM-Integrated Applications
  10. Simon Willison's Blog on Prompt Injection
  11. Embrace the Red - AI Security Blog by Johann Rehberger
  12. Trail of Bits - AI/ML Security Research

RAG Security

  1. Riding the RAG Trail: Access, Permissions and Context
  2. Securing Risks with RAG Architectures
  3. Mitigating Security Risks in Retrieval Augmented Generation (RAG)
  4. RAG: The Essential Guide
  5. Why RAG is revolutionising GenAI

MCP & Agent Security

  1. Invariant Labs: MCP Security Notification Tool
  2. Pillar Security: MCP Security Research
  3. Agentic Security Risks - OWASP
  4. Tool Poisoning Attacks in MCP

LLM Attacks

  1. Web LLM attacks - PortSwigger
  2. Prompt injection jailbreaking
  3. LLM Attacks - Universal and Transferable Adversarial Attacks on Aligned LLMs
  4. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications

Online Courses (Paid/Free)

  1. Stanford CS-324: Large Language Models
  2. Princeton COS 597G: Understanding Large Language Models
  3. Coursera: GenAI with LLM
  4. Coursera: Generative AI Engineering with LLMs Specialization
  5. Coursera: Generative AI for Cybersecurity Professionals (IBM)
  6. Coursera: AI for Cybersecurity (JHU)
  7. AttackIQ: The foundation of AI Security
  8. Microsoft AI Red Teaming 101 (Microsoft Learn) - Free training on GenAI vulnerabilities, single/multi-turn attacks, spotlighting defenses, and PyRIT automation.
  9. SANS SEC545: GenAI and LLM Application Security - RAG/vector-DB security, prompt injection, MLOps hardening, and agentic AI security (maps to GIAC GAIPS).

AI Security Certifications

  1. Certified AI Security Professional (CAISP) by Practical DevSecOps – Securing AI systems, models, and pipelines against adversarial threats, LLM vulnerabilities, AI supply chain risks, data poisoning, and AI-specific security frameworks. Hands-on, practitioner-level skills.
  2. GIAC AI Platform Security (GAIPS) - Hands-on (CyberLive) certification for auditing and securing GenAI applications, LLM pipelines, and agentic AI systems.
  3. GIAC AI Security Automation Engineer (GASAE) - Validates using AI/automation across offensive, defensive, and cloud security operations.
  4. ISC2 AI Security Certificate - On-demand course bundle covering AI fundamentals, AI risk management, secure-by-design, and global AI regulation.

Tools of Trade

Tools for defending, scanning, and auditing GenAI systems.

Defensive / Scanning

  1. LLM Guard - Information extraction and security for LLMs.
  2. Model Scan - Scanning models for serialization attacks.
  3. Rebuff - Prompt injection detection.
  4. NB Defense - Notebook security.
  5. Protect AI's OSS Portfolio
  6. LLM Guard Playground
  7. Cisco MCP Scanner - Scans MCP servers/tools for poisoning and prompt injection (YARA + LLM-as-judge).
  8. Snyk Agent Scan - Inventories and scans AI agents, MCP servers, and skills for 15+ risks (successor to Invariant mcp-scan).
  9. Fickling (Trail of Bits) - Decompiler, static analyzer, and safety scanner for malicious pickle/PyTorch model files.
  10. ModelAudit - Static scanner detecting malicious code/backdoors across 40+ ML model file formats.
  11. AIsbom - CLI that scans model files for malware and generates CycloneDX/SPDX AI SBOMs.

Offensive / Red Teaming

  1. AI/ML Exploits
  2. Garak - LLM Vulnerability Scanner
  3. PyRIT - Python Risk Identification Toolkit for GenAI (Microsoft)
  4. Counterfit - AI Security Testing (Microsoft)
  5. ART - Adversarial Robustness Toolbox (IBM)
  6. promptmap - Prompt Injection Testing
  7. DeepTeam - LLM & AI-Agent Red Teaming Framework - 50+ vulnerability types and 20+ attack methods mapped to OWASP/NIST/MITRE.
  8. Promptfoo - LLM Testing & Red Teaming - Generates adversarial inputs to find prompt injection, jailbreaks, and data leakage, with CI/CD integration.

Guardrails & Firewalls

  1. Guardrails AI - Input/output validation for LLMs.
  2. NeMo Guardrails (NVIDIA) - Programmable guardrails for LLM applications.
  3. Vigil - LLM Prompt Injection Detection
  4. Lakera Guard - Real-time AI security for prompt injection and data leakage.
  5. Trylon Gateway - Self-hosted open-source AI firewall/proxy applying custom guardrails (prompt-injection defense, PII redaction).
  6. Bifrost AI Gateway - High-performance open-source AI gateway unifying 20+ LLM providers with governance and policy enforcement.

Security Practices and CTFs

Practice your skills with these vulnerable applications and challenges.

  1. Gandalf - Lakera AI - LLM security challenge.
  2. Prompt Airlines - AI security challenges, CTF style.
  3. Certified AI/ML Pentester (C-AI/MLPen) Exam - The SecOps Group
  4. Damn Vulnerable MCP Server - Deliberately vulnerable MCP implementation.
  5. Vulnerable MCP Servers Lab - Collection of vulnerable servers.
  6. FinBot Agentic AI CTF - Agentic Security CTF.
  7. OWASP WrongSecrets - Includes an LLM/AI secrets-leakage challenge.
  8. Huntr.com - World’s first bug bounty platform for AI/ML.
  9. HackAPrompt - Prompt hacking competition.
  10. Crucible by Dreadnode - AI/ML security challenges and CTFs.
  11. AI Goat - Vulnerable LLM CTF built on AWS.
  12. Microsoft AI Red Teaming Playground Labs - Hands-on red-teaming challenges (prompt injection, indirect injection, guardrail bypass) with Docker/Kubernetes deployment.
  13. PortSwigger Web Security Academy: Web LLM Attacks - Free official hands-on labs on exploiting LLM APIs, excessive agency, and prompt injection.

AI Red Teaming

Resources and methodologies for red teaming AI/GenAI systems.

  1. Microsoft AI Red Team
  2. Anthropic Red Teaming Research
  3. MITRE ATLAS - Tools, Data & Case Studies (GitHub)
  4. AI Red Teaming Guide - Humane Intelligence
  5. Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities
  6. OWASP GenAI Red Teaming Guide - Methodology for red teaming GenAI/LLM applications.
  7. CSA Agentic AI Red Teaming Guide - Red teaming approach tailored to autonomous agents.

GenAI Security Attacks, Breaches & Incidents

Notable real-world incidents involving GenAI and LLM security.

  1. Anthropic Disrupts First AI-Orchestrated Cyber Espionage Campaign (Nov 2025) - A Chinese state-sponsored group jailbroke Claude Code to autonomously execute ~80-90% of an espionage campaign against ~30 global targets; the first documented largely AI-run cyberattack.
  2. CamoLeak: GitHub Copilot Chat Private Source-Code Exfiltration (Oct 2025) - CVSS 9.6 prompt injection hidden in PRs leaks private code and secrets via GitHub's own Camo image proxy.
  3. Check Point Researchers Expose Critical Claude Code Flaws - CVE-2025-59536 and CVE-2026-21852: Enabling Remote Command Execution and API Key Theft.
  4. Anthropic: "Vibe-Hacking" Extortion Using Claude Code (Aug 2025) - A criminal used Claude Code to automate intrusions and craft targeted extortion against 17+ organizations, alongside AI-built ransomware-as-a-service.
  5. PromptLock: First Known AI-Powered Ransomware (Aug 2025, PoC) - ESET found ransomware that uses a local LLM via the Ollama API to generate malicious scripts on the fly (later assessed as an academic proof-of-concept).
  6. Replit AI Agent Deletes a Production Database (Jul 2025) - Replit's AI coding agent wiped a live production database during a code freeze, then fabricated data and falsely claimed rollback was impossible.
  7. Amazon Q VS Code Extension Compromised with Data-Wiping Prompt (Jul 2025) - A malicious prompt to wipe files and delete AWS resources was slipped into the official Amazon Q extension (964k+ installs).
  8. CVE-2025-6514: Critical RCE in mcp-remote (Jul 2025) - A malicious MCP server could achieve full remote code execution (CVSS 9.6) on clients running mcp-remote; the first real-world RCE against an MCP client.
  9. EchoLeak (CVE-2025-32711): Zero-Click Data Theft in Microsoft 365 Copilot (Jun 2025) - A single crafted email could silently exfiltrate organizational data from M365 Copilot with no user interaction; the first known zero-click exploit against an AI agent.
  10. Policy Puppetry: Universal Jailbreak Bypassing All Major LLMs (Apr 2025) - HiddenLayer disclosed a single transferable prompt that bypasses safety guardrails across OpenAI, Google, Anthropic, Meta, DeepSeek, and others.
  11. Singapore Finance Director Scammed via Deepfake Executive Video Call (Mar 2025) - A finance director transferred ~US$499,000 after a Zoom call in which company executives were AI-generated deepfakes.
  12. nullifAI: Malicious ML Models on Hugging Face Evade Picklescan (Feb 2025) - ReversingLabs found malicious Pickle-based models using broken/7z-wrapped pickles to bypass scanners and deliver a reverse shell (PoC).
  13. Anthropic: Chinese AI Firms Created 24,000 Fraudulent Accounts For 'Distillation Attacks'
  14. DeepSeek Exposed Database Leaking Chat History and API Keys (Jan 2025) - Wiz found a publicly accessible, unauthenticated DeepSeek ClickHouse database exposing 1M+ log lines including plaintext chats and secrets.
  15. LiteLLM on PyPI Was Compromised, What the Attack Changed and What Defenders Should Do Now
  16. The Day Chevrolet's AI Chatbot Tried to Sell a $70,000 SUV for $1
  17. Air Canada Chatbot Provides Wrong Info (2024) - Airline held liable for chatbot hallucinating refund policy.
  18. Samsung bans use of generative AI tools like ChatGPT after April internal data leak (2023)
  19. AI-powered Bing Chat spills its secrets via prompt injection attack (2023)
  20. ChatGPT Data Leak Bug (2023) - Bug exposed chat history titles and payment info of other users.
  21. GitHub Copilot Leaking Secrets (2023) - AI code assistant reproducing secrets from training data.
  22. Microsoft Tay Bot Manipulation (2016) - Twitter chatbot manipulated into generating offensive content.

Regulatory Frameworks & Governance

  1. EU AI Act - EU regulation on artificial intelligence.
  2. NIST AI Risk Management Framework (AI RMF) - NIST's voluntary framework for managing AI risks.
  3. ISO/IEC 42001:2023 - AI Management System Standard
  4. U.S. AI Executive Order (2025): Removing Barriers to American Leadership in AI - The Trump administration's Jan 2025 EO; it revoked Biden's EO 14110 (2023) on Safe, Secure & Trustworthy AI.
  5. India AI Governance Guidelines (IndiaAI / MeitY) - India's national AI strategy and governance guidance.
  6. Singapore Model AI Governance Framework

Newsletters & Communities

  1. OWASP GenAI Slack Channel - Join #project-top10-for-llm channel.
  2. AI Village (DEF CON) - Community focused on AI security research.
  3. MLSecOps Community - Community for ML security operations.
  4. The AI Security Newsletter by Ken Huang
  5. Protect AI Blog (now part of Palo Alto Networks)

Contributors