GenAI Security Study Plan (GenAI/LLM)

August 12, 2026 · View on GitHub

This study plan covers all the topics, concepts, blogs, videos, books, videos, newsletters etc. by keeping GenAI security in mind.

GenAI Security Study Plan

It should take 6-9 months to be good at GenAI security so that you can do one or more of the below listed things:

  1. LLM pentesting
  2. GenAI security assessment
  3. Design and implement secure GenAI/LLM architectures for organizations.
  4. Understanding of GenAI from GRC perspective
  5. Knowledge of different GenAI security frameworks
  6. AI enabled Threat Modeling or Threat Modeling of AI systems
  7. Good grip on LLM safety, LLM Guardrails, Responsible AI, AI ethic etc.

It would help you in your current work as well as finding a new work using GenAI security skills.

Note: I am not writing anything that would require core AI/ML skills. It's all are done after keeping security focus in mind.

Important


This field is still evolving, so our repo would too! Stay tuned!

Organizational Capabilities that you can be job ready after going through mentioned study plan

Security Assessments & Audits

  • Conduct comprehensive GenAI security assessments using the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026
  • Perform LLM application penetration testing and vulnerability assessments
  • Score agentic AI findings with OWASP AIVSS instead of forcing them into plain CVSS
  • Audit RAG (Retrieval Augmented Generation) implementations for security risks
  • Evaluate prompt injection and jailbreaking vulnerabilities
  • Assess model security, including adversarial attacks and data poisoning risks
  • Review AI/ML supply chain security (model provenance, dependencies, third-party APIs)

Governance, Risk & Compliance (GRC)

  • Develop GenAI security policies and procedures aligned with NIST AI RMF and ISO/IEC 42001:2023 (AIMS)
  • Create AI governance frameworks and risk management strategies
  • Implement compliance controls for AI regulations (EU AI Act (Regulation (EU) 2024/1689), India AI Governance Guidelines, etc.) — note the Digital Omnibus (agreed May-Jun 2026) pushed most high-risk (Annex III) obligations to Dec 2027 and Annex I to Aug 2028; only Article 50 transparency duties and GPAI enforcement land Aug 2026
  • Establish AI ethics and responsible AI practices
  • Design AI security awareness training programs for employees
  • Create incident response plans specifically for AI/ML security incidents

Architecture & Implementation

  • Design secure AI/ML pipelines and infrastructure
  • Implement secure GenAI architectures (secure RAG, fine-tuning, inference)
  • Deploy AI security tools (LLM Guard, model scanning, prompt filtering)
  • Establish secure model deployment and MLOps practices
  • Design data privacy controls for AI training and inference data
  • Implement monitoring and logging for AI systems

Risk Management & Threat Modeling

  • Conduct AI/ML specific threat modeling exercises
  • Assess business risks associated with GenAI implementations
  • Develop risk mitigation strategies for AI adoption
  • Create AI security metrics and KPIs for organizational reporting
  • Establish AI risk registers and continuous monitoring processes

Security Engineering & DevSecOps

  • Integrate AI security into CI/CD pipelines
  • Implement security testing for AI/ML models and applications
  • Design secure model training environments and data handling processes
  • Establish model version control and security scanning practices
  • Create automated security testing for prompt injection and other LLM vulnerabilities

Incident Response & Forensics

  • Investigate AI/ML security incidents and breaches, using the OWASP GenAI Incident Response Guide v1.0 as the reference framework
  • Tell an AI incident apart from an ordinary cyber incident — the guide gives diagnostic criteria by AI stack layer and a quick-reference table (OWASP GenAI Incident Response Guide v1.0)
  • Develop playbooks for AI-specific security incidents (attacks on AI systems, on AI supply chains, and on third-party model providers)
  • Build an AI asset inventory and stakeholder map before an incident, and pay particular attention to non-human identities (OWASP GenAI Incident Response Guide v1.0)
  • Perform forensic analysis on compromised AI systems
  • Create incident classification systems for AI/ML security events, including an AI incident severity matrix and a defined blast radius per incident
  • Learn from the guide's worked vignettes: Air Canada chatbot, Microsoft Tay, the "EchoLeak" Microsoft Copilot exploit, MathGPT code execution, and the ChatGPT "Operator" agent data leak

Consulting & Advisory Services

  • Provide GenAI security consulting to organizations
  • Conduct security reviews of vendor AI solutions
  • Advise on secure AI procurement and third-party risk management
  • Lead AI security transformation initiatives
  • Mentor and train internal security teams on AI security

Note


ToC will highlight GenAI based concepts and learning reqources as and when we come across some awesome learning materials.

Study Plan ToC:

  1. GenAI/LLM Fundamental Concepts - 4 weeks
  2. Prompt Engineering - 1 week
  3. RAG (Retrieval Augmented Generation) - 1-2 weeks
  4. Fine Tuning - 2 weeks
  5. AI Agents - 1 week
  6. Agentic AI - 1 week
  7. MCP (Model Context Protocol) - 1 week
  8. GenAI for the Security Engineer - 1 week
  9. Certifications - on your bandwidth and wish
  10. GenAI Interview Questions
  11. GenAI Security Tools

GenAI Fundamental Concepts

Duration: 4 weeks

Week 1: AI/ML Foundations & LLM Basics

Week 2: LLM Security Fundamentals

  • OWASP Top 10 for LLM Applications 2026 (current list — start here)

    • OWASP Top 10 for LLM Applications 2026 (v2026, August 2026)
    • LLM01:2026 Prompt Injection
    • LLM02:2026 Sensitive Information Disclosure
    • LLM03:2026 Excessive Agency
    • LLM04:2026 Supply Chain
    • LLM05:2026 Data and Model Poisoning
    • LLM06:2026 Unbounded Consumption
    • LLM07:2026 Misinformation
    • LLM08:2026 Hidden Context Exposure (re-scoped from 2025's "System Prompt Leakage")
    • LLM09:2026 Vector and Embedding Weaknesses
    • LLM10:2026 Improper Output Handling
    • Note: the 2026 list is the first to validate the practitioner vote against a corpus of real incidents (7,714 collected, 6,639 classified), and its Appendix A maps every entry to MITRE ATLAS, ATT&CK, CWE, NIST AI 600-1 and OWASP AIVSS (OWASP Top 10 for LLM Applications 2026)

  • How the list evolved (useful for interviews and for reading older reports)

    • OWASP Top 10 for LLM Applications 2023 v1.1 (historical)
    • Track what moved: Excessive Agency and Unbounded Consumption escalated in 2026; Supply Chain, Data & Model Poisoning and Improper Output Handling were deprioritized (OWASP Top 10 for LLM Applications 2026)
    • Retired/renamed entries you will still see in the wild: Insecure Plugin Design, Model Denial of Service, Overreliance, Model Theft, System Prompt Leakage

  • Common Attack Vectors

    • Prompt Injection and Jailbreaking
    • Data poisoning attacks
    • Model extraction and theft
    • Adversarial examples
    • Membership inference attacks
    • Hidden context exposure (system prompts, tool definitions, retrieved context leaking to users)

  • GenAI Data Security (data-layer risks, separate from model-layer risks)

Week 3: AI Governance & Compliance

  • Regulatory Frameworks & Standards

  • Implementation Frameworks (turning governance into controls)

  • AI Ethics & Responsible AI

    • Bias and fairness in AI systems
    • Transparency and explainability
    • Privacy and data protection
    • Accountability and human oversight

Week 4: Threat Modeling & Risk Assessment

Hands-on Practice:

Prompt Engineering

Duration: 1 week

Understanding Prompt Engineering

  • Prompt Engineering Fundamentals

    • What is prompt engineering and why it matters for security
    • Types of prompts: zero-shot, few-shot, chain-of-thought
    • Prompt structure and best practices
    • Context window limitations and management

  • Security-Focused Prompt Engineering

    • Defensive prompt engineering techniques
    • Input validation through prompts
    • Output sanitization strategies
    • Prompt injection prevention techniques

Advanced Prompt Techniques

  • Prompt Injection Attacks

    • Direct prompt injection
    • Indirect prompt injection
    • Jailbreaking techniques
    • Prompt leaking attacks

  • Defensive Strategies

    • Prompt templates and parameterization
    • Input filtering and validation
    • Output monitoring and filtering
    • Role-based prompt design

Hands-on Practice:

  • Practice prompt injection techniques on safe platforms
  • Design secure prompt templates
  • Test prompt robustness against various attack vectors

RAG (Retrieval Augmented Generation)

Duration: 1-2 weeks

Week 1: RAG Fundamentals

  • Understanding RAG Architecture

  • RAG Implementation Patterns

    • Simple RAG vs Advanced RAG
    • Multi-step reasoning with RAG
    • Hybrid search approaches
    • RAG with fine-tuned models

Week 2: RAG Security (Optional - for deeper understanding)

Hands-on Practice:

  • Build a simple RAG system with security controls
  • Test for information leakage vulnerabilities
  • Implement access controls for knowledge bases

Fine Tuning

Duration: 2 weeks

Week 1: Fine-Tuning Fundamentals

  • Understanding Fine-Tuning

    • Pre-training vs fine-tuning vs prompt engineering
    • Types of fine-tuning: full, parameter-efficient (LoRA, QLoRA)
    • When to use fine-tuning vs other approaches
    • Data requirements and preparation

  • Fine-Tuning Techniques

    • Supervised fine-tuning (SFT)
    • Reinforcement Learning from Human Feedback (RLHF)
    • Constitutional AI approaches
    • Domain-specific fine-tuning

Week 2: Fine-Tuning Security

  • Security Considerations in Fine-Tuning

    • Training data security and privacy
    • Model poisoning through fine-tuning
    • Backdoor attacks in fine-tuned models
    • Model extraction risks

  • Secure Fine-Tuning Practices

    • Data sanitization and validation
    • Secure training environments
    • Model versioning and provenance
    • Testing fine-tuned models for security

Hands-on Practice:

  • Fine-tune a small model with security considerations
  • Test for data leakage in fine-tuned models
  • Implement secure fine-tuning pipelines

AI Agents

Duration: 1 week

Understanding AI Agents

  • AI Agent Fundamentals

    • What are AI agents and how they differ from simple LLMs
    • Agent architectures: ReAct, Plan-and-Execute, Multi-agent systems
    • Tool use and function calling
    • Memory and state management in agents

  • Types of AI Agents

    • Conversational agents
    • Task-specific agents
    • Autonomous agents
    • Multi-agent systems and collaboration

AI Agent Security

  • Security Risks with AI Agents

    • Excessive agency and unauthorized actions
    • Tool misuse and privilege escalation
    • Agent-to-agent communication security
    • Persistent memory security risks
    • Confused deputy attacks — the classic access-control flaw, back in a new form. An agent holding broad delegated credentials can be tricked by any content it reads (an email, a GitHub issue, a retrieved doc, a tool output) into acting with its operator's full authority. The February 2026 Cline AI coding-assistant compromise is the worked example: a crafted GitHub issue title triggered an authenticated coding session to install an attacker-controlled package, which then shipped as an official update to roughly 4,000 developer machines (Confused Deputy Attacks on Autonomous AI Agents, CSA AI Safety Initiative, 2026 — note: published as unofficial AI-assisted research, so treat the incident detail as secondary sourcing)
    • Skill-layer risks: agent "skills" execute with the host agent's full privileges and are distributed through registries with weak provenance — see AST01 Malicious Skills and AST02 Supply Chain Compromise, both rated Critical (OWASP Agentic Skills Top 10 v0.5). Covered in depth in the Agentic AI section below

  • Securing AI Agents

    • Principle of least privilege for agents
    • Action validation and approval workflows
    • Monitoring agent behavior and decisions
    • Secure tool integration patterns
    • Read the mitigation playbooks in OWASP Agentic AI — Threats and Mitigations v1.0, organised across agent design, memory, planning & autonomy, tool use, and deployment & operations

  • Agent Identity & Delegated Authorization (emerging, live gap area)

    • Why OAuth 2.0 breaks down here: it assumes a deterministic client, but an agent's action plan is generated at runtime, so a bearer token can no longer be read as evidence of user intent
    • Agentic JWT (A-JWT): A Secure Delegation Protocol for Autonomous AI Agents (arXiv:2509.13597, Sep 2025) — proposes binding each agent action to a cryptographically verifiable user intent and a chained delegation assertion, with per-agent proof-of-possession keys to stop replay and in-process impersonation
    • Related reading: non-human identity management, workload identity, and short-lived scoped credentials for agents

Hands-on Practice:

  • Build a simple AI agent with security controls
  • Test agent behavior under various scenarios
  • Implement monitoring for agent actions
  • Reproduce a confused-deputy scenario in a sandbox: give a test agent a broad token, feed it attacker-controlled content through a tool output, and observe what it does

Agentic AI

Duration: 1 week

Advanced Agentic Systems

  • Agentic AI Concepts

    • Autonomous decision-making systems
    • Goal-oriented AI behavior
    • Planning and reasoning in agentic systems
    • Human-AI collaboration patterns
    • Landscape read before you go deep: State of Agentic AI Security and Governance v2.01 (June 2026) — agent taxonomy by operational role, implementation and composition patterns, autonomy level as a cross-cutting dimension, and where AI safety and AI security actually differ

  • Agentic AI Architectures

    • Multi-agent orchestration
    • Hierarchical agent systems
    • Distributed agentic networks
    • Agent communication protocols
    • Securing Agentic Applications Guide v1.0 (July 2025) — attack-surface analysis plus per-architecture hardening actions for single-agent, central-orchestrator, and swarm topologies, and for the key operational capabilities (API access, code execution, web use)

Security in Agentic AI

  • OWASP Top 10 for Agentic Applications 2026 (ASI)

    • OWASP Top 10 for Agentic Applications 2026 (December 2025)
    • ASI01: Agent Goal Hijack
    • ASI02: Tool Misuse and Exploitation
    • ASI03: Identity and Privilege Abuse
    • ASI04: Agentic Supply Chain Vulnerabilities
    • ASI05: Unexpected Code Execution (RCE)
    • ASI06: Memory & Context Poisoning
    • ASI07: Insecure Inter-Agent Communication
    • ASI08: Cascading Failures
    • ASI09: Human-Agent Trust Exploitation
    • ASI10: Rogue Agents
    • Also read Appendix D (the ASI agentic exploits & incidents tracker) and Appendix C, which maps the list to the OWASP Non-Human Identities Top 10 (OWASP Top 10 for Agentic Applications 2026)

  • Agent Skills Security (the layer most teams are still missing)

    • OWASP Agentic Skills Top 10 v0.5 (June 2026, OWASP Incubator project)

    • Why skills are their own attack surface: skills are the behaviour layer — reusable units that tell an agent what to do and grant it the tools to do it. The industry hardened the model layer and the MCP tool layer; the skill layer in between grew fast and under-protected. Skills execute with the host agent's full privileges, blend natural-language instructions with executable code, and ship through registries that mostly lack the provenance controls of mature package ecosystems (OWASP Agentic Skills Top 10 v0.5)

    • The ten risks — two Critical, four High, four Medium:

      IDRiskSeverityKey mitigation
      AST01Malicious SkillsCriticalSigning, registry scanning
      AST02Supply Chain CompromiseCriticalProvenance, transparency logs
      AST03Over-Privileged SkillsHighLeast-privilege manifests
      AST04Insecure MetadataHighStatic analysis, manifest linting
      AST05Unsafe DeserializationHighSafe parsers, sandboxed loading
      AST06Weak IsolationHighContainerization, sandboxing
      AST07Update DriftMediumImmutable pinning, hash verification
      AST08Poor ScanningMediumBehavioral + semantic scanning
      AST09No GovernanceMediumInventory, audit, identity controls
      AST10Cross-Platform ReuseMediumUniversal format, re-validation

    • The dual attack surface — this is the part that breaks traditional scanning. A malicious skill attacks through both the code layer (shell scripts, Python calls) and the natural-language instruction layer (markdown prose in SKILL.md that simply tells the agent to do something). Snyk's ToxicSkills research found 100% of malicious skills combined both vectors — so a package scanner that only reads code will miss half the payload (OWASP Agentic Skills Top 10 v0.5)

    • Attack patterns worth knowing by name: typosquatting (google-workspace vs gogle-workspace); social-engineering "Prerequisites" sections that instruct the user to paste attacker-supplied install commands; ClickFix-style fake "setup required" prompts; SOUL.md persistence, where a backdoor is written into the agent's identity file and survives skill uninstall; memory poisoning via MEMORY.md so the agent executes attacker commands in future sessions; and persistent WebSocket connections to attacker C2 for live command execution

    • Real-world evidence (cite these — skill security is often dismissed as theoretical):

      • ClawHavoc, January 2026: 1,184 malicious skills across 12 publisher accounts sharing a single C2 IP, delivering Atomic Stealer (AMOS) against macOS crypto wallets, SSH keys and browser credentials. At peak infection, five of the seven most-downloaded ClawHub skills were confirmed malware
      • Snyk, February 2026: three lines of markdown in a SKILL.md were enough to exfiltrate SSH keys
      • USENIX Security 2026 measurement study (Liu et al., arXiv:2602.06547): 98,380 skills analysed across public marketplaces; 157 confirmed malicious carrying 632 vulnerabilities (avg 4.03 each); 73.2% implemented shadow features hidden from the user, and 54.1% traced back to a single publisher cluster

    • Controls to design for, in rough order of leverage: require ed25519 signatures and reject unsigned installs; Merkle-root signing for the registry itself; scan at both publish time and install time using behavioural analysis rather than pattern matching; execute skills in containers/sandboxes with no network by default; hash-pin installed skills and alert on any modification; never auto-execute a "Prerequisites" section without explicit human review; surface publisher trust level, install count and scan status in the install UI; structured audit logging of skill actions (OWASP Agentic Skills Top 10 v0.5)

    • Framework mapping: AST01 maps to LLM Supply Chain and LLM01 Prompt Injection (indirect), plus ASVS V14 Configuration. In CSA MAESTRO terms it lands primarily on Layer 7 (Agent Ecosystem — registry compromise, marketplace manipulation), with Layer 3 (Agent Frameworks), Layer 6 (Security & Compliance), Layer 4 (Deployment & Infrastructure) and Layer 5 (Evaluation & Observability) also in scope

    • Two caveats when you cite this: the document is v0.5, an OWASP Incubator project — directionally strong, not yet a settled standard; and its OWASP mapping table uses the 2025 LLM numbering (Supply Chain as LLM03), which is LLM04:2026 in the current list — translate before you put it in a report

  • Unique Security Challenges

    • Emergent behaviors in agentic systems
    • Goal misalignment and specification gaming
    • Inter-agent security and trust
    • Scalability of security controls
    • Multi-agent communication attacks — see Red Teaming LLM Multi-agent Systems via Communication Attacks (arXiv:2502.14847) for the attack model

  • Zero Trust Architecture for Agents

    • Zero Trust for AI Agents (Anthropic/Claude) — a tiered capability framework, current agentic threat vectors, an agent implementation workflow, and defensive operations at machine speed; written for CISOs (Parts I-II) and architects/engineers (Parts III-V). Core premise: traditional access controls will not stop an agent misusing legitimate permissions, so architect for breach from day one (Claude — Zero Trust for AI Agents)
    • Design Principles for LLM-based Systems with Zero Trust: Foundation for Secure Agentic Systems — a joint publication of Germany's BSI and France's ANSSI; useful as a government-grade, vendor-neutral counterpart to the above
    • Practical translation: no standing broad credentials, per-action authorization, blast-radius limits proportional to autonomy, and human-in-the-loop gates on irreversible actions

  • Governance for Agentic AI

    • Establishing boundaries and constraints
    • Monitoring and auditing agentic behavior
    • Human oversight and intervention mechanisms
    • Ethical considerations in autonomous systems
    • Score agentic risks consistently with OWASP AIVSS v0.8 rather than raw CVSS

Hands-on Practice:

  • Design security controls for agentic systems
  • Analyze case studies of agentic AI failures
  • Develop monitoring strategies for autonomous agents
  • Take one agentic feature you own and walk it end-to-end against ASI01-ASI10, then score the top three findings with AIVSS
  • Inventory every agent skill installed across your team's agent platforms — most orgs cannot answer this, which is AST09 (No Governance) in practice
  • Write a deliberately malicious test skill in a sandbox that attacks through the prose layer only (no malicious code), then run it past whatever scanner you currently trust and see if it is caught
  • Draft a skill-approval policy: signature required, least-privilege manifest, hash-pinned version, no auto-executed "Prerequisites", sandboxed execution

MCP (Model Context Protocol)

Duration: 1 week

Understanding MCP

  • MCP Fundamentals

    • What is Model Context Protocol
    • MCP architecture and components
    • Client-server communication patterns
    • Resource management and sharing

  • MCP Implementation

    • Setting up MCP servers and clients
    • Resource discovery and access
    • Tool integration through MCP
    • Context sharing between applications

MCP Security

  • The MCP Threat Landscape

    • Tool poisoning and rug-pull attacks, prompt injection via tool output, memory poisoning, tool interference (A Practical Guide for Securely Using Third-Party MCP Servers v1.0, October 2025)
    • Why MCP amplifies impact: unlike a traditional API, an MCP server typically runs with delegated user permissions, exposes dynamic tool-based surfaces, and can chain multiple tool calls — so a single flaw compounds (A Practical Guide for Secure MCP Server Development v1.0)

  • Building MCP Servers Securely

    • A Practical Guide for Secure MCP Server Development v1.0 (February 2026) — covers secure MCP architecture, safe tool design, data validation & resource management, prompt-injection controls, authn/authz, secure deployment & updates, governance, and continuous validation
    • Use its MCP Security Minimum Bar (Review Checklist) as your go/no-go gate before any MCP server ships

  • Consuming Third-Party MCP Servers

  • Best Practices

    • Secure MCP server deployment
    • Client-side security measures
    • Monitoring MCP interactions
    • Incident response for MCP systems — build MCP-specific playbooks for suspected tool poisoning and data leakage, with explicit containment, eradication and recovery steps (OWASP GenAI Incident Response Guide v1.0)
    • Agent identity carries into MCP too: scope tokens per tool, avoid one shared credential across agents (see the Agent Identity & Delegated Authorization notes under AI Agents)

Hands-on Practice:

  • Set up a secure MCP environment
  • Implement access controls for MCP resources
  • Test MCP security configurations
  • Review a real third-party MCP server against the MCP Security Minimum Bar checklist and write up what you would block

GenAI for the Security Engineer

Duration: 1 week

This section flips the lens: instead of securing GenAI systems, it covers using GenAI fundamentals to be a more effective security engineer day to day, and where GenAI is reshaping your own job.

High-Leverage Fundamentals

  • The concepts that actually pay off in practice (not the deepest ML theory, the ones that change how you reason about risk)
    • Context window as trust boundary — everything in it is potentially instructive; this single idea collapses prompt injection, memory poisoning, RAG poisoning, and confused-deputy into one mental model instead of four
    • Tokenization, embeddings, and vector similarity — needed to reason about LLM09:2026 Vector and Embedding Weaknesses and encoding-based filter bypasses
    • Inference-time vs. training-time risk — decides whether a risk is yours or your model vendor's; most enterprise GenAI risk is inference-time and architectural (NIST AI 100-2e2025)
    • Tool/function calling mechanics — the line where a model stops generating text and starts acting; this is where excessive agency becomes real
    • Agent memory types (short-term, episodic, vector, identity/instruction files) — memory is the persistence mechanism attackers target; see the Agentic AI section's coverage of skill/memory poisoning
    • Non-deterministic evaluation — an LLM control can't be pass/failed with one test run; write acceptance criteria as evals, not test cases (OWASP GenAI Red Teaming Guide v1.0)
    • The differentiator: being able to name the mechanism behind a risk ("this retriever pulls untrusted third-party content into a context window that also holds a tool-calling system prompt with write scope") instead of citing a Top 10 label

AI-Assisted Code: A New Supply-Chain Input

  • What AI-generated code actually looks like at scale
    • Sonar's analysis of 7.9B lines of code across 970,000 developers and 40,000 organizations found ~1,200 security issues per million lines of code, ~170 confirmed vulnerabilities per MLOC, and roughly 1-2 security issues introduced per developer per month; log injection was the most common vulnerability and hardcoded credentials the most common hotspot, with 50% of discovered secrets being database passwords (Sonar, The State of Code, Vol. 2: Security, July 2025)
    • Sonar's framing is the useful part: "AI coding tools are excellent mimics" — they reproduce the patterns already in your codebase, so your existing code quality becomes a supply-chain input to everything AI generates from it
    • Practical implication: fixing your repo's most common recurring vulnerability pattern also reduces the AI-generated version of it

AI TRiSM and the Organizational Gap

  • Where the real gap is (organizational, not technical)
    • Gartner: nearly 90% of enterprises are still researching or piloting GenAI, and most have not yet put AI TRiSM technical controls or policies in place (Gartner, AI in Cybersecurity: Define Your Direction)
    • AI TRiSM control taxonomy: content anomaly detection, data protection, application security, explainability/transparency, model management (ModelOps), adversarial resistance — split between what the AI builder/owner must implement and what the AI user must buy to fill gaps
    • Gartner lists "unmanaged, uncontrolled use of confidential data in third-party applications" (shadow AI) as a direct and urgent risk, alongside erroneous decision-making from over-trusted AI outputs
    • Honest caveat worth carrying into exec conversations: Gartner's own read is that AI in cybersecurity "hasn't yet fulfilled its promises" — full automation and quantifiable outcomes sit on the hype side, while cost/resources, human augmentation, and new threats sit on the reality side

First Moves: Where to Have Impact First

  • Four moves, roughly in priority order
    1. AI asset inventory + shadow AI discovery — nobody else in most orgs is doing this yet, and it is immediately useful (Gartner; OWASP GenAI Data Security 2026 DSGAI03 Shadow AI & Unsanctioned Data Flows)
    2. A GenAI feature intake checklist, so security is consulted before build instead of at launch
    3. An agent/skill inventory — most orgs cannot currently answer "what agent skills are installed" (OWASP Agentic Skills Top 10 AST09)
    4. AI incident classification bolted onto existing IR, using the OWASP GenAI Incident Response Guide v1.0's AI-vs-cyber diagnostic criteria

Hands-on Practice:

  • Run a shadow-AI discovery pass against one business unit and compare it to what security already knew about
  • Draft a one-page GenAI feature intake checklist for your org
  • Pick one recurring vulnerability pattern in a repo you own and check whether AI-assisted commits are reproducing it
  • Map your team's current AI security controls against the six AI TRiSM categories and mark the gaps

Certifications

Duration: Based on your bandwidth and goals

AI/ML Security Certifications

  • Certified AI/ML Pentester

  • Cloud AI Security Certifications

    • AWS Machine Learning Specialty
    • Google Cloud Professional ML Engineer
    • Azure AI Engineer Associate
    • Focus on security aspects of cloud AI services

Vendor-Specific Certifications

  • OpenAI Safety and Alignment
  • Anthropic Constitutional AI
  • Microsoft Responsible AI
  • Google AI Ethics

Preparation Resources:

GenAI Interview Questions

Technical Questions

  • LLM Fundamentals

    • Explain the transformer architecture and its security implications
    • What are the key differences between GPT, BERT, and T5 models?
    • How do attention mechanisms work and what security risks do they pose?
    • Describe the training process of large language models

  • Security-Specific Questions

    • Walk through the OWASP Top 10 for LLM Applications 2026 and provide examples — and explain what changed from 2025 (e.g. Excessive Agency rising to LLM03, System Prompt Leakage re-scoped into Hidden Context Exposure)
    • Walk through the OWASP Top 10 for Agentic Applications 2026 (ASI01-ASI10) and give a concrete attack scenario for three of them
    • How would you test an LLM application for prompt injection vulnerabilities?
    • Explain the difference between direct and indirect prompt injection
    • What is a confused deputy attack on an AI agent, and why does OAuth 2.0 not solve it?
    • What are the main security considerations when implementing RAG?
    • How would you secure a fine-tuning pipeline?
    • How would you threat model an MCP server, and what is your minimum bar before it ships?
    • Why is CVSS alone a poor fit for agentic AI risk, and what does AIVSS add?

Scenario-Based Questions

  • Risk Assessment Scenarios

    • "A company wants to implement a customer service chatbot using GPT-4. What security risks would you identify?"
    • "How would you conduct a security assessment of an existing LLM application?"
    • "Design a secure architecture for a RAG-based document Q&A system"

  • Incident Response Scenarios

    • "An LLM application is leaking sensitive customer data. How would you investigate?"
    • "Users report that the chatbot is providing inappropriate responses. What's your approach?"
    • "A competitor seems to have extracted your fine-tuned model. How do you respond?"

Governance and Compliance

  • Regulatory Questions
    • How does the EU AI Act impact LLM deployments?
    • What are the key components of NIST AI RMF?
    • What is ISO/IEC 42001:2023 and when would you pursue certification against it?
    • How do the India AI Governance Guidelines differ in approach from the EU AI Act?
    • How would you implement AI governance in an organization?
    • What metrics would you use to measure AI security posture?

Hands-on Technical Challenges

  • Practical Exercises
    • Demonstrate prompt injection techniques
    • Show how to implement LLM Guard or similar tools
    • Explain model scanning and vulnerability detection
    • Design monitoring and alerting for LLM applications

GenAI Security Tools

Open Source Security Tools

  • LLM Guard by ProtectAI

    • GitHub Repository
    • Playground
    • Input/output filtering and sanitization
    • Prompt injection detection
    • Sensitive data detection and redaction

  • Model Scanning Tools

    • ModelScan by ProtectAI
    • Scans AI/ML models for security vulnerabilities
    • Detects malicious code in model files
    • Supports multiple model formats

  • AI/ML Exploit Tools

    • AI Exploits by ProtectAI
    • Collection of AI/ML security exploits
    • Educational and testing purposes
    • Demonstrates common attack vectors

Commercial Security Platforms

  • Lakera Guard

    • Real-time LLM security monitoring
    • Prompt injection detection
    • Content filtering and moderation
    • API-based integration

  • Robust Intelligence

    • AI security and monitoring platform
    • Model validation and testing
    • Continuous monitoring for drift and attacks
    • Enterprise-grade security controls

  • WhyLabs

    • ML monitoring and observability
    • Data drift detection
    • Model performance monitoring
    • Security-focused analytics

Testing and Assessment Tools

  • Garak

    • LLM vulnerability scanner
    • Automated testing for various attack types
    • Extensible framework for custom tests
    • Community-driven development

  • PromptFoo

    • LLM evaluation and testing framework
    • Security-focused test cases
    • Automated red teaming capabilities
    • Integration with CI/CD pipelines

Risk Scoring & Framework Tooling

  • OWASP AIVSS (AI Vulnerability Scoring System)

    • AIVSS project and calculator (v0.8)
    • CVSS v4.0 base score plus an Agentic AI Risk Score (AARS) from 10 agentic amplification factors
    • JSON report schema for machine-readable findings
    • Maps to the OWASP Agentic AI Top 10 and CSA MAESTRO layers

  • MCP Security Scanners

    • Automated scanners for MCP tool definitions and server manifests — see the tools listed in the OWASP third-party MCP server cheat sheet
    • Use alongside the MCP Security Minimum Bar review checklist

Bug Bounty and Research Platforms

Monitoring and Observability

  • LangSmith

    • LLM application monitoring
    • Trace analysis and debugging
    • Performance and security metrics
    • Integration with LangChain

  • Weights & Biases

    • ML experiment tracking
    • Model monitoring and versioning
    • Security-focused metrics and alerts
    • Team collaboration features

Cloud-Native Security Tools

  • AWS Bedrock Guardrails

    • Content filtering and safety controls
    • Custom guardrail policies
    • Real-time monitoring and blocking
    • Integration with AWS services

  • Azure AI Content Safety

    • Content moderation and filtering
    • Custom classification models
    • API-based integration
    • Multi-language support

  • Google Cloud AI Platform Security

    • Model security scanning
    • Access controls and IAM
    • Audit logging and monitoring
    • Compliance reporting

Implementation Checklist

  • Evaluate tools based on your specific use case
  • Set up monitoring and alerting for LLM applications
  • Implement input/output filtering and validation
  • Deploy model scanning in CI/CD pipelines
  • Establish incident response procedures
  • Regular security assessments and penetration testing
  • Stay updated with latest tools and techniques

Additional Resources

  1. Courses & University Materials

  2. Security Guides & Checklists

  3. Articles & Blogs

  4. Tools & Platforms

  5. Challenges & CTFs

  6. Videos

  7. Agentic AI, AI Agents & MCP Security

  8. Governance, Standards & Regional Regulation