C07: Model Behavior, Output Control & Safety Assurance

September 8, 2026 · View on GitHub

Source: 1.01-dev/en/0x10-C07-Model-Behavior.md Requirements: 14 | Sections: 4

Control Objective

This control category ensures that model outputs are technically constrained, validated, and monitored so that unsafe, malformed, or high-risk responses cannot reach users or downstream systems. Schema output validation applies to applications that expect structured output such as JSON, XML, or typed function-call responses; free-form text outputs still need safety, privacy, grounding, and downstream handling controls, but are not themselves in scope for strict schema validation.

Section Pages

Section	Title	Reqs	Page
C7.1	Output Format Enforcement	3	C07-01-Output-Format-Enforcement
C7.2	Hallucination Detection & Mitigation	3	C07-02-Hallucination-Detection
C7.3	Output Safety	4	C07-03-Output-Safety-Privacy-Explainability
C7.4	Source Attribution & Citation Integrity	4	C07-04-Source-Attribution-Citation-Integrity

Threat Landscape

Known attacks, real-world incidents, and threat vectors relevant to this chapter:

Hallucination at scale — Stanford's 2026 AI Index logged AA-Omniscience hallucination rates from 22% (Grok 4.20 Beta 0305) to 94% (gpt-oss-20B) across 26 frontier models, with safety incidents up 55% year-over-year and the Transparency Index down 18 points. LLMs hallucinate at least 75% of the time on court rulings. Damien Charlotin's public database now tracks 1,300+ court and tribunal cases globally where AI-generated hallucinations were called out in legal filings, with sanctions escalating from initial $1K–5K fines to a $110,000 federal sanction in Oregon in April 2026 — the largest US AI-hallucination penalty to date, after two lawyers submitted 23 fabricated citations and eight invented quotations. Global cost of AI hallucinations was estimated at $67.4 billion in 2024.
Output injection to downstream systems — EchoLeak (CVE-2025-32711, CVSS 9.3) demonstrated zero-click data exfiltration from Microsoft 365 Copilot via prompt injection in emails. LLM outputs passed to eval(), shell commands, or rendered in browsers without encoding enable RCE, XSS, and SQL injection (OWASP LLM05:2025).
Model-triggered outbound requests — C7.3.3 covers disabling automatic external resource loading or restricting it to an explicit origin allowlist. EchoLeak exploited auto-fetched images and a Teams proxy path; Varonis' January 2026 Reprompt research showed a related single-click pattern where Copilot's q URL parameter executed attacker-supplied prompts, then chained fetches to attacker-controlled URLs to leak user context.
Localhost control-plane crossing — Microsoft's June 18, 2026 AutoJack research showed a browsing agent rendering untrusted web content could reach AutoGen Studio's localhost MCP WebSocket and spawn arbitrary processes. The individual weaknesses were ordinary — localhost origin trust, an MCP auth skip, and URL-supplied StdioServerParams — but the chain matters for C7: model-influenced browser output and tool parameters must be treated as attacker-controlled before they reach local services, even when the service only listens on loopback.
Agent-to-agent prompt escalation — ServiceNow's BodySnatcher vulnerability (CVE-2025-12420, CVSS 9.3, patched Oct 2025) showed how second-order prompt injection lets a low-privilege AI agent recruit a higher-privilege agent to exfiltrate data, modify records, or escalate to admin roles — even with built-in prompt injection protections enabled. Default configuration options (agent discovery, automatic team grouping) created the attack surface.
Deepfake misuse and legislative response — Arup lost $25.6M to a deepfake video conference (Jan 2024). Deepfake attacks occurred at one every five minutes in 2024. The TAKE IT DOWN Act (May 2025) criminalizes non-consensual intimate deepfakes in the U.S. The DEFIANCE Act passed the U.S. Senate unanimously in January 2026, establishing civil liability up to $150,000 for non-consensual sexually explicit deepfakes (awaiting House vote as of March 2026).
Citation fabrication — NeurIPS 2025 had 100+ hallucinated citations across 53+ accepted papers. Studies show ChatGPT fabricates 39-55% of citations; even GPT-4 produces 18-29% fabricated references.
Safety filter bypasses — JBFuzz (2025) achieves ~99% jailbreak success across GPT-4o, Llama 3, Gemini 2.0, DeepSeek-R1 in under 1 minute. Skeleton Key, Crescendo, and Context Compliance Attack represent distinct bypass families.
PII and training data leakage — Researchers extracted 10,000+ verbatim memorized examples from ChatGPT for ~$200. Multi-query attacks achieve 39% PII extraction rates. 77% of employees leak data via ChatGPT.
Framework-level serialization injection — CVE-2025-68664 ("LangGrinch," CVSS 9.3) demonstrated that LLM-controlled output fields (e.g., additional_kwargs, response_metadata) can trigger deserialization of attacker-crafted objects through LangChain Core's dumps()/dumpd() functions. A single prompt injection can exploit 12 distinct serialization flows — event streaming, logging, message history, caching — to extract environment variable secrets or instantiate arbitrary objects. Affects langchain-core prior to 1.2.5/0.3.81 (~98M monthly downloads). Patched December 2025.
Excessive agency — 80% of organizations report encountering risky AI agent behavior. Incidents include production environment modifications during code freezes and autonomous file destruction. The OWASP Top 10 for Agentic Applications (December 2025) catalogs agentic-specific risks including goal misalignment, tool misuse, delegated trust failures, and inter-agent communication exploits.
System prompt leakage — Now OWASP LLM07:2025. All frontier models tested leaked viable prompt approximations under argument-based jailbreaks.
Prompt injection in 73% of production deployments — As of early 2026, security audits find prompt injection vulnerabilities in 73% of assessed production AI deployments, yet only 34.7% of organizations have deployed dedicated defenses. CVE-2026-33654 demonstrated unauthenticated indirect prompt injection via email channel processing, and CVE-2025-53773 showed GitHub Copilot RCE through prompt injection modifying .vscode/settings.json.
Form-based prompt injection (ShareLeak) — CVE-2026-21520 (CVSS 7.5, patched January 15, 2026) showed Copilot Studio concatenating SharePoint form submissions directly with system instructions with no sanitization between the form input and the model. An attacker filling a public comment field with a crafted payload injected a fake system role, redirected the agent to query connected SharePoint Lists for customer data, and exfiltrated it via Outlook to an attacker-controlled address. Capsule Security reported a parallel vulnerability ("PipeLeak") in Salesforce Agentforce with similar mechanics. Microsoft's patch blocked the specific payload, but researchers demonstrated data still exfiltrated using adjacent variants — a reminder that filter-patch responses are structurally incomplete against prompt-injection attack classes.
Chain-of-logic jailbreak (CVE-2026-3098) — A multi-stage jailbreak disclosed February 2026 combining Identity Reassignment, Refusal Suppression, Output Prefix Enforcement, Strict Formatting Constraints, Length Amplification, Encoded Query Transformation, and Recursive Behavioral Reinforcement. Each component alone is weak; combined, they exert cumulative pressure on the instruction-arbitration layer and bypass safety alignment across most commercial LLMs. Related research ("Hidden You Malicious Goal Into Benign Narratives," arXiv 2404.04849) showed logic-chain injection distributes a malicious target across benign narration sentences, evading pattern-matching defenses.
LangChain/LangGraph framework CVE wave (March 27, 2026) — Unit 42 and Cyera disclosed three concurrent CVEs spanning output-handling and deserialization flaws: CVE-2025-68664 (LangGrinch serialization injection, patched in langchain-core 0.3.81/1.2.5 — now adds allowed_objects allowlist, blocks Jinja2 templates, disables secrets_from_env by default), CVE-2026-34070 (path traversal in langchain_core/prompts/loading.py allowing reads of Docker configs, SSH keys, .env; patched in langchain-core 1.2.22), and CVE-2025-67644 (SQL injection in LangGraph SQLite checkpoint metadata filter keys; patched in langgraph-checkpoint-sqlite 3.0.1). Three CVEs in one week established framework code paths as a dominant output-safety attack surface.
Prompt-to-RCE through tool parameters — Microsoft's May 7, 2026 Semantic Kernel research showed how prompt injection can turn model-selected tool arguments into host-level execution primitives: CVE-2026-26030 abused an In-Memory Vector Store filter path in Python Semantic Kernel before 1.39.4, while CVE-2026-25592 (CVSS 10.0) exposed .NET SessionsPythonPlugin file-transfer helpers — including a DownloadFileAsync helper that had been accidentally tagged [KernelFunction] with no path validation — to model control before 1.71.0. A single prompt was sufficient to launch calc.exe on the host. Microsoft's fix layered four defenses: an AST node-type allowlist, a function-call allowlist, a dangerous-attribute blocklist, and a name-node restriction; Microsoft also recommended disabling AutoInvokeKernelFunctions on any agent that can reach disk, shell, or production data and running those agents in manual function-calling mode. Flowise's CSV Agent CVE-2026-41264 followed the same pattern: LLM-generated Python code executed without adequate sandboxing in affected versions through 3.0.13. The control lesson is classic ASVS applied to model output: every model-influenced function argument crossing into code, filesystem, database, browser, or network APIs is untrusted input, and any helper visible to the model needs to be treated as a deliberate, audited export rather than a documentation convenience.
AI IDE and MCP supply-chain prompt-to-RCE (April 2026) — OX Security's April 15, 2026 advisory disclosed a class of vulnerabilities in agentic IDEs where attacker-controlled content (HTML embedded in a project, files in .agent/, RAG-retrieved content) injected instructions that silently rewrote mcp.json, registered an attacker-controlled STDIO MCP server, and triggered arbitrary command execution when the IDE next launched. CVE-2026-30615 in Windsurf 1.9544.26 was the only fully zero-click case — no approval dialog, no confirmation, no user interaction beyond loading the project. Cursor, Claude Code, and Gemini-CLI required at least one user step but still chained to RCE. Pillar Security separately disclosed a prompt-injection-to-RCE and sandbox escape in Google's Antigravity IDE — malicious Markdown placed in .agent/ was processed during normal content ingestion and combined with Antigravity's file-creation capability to break out of the sandbox; reported January 7, 2026, fixed February 28, 2026. The shared lesson: model-influenced edits to local configuration files (mcp.json, .vscode/settings.json, .cursor/, agent rules directories) are tool-parameter writes in disguise and need the same trust-boundary treatment as eval() or shell exec.
"Comment and Control" credential exfiltration via PR metadata (April 15, 2026) — Aonan Guan (Wyze Labs) with Zhengyu Liu and Gavin Zhong (Johns Hopkins) demonstrated that ordinary GitHub PR titles, issue bodies, issue comments, and HTML comments invisible to humans become a command-and-control channel for the AI coding agents that read them. Three production agents on GitHub Actions were hijacked into leaking secrets: Anthropic's Claude Code Security Review (PR title injection → exfil via PR comments and Actions logs), Google's Gemini CLI Action (issue body with a fake "Trusted Content Section" → exfil via issue comments), and GitHub Copilot Agent / SWE Agent (HTML comments in issue body → exfil via git commits inside the PR). The reachable secrets included ANTHROPIC_API_KEY, GEMINI_API_KEY, GITHUB_TOKEN (ghs_*), GITHUB_COPILOT_API_TOKEN (ghu_*), GITHUB_PERSONAL_ACCESS_TOKEN, COPILOT_JOB_NONCE, and any other secret exposed in the workflow runner environment. Anthropic's own system card already states the action is "not hardened against prompt injection"; the vendor response telegraphs the structural reality — Anthropic initially scored the issue CVSS 9.4 then downgraded to "None" and recommended tool restrictions (--disallowed-tools 'Bash(ps:*)'); GitHub closed the report as "informative" before reopening when researchers produced reverse-engineered runtime filter code, then resolved it as an "architectural limitation"; bounties were $100, $1,337, and $500 respectively. No CVE was assigned. The control lesson is that allowlisting tools (--allowed-tools) is the only stable defense, agent workflows on PRs must run with read-only credentials separated from any write-scope token, and PR/issue text must be treated as model-influenced output the moment it enters an agent's context — not as trusted workflow input.
CI/CD agent prompt injection — GitInject (June 2026) moved coding-agent prompt-injection testing into real GitHub workflow execution rather than simulated tool calls. The authors reproduced eleven attacks across four workflow families, including config-file injection through AGENTS.md, CLAUDE.md, and GEMINI.md loaded as trusted operator guidance from a pull-request branch. For C7, this reinforces that generated review output, shell commands, and repository edits need infrastructure-enforced gates: read-only tokens for untrusted events, no write-scope secrets on PR-triggered jobs, and policy checks outside the model context.
Mass-exploited AI-agent builder CVEs — Langflow CVE-2026-33017 (CVSS 9.8, disclosed March 17, 2026, fixed in 1.9.0) exposed the public POST /api/v1/build_public_tmp/{flow_id}/flow endpoint to unauthenticated callers, which routed attacker-supplied JSON node definitions straight into exec() with no sandboxing. Sysdig's threat research team observed in-the-wild exploitation within roughly 20 hours of the advisory, before any public PoC existed, and CISA added the CVE to its KEV catalog. Mass scanning was trivial because the endpoint was unauthenticated by design. Flowise CVE-2025-59528 (CVSS 10.0) saw active exploitation in April 2026. n8n's "Ni8mare" CVE-2026-21858 (CVSS 10.0, disclosed January 2026, fixed in 1.121.0) joined the same category — a Content-Type confusion in the webhook handler let unauthenticated callers overwrite internal variables and chain to arbitrary file read and full RCE; Shadowserver counted more than 24,700 unpatched instances exposed online (12,300 in North America, 7,800 in Europe) by early February, and CISA flagged the CVE as actively exploited in March 2026. The same project disclosed a parallel authenticated RCE (CVE-2026-21877, CVSS 10.0) and three core-node CVEs (CVE-2026-44789/44790/44791) across HTTP Request, Git, and XML nodes. nginx-ui's "MCPwn" CVE-2026-33032 (CVSS 9.8; upstream advisory lists all versions affected and no patched version) added a different angle: the /mcp_message endpoint shipped with an empty IP-whitelist that the middleware treated as "allow all" and never called AuthRequired(), so any network attacker on port 9000 could invoke every MCP tool — including config rewrite, restart, and arbitrary file write — with no Authorization header. Pluto Security counted roughly 2,689 publicly reachable instances; CISA listed the CVE among actively exploited vulnerabilities in March 2026. These incidents reinforced that "AI agent" builder platforms occupy the same attack-surface tier as legacy web admin consoles — they need the same hardening expectations, not relaxed defaults because they are "internal."
Unsafe framework deserialization kept moving after the March cluster — GitLab's May 8, 2026 advisory for CVE-2026-44843 described LangChain runtime paths that could revive attacker-supplied LangChain-serializable constructor dictionaries through overly broad load() allowlists. This is narrower than arbitrary Python object deserialization, but it keeps the pressure on allowlist design, inert input schemas, and defensive defaults around model-controlled structured payloads.
LiteLLM supply-chain poisoning and follow-on CVEs (March–May 2026) — Malicious litellm==1.82.7 and 1.82.8 packages were live on PyPI for about 40 minutes starting 10:39 UTC on March 24, 2026 before quarantine. The payload harvested cloud, crypto, Slack, and Discord keys from developer environments. The pressure on LiteLLM continued through April and May: CVE-2026-30623 (April 15, 2026) showed that the proxy's MCP server-creation endpoint forwarded the command field of a transport: stdio server straight to StdioServerParameters as a subprocess, giving any authenticated caller arbitrary command execution as the LiteLLM process; the fix in v1.83.6-nightly added an explicit MCP_STDIO_ALLOWED_COMMANDS allowlist (npx, uvx, python, python3, node, docker, deno) and a PROXY_ADMIN-only role gate. CVE-2026-42208 (May 8, 2026, CVSS 9.3) then exposed a pre-authentication SQL injection in proxy API key validation — a specially crafted Authorization header reached the database through the proxy's error-handling path, and Sysdig observed targeted exploitation within ~36 hours of disclosure. CISA added the CVE to KEV the same day, with FCEB agencies required to patch to v1.83.7 by May 11. Sysdig framed the blast radius as "closer to a cloud-account compromise than a typical web-app SQL injection" because a single litellm_credentials row commonly holds an OpenAI organization key, an Anthropic console key with workspace admin rights, and an AWS Bedrock IAM credential. Coverage of the original supply-chain incident framed the broader lesson as "the AI supply chain is actually an API supply chain" — output-safety controls at the model layer are bypassed entirely when the SDK or gateway itself is compromised.
OpenClaw and CrewAI agent-platform CVE clusters — Between March 18-21, 2026, nine CVEs were publicly disclosed for the popular open-source AI agent platform OpenClaw, including a CVSS 9.9 privilege-escalation flaw in the WebSocket gateway's scope handling allowing any authenticated user to escalate to admin. CERT/CC VU#221883 then disclosed four CrewAI vulnerabilities spanning code-execution fallback, arbitrary local file read, and SSRF when prompt injection reaches Code Interpreter or RAG tools. Paired with multi-week MCP-server exposure (8,000+ MCP servers found publicly reachable in February 2026 analysis), the agent ecosystem itself is now a primary output-safety concern.
Default-disabled-auth in agent platforms (CVE-2026-44338) — PraisonAI's legacy Flask API server (src/praisonai/api_server.py) hardcoded AUTH_ENABLED = False and AUTH_TOKEN = None, so its check_auth() helper returned True whenever authentication was disabled and the "protected" GET /agents and POST /chat routes failed open. Any caller who could reach the port could read agent metadata and trigger the configured agents.yaml workflow — invoking whatever LLM-provider keys, code interpreters, file I/O, and HTTP tools that workflow exposed. The advisory published at 13:56 UTC on May 11, 2026; Sysdig observed a CVE-Detector/1.0 scanner probing the exact vulnerable endpoint within 3 hours 44 minutes, with the first targeted request at 17:40 UTC the same day. Affected versions 2.5.6 through 4.6.33; patched in 4.6.34 (binds to 127.0.0.1 by default and supports --api-key). Treat any "internal" agent builder UI as internet-exposed for threat-modeling purposes — the disclosure-to-exploit window for AI infrastructure is now routinely measured in hours, not days.
Prompt-controlled shell tools via denylist bypass (CVE-2026-2256) — ModelScope's MS-Agent (ms-agent v1.6.0rc1 and earlier) exposes a Shell tool whose only guardrail is a regex-based check_safe() denylist. Attacker-controlled content that the agent retrieves — a malicious README, a poisoned web page, a planted code comment — can carry shell metacharacters or alternative-syntax payloads that bypass the denylist and reach the Shell tool intact, giving prompt injection a direct path to OS command execution with agent privileges. CERT/CC's VU#431821 documented the issue with no vendor patch at coordination time. The defensive lesson is classic ASVS applied to model output: regex blocklists never hold against adversarial input, every model-influenced tool argument crossing into a shell/exec path needs allowlisted, structured, parameterized invocation rather than string concatenation, and tools that genuinely need shell access should be feature-flagged off by default and gated behind explicit human approval through C09 oversight controls.
Agent command-stream eavesdropping (CVE-2026-32173) — Enclave AI's Yanir Tsarimi disclosed on April 2, 2026 that Microsoft's Azure SRE Agent (a cloud-operations AI agent) routed its real-time agent telemetry — live command streams, internal LLM reasoning, tool-call payloads, and credentials — through a SignalR Hub whose authentication checks validated the caller's Entra ID token but never verified that the caller belonged to the target tenant. Any signed-in Entra ID identity could silently subscribe to another customer's agent stream. Microsoft tracks the issue as CVE-2026-32173 (CVSS 8.6, Information Disclosure). The systemic point is that agent observability — the very logs and reasoning traces C12.1 logging controls expect to capture — is itself a high-value channel: a model output filter is irrelevant if internal reasoning, tool arguments, and ephemeral credentials are streamed to a hub with missing tenant isolation. The fix path is multi-tenant authorization at the connection layer, signed per-tenant subscription tokens, and end-to-end encryption for agent telemetry that contains tool-parameter or credential material.
Multimodal model-server SSRF (CVE-2026-33626) — LMDeploy's load_image() in lmdeploy/vl/utils.py (vision-language inference path, /v1/chat/completions) fetched any image_url without resolving its hostname, blocklisting private networks, or restricting schemes. GitHub published the advisory on April 21, 2026 at 15:04 UTC; Sysdig's honeypot saw the first exploit attempt 12 hours 31 minutes later, before any public PoC. In an eight-minute session the attacker hit AWS IMDS (http://169.254.169.254/latest/meta-data/iam/security-credentials/), Redis (127.0.0.1:6379), a MySQL port-scan, the OpenAPI surface, the /distserve/p2p_drop_connect kill-switch, and an OOB DNS callback for blind-SSRF confirmation. CVSS 7.5, fixed in 0.12.3. The control lesson is that C7.3's "no model-driven outbound requests to non-allowlisted origins" applies to the inference server's own data-fetch paths, not only the chat UI — multimodal endpoints that accept URLs need the same allowlist, scheme restriction, and link-local/private-IP blocking as any other server-side fetcher.
vm2 sandbox-escape wave (May 13, 2026) — Six concurrent CVEs in the deprecated-but-still-transitive vm2 Node.js sandbox turned prompt injection into host RCE wherever an AI agent framework loads it as a code-interpreter substrate. CVE-2026-43997 (CVSS 10.0) returns the host Object from within the sandbox; CVE-2026-43999 (CVSS 9.9) bypasses vm2 builtin restrictions; CVE-2026-22709 abuses Promise callback sanitization; CVE-2026-26956 escapes via Node.js 25 internals; CVE-2026-45411 chains property lookups to break confinement; and CVE-2026-24781 escapes through util.inspect (CVSS 9.8). vm2 has been officially deprecated, but it remains a transitive dependency in many AI agent frameworks, code-interpreter plugins, no-code automation builders, and integration platforms — so a model with internet access plus a tool that runs LLM-authored JavaScript inside vm2 is now a complete kill chain on its own. The migration target is isolated-vm (V8 isolates) or a hardened OS-level sandbox; "we use vm2" should now be read as "we have an unpatched RCE pending." Pair this with the Microsoft Semantic Kernel and Flowise findings and the pattern is clear: every model-driven code-execution path needs its own threat model, not a single shared assumption that the framework's sandbox holds.
IDE output-filter bypass via extension-host IPC (CVE-2026-41109) — Disclosed May 12, 2026 (CVSS 8.8), affecting Visual Studio Code prior to 1.128.1 and the GitHub Copilot extension prior to v1.43.20260512. A logic flaw in the IPC channel between the VS Code extension host and Copilot let an attacker stream raw, unfiltered model output directly into the editor — bypassing the built-in safety classifier and any organization-level policy filter, suppressing approval workflows, silently disabling telemetry consent, and leaking environment variables and API keys back to Copilot servers. The C7.3 lesson is that output-safety filters have to live on the same trust boundary as output generation: a classifier that is detachable at an IPC layer is no longer a control surface. Enterprise deployments that mandated Copilot policy filters via organization settings discovered that those filters never reached the editor stream until the patched extension shipped, so any logging that assumed the classifier had run was structurally incomplete for the disclosure window.
Encoding and representation smuggling in model output — C7.3.4 covers hidden, encoded, and misleading content in model outputs. Johann Rehberger's ASCII Smuggler research and February 2026 "Scary Agent Skills" disclosure showed Unicode-tag characters (U+E0000–E007F) carrying hidden ASCII payloads appended to ordinary emoji, variation-selector encoding (256 selectors placeable anywhere in a string), and "Sneaky Bits" binary encoding using invisible-times (U+2062) and invisible-plus (U+2064) operators — any of which can smuggle a prompt-injection instruction past a human reviewer or smuggle data back out through a model's output. Homoglyph substitution (Latin/Cyrillic/Greek lookalikes) and mixed-direction (bidi override U+202A–U+202E, U+2066–U+2069) text round out the artifact set. The control lesson is that output filters which only match visible text are structurally blind: outputs need Unicode-category normalization, tag/zero-width/bidi stripping, and confusables checks before they are returned or passed downstream, and red-team probes (garak ascii_smuggling, promptfoo's ASCII-smuggling plugin) should confirm the sanitizer catches all three encoding families.
vLLM inference-server auth bypass and supply-chain build flaw (June 2026) — vLLM, one of the most widely deployed open-source inference servers, disclosed CVE-2026-48746 (CVSS 9.1, GHSA-94f4-hr76-p5j6, fixed in 0.22.0) on June 2, 2026: injecting / or ? into the Host header desynchronizes the URL that Starlette's AuthenticationMiddleware inspects from the path FastAPI actually routes, so unauthenticated callers reach API-key-protected inference endpoints. A week later GHSA-jrf6-vqxq-pjv2 (CVSS 8.8, fixed 0.22.1) showed the vllm/vllm-openai Docker image setting UV_INDEX_STRATEGY="unsafe-best-match" globally while installing flashinfer-jit-cache, so a malicious PyPI package could be preferred over the legitimate custom-index build and execute as root at image-build time — backdooring every derived container. The lessons are classic but newly load-bearing for AI infra: in-app middleware is not a substitute for an RFC-conforming reverse proxy enforcing authentication at the network edge, and index-strategy scope must be pinned per-package so model-serving build pipelines cannot be dependency-confused.
LiteLLM command injection reaches CISA KEV (CVE-2026-42271, June 8, 2026) — Distinct from the May 8–11 LiteLLM advisories, CVE-2026-42271 (CVSS 8.7) exposed two MCP-preview endpoints (POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list) that accept a full stdio server configuration — command, args, env — and let any authenticated user run host commands; Horizon3.ai chained it with CVE-2026-48710 (CVSS 6.5, a Starlette "BadHost" host-header validation bypass) to reach fully unauthenticated RCE. CISA added it to the Known Exploited Vulnerabilities catalog on June 8, 2026 after observing active exploitation. Fixed in LiteLLM 1.83.7 and Starlette 1.0.1. The control lesson echoes the April MCP-stdio cluster: any endpoint that forwards a model-or-caller-supplied command field to a subprocess needs an allowlist and an admin-only role gate, and AI gateways holding consolidated provider keys must be patched and network-restricted as Tier-0 infrastructure.
MCP ecosystem zero-days at scale (VIPER-MCP, arXiv 2605.21392, late May 2026) — An academic taint-analysis framework auto-audited 39,884 Model Context Protocol server repositories and confirmed 106 zero-day vulnerabilities with reproducible proof-of-concept exploit traces, of which 67 have received CVE IDs through responsible disclosure (reported false-positive rate 4.6%, false-negative rate 7.7%). The study quantifies what individual MCP advisories had only suggested: unsanitized-input-to-dangerous-sink flaws are systemic across the MCP server population, not isolated to a few popular projects. The defensive implication for C7 is that any tool output or MCP response entering a model's context is untrusted input, tool execution should be sandboxed, and third-party MCP servers need their own vetting before an agent is allowed to connect.
OpenClaw "Claw Chain" sandbox-and-trust-flag cluster (May 15, 2026) — A second OpenClaw disclosure, separate from the March 18–21 cluster, in which Cyera chained four flaws patched in OpenClaw 2026.4.22 (April 23 release). CVE-2026-44112 (CVSS 9.6) and CVE-2026-44113 (CVSS 7.7) are TOCTOU races in the OpenShell managed sandbox that let agent-driven file operations escape the intended mount root for both writes and reads. CVE-2026-44115 (CVSS 8.8) shows command validation and shell execution disagreeing about semantics — environment variables (API keys, tokens, credentials) expand inside unquoted heredocs and return through commands that looked safe at validation time, a textbook example of why filter-time and execution-time string handling must match. CVE-2026-44118 (CVSS 7.8) trusted a client-controlled senderIsOwner boolean for authorization, so a non-owner loopback client could impersonate the owner and rewrite gateway configuration, cron schedules, and execution-environment settings; the fix issues separate owner/non-owner bearer tokens and derives senderIsOwner from which token authenticated the request. The structural lesson is twofold: model-influenced shell strings need real parameterization rather than regex/heredoc heuristics (the same lesson as CVE-2026-2256 in MS-Agent), and "ownership" must come from the authenticated token, not from a flag the caller sets — the same trust-derivation antipattern visible in the LangGrinch dumps() flow, the AUTH_ENABLED = False PraisonAI default, and the LiteLLM StdioServerParameters passthrough.

Notable Incidents & Research

Date	Incident / Paper	Relevance	Link
Jun 2025	EchoLeak (CVE-2025-32711, CVSS 9.3) — zero-click Microsoft 365 Copilot exfiltration	Crafted email causes Copilot to exfil OneDrive/SharePoint/Teams data without user interaction	Hack The Box
Jan 2026	NeurIPS 2025 hallucinated citations — 100+ fabricated refs in 53+ papers	Citation fabrication unnoticed by peer review; blended real papers with invented co-authors	Fortune
Jan 2026	Reprompt — single-click Microsoft Copilot Personal exfiltration	`q`-parameter prompt execution plus chained outbound fetches showed why deep links, regenerated requests, and model-rendered URLs need persistent allowlist checks	Varonis
Jan 2024	Arup deepfake video conference — $25.6M wire fraud	All participants on the call were deepfakes; worker authorized 15 wire transfers	BBC News
May 2025	TAKE IT DOWN Act signed into U.S. federal law	Criminalizes non-consensual intimate deepfakes; platforms must remove within 48 hours	Congress.gov
2025	JBFuzz — ~99% jailbreak success across frontier models	Black-box fuzzing requiring ~7 queries per harmful question, under 1 minute	Research
Mar 2025	Context Compliance Attack (CCA) — simplest effective jailbreak	Injecting fabricated assistant response into conversation history	Microsoft Research
2023-05	Air Canada chatbot hallucination (bereavement policy)	Hallucinated refund policy led to legal liability	CBC News
2023-06	Lawyers sanctioned for ChatGPT-fabricated case citations	486+ documented cases by 2025; $2K penalties and mandatory CLE	Reuters
2024-01	Taylor Swift deepfake — 47M views before takedown	Non-consensual AI-generated explicit imagery exposed platform moderation failures	NYT
Nov 2025	ServiceNow BodySnatcher (CVE-2025-12420, CVSS 9.3) — agent-to-agent prompt injection	Low-privilege agent recruits high-privilege agent to exfiltrate data, escalate to admin role	The Hacker News
Jan 2026	DEFIANCE Act passes U.S. Senate unanimously	Civil liability up to $150K for non-consensual deepfakes; spurred by Grok deepfake crisis	Roll Call
Feb 2026	India IT Rules 2026 — world's first binding synthetic content provenance mandate	Mandatory irremovable metadata and labeling for all AI-generated content; 10-day notice period	Mondaq
Jul 2025	MetaQA — metamorphic hallucination detection (ACM FSE 2025)	Self-contained detection via prompt mutation; outperforms SelfCheckGPT by 0.15-0.37 F1; works on closed-source models	ACM
Dec 2025	LangGrinch (CVE-2025-68664, CVSS 9.3) — serialization injection in LangChain Core	LLM output fields trigger deserialization of attacker objects; 12 exploitable flows including secrets extraction	Cyata
Dec 2025	OWASP Top 10 for Agentic Applications released	First standardized risk taxonomy for autonomous AI agents — covers tool misuse, delegated trust, inter-agent exploits	OWASP
Apr 2025	COPIED Act (S.1396) introduced in U.S. Senate	Bipartisan bill mandating content provenance via NIST standards; prohibits tampering with provenance metadata	Congress.gov
Mar 2026	"Fast and Faithful" — real-time long-document RAG verification	63ms/sample verification for documents up to 32K tokens; +817% hallucination recall vs 8K-truncated baselines	arXiv
Mar 2026	RT4CHART — retromorphic testing for RAG hallucination detection	Claim-level hierarchical verification against retrieved context; reported 0.776 answer-level F1 on RAGTruth++ and found 1.68x more hallucinations after re-annotation	arXiv
Jan 2026	Copilot Studio ShareLeak (CVE-2026-21520, CVSS 7.5)	Form-input concatenated directly with system prompt; SharePoint data exfiltrated via Outlook	VentureBeat
Feb 2026	Chain-of-Logic Injection jailbreak (CVE-2026-3098)	Seven-component cumulative-pressure jailbreak effective across most current LLMs	GitHub
Feb 2026	Claude Code config-injection CVEs (CVE-2025-59536, CVSS 8.7)	Malicious `.claude/settings.json` Hooks give RCE when developer opens a repo	Check Point Research
Mar 2026	CrewAI CERT/CC VU#221883	Prompt injection into Code Interpreter or RAG tools could chain RCE, arbitrary local file read, and SSRF across CVE-2026-2275, 2285, 2286, and 2287	CERT/CC
Mar 2026	LangChain/LangGraph triple-CVE disclosure (Unit 42)	CVE-2025-68664 serialization, CVE-2026-34070 path traversal, CVE-2025-67644 SQL injection	The Hacker News
Mar 2026	LiteLLM PyPI supply-chain attack	Malicious 1.82.7 / 1.82.8 live ~40 min; harvested cloud, crypto, Slack, Discord keys	OX Security
Apr 2026	Flowise CSV Agent prompt-injection RCE (CVE-2026-41264)	CSV Agent evaluated LLM-generated Python without adequate sandboxing; affected Flowise and flowise-components through 3.0.13, patched in 3.1.0	GitHub Advisory
Mar 2026	OpenClaw agent platform — 9 CVEs incl. CVSS 9.9 WebSocket escalation	Authenticated user escalates to admin via scope handling flaw	IronPlate
May 2026	Semantic Kernel prompt-to-RCE research	CVE-2026-26030 (Python <1.39.4) and CVE-2026-25592 (.NET <1.71.0, CVSS 10.0) — a single prompt could launch `calc.exe`; fixed with a four-layer AST allowlist, function-call allowlist, attribute blocklist, and name-node restriction	Microsoft
Mar 2026	Langflow CVE-2026-33017 (CVSS 9.8) — unauthenticated RCE in `build_public_tmp` endpoint	Exploited in the wild within ~20 hours of disclosure, before any public PoC; CISA added to KEV; fixed in 1.9.0	Sysdig
May 2026	LangChain unsafe deserialization advisory (CVE-2026-44843)	Overly broad `load()` allowlists could revive attacker-controlled LangChain-serializable objects from untrusted structured payloads	GitLab Advisory
Apr 2026	Windsurf zero-click MCP prompt-injection RCE (CVE-2026-30615)	Injected HTML rewrote `mcp.json`, registered attacker STDIO MCP server, executed binary on next launch; no user interaction required	PolicyLayer
Apr 2026	OX Security MCP supply-chain advisory — Cursor, Claude Code, Gemini-CLI, Windsurf	Class of agentic-IDE bugs where model output silently rewrites local MCP config and triggers RCE; one zero-click case, three one-click	The Hacker News
Feb 2026	Google Antigravity prompt-injection RCE & sandbox escape	Malicious Markdown in `.agent/` triggered file-creation primitives that broke the IDE sandbox; reported Jan 7, fixed Feb 28, 2026	Pillar Security
Feb 2026	8,000+ exposed MCP servers observed	Publicly reachable MCP endpoints without authentication became an indirect-injection vector	Medium
Apr 2026	LiteLLM MCP stdio command injection (CVE-2026-30623)	`transport: stdio` MCP server creation passed `command` field straight to `StdioServerParameters` subprocess — authenticated RCE on the proxy host; fixed in v1.83.6-nightly with an `MCP_STDIO_ALLOWED_COMMANDS` allowlist and `PROXY_ADMIN` role gate	LiteLLM Advisory
May 2026	LiteLLM pre-auth SQL injection (CVE-2026-42208, CVSS 9.3)	Crafted `Authorization` header reached the proxy database through error-handling path; targeted exploitation within ~36 hours; CISA KEV May 8, FCEB patch deadline May 11; blast radius "closer to cloud-account compromise" because credential rows hold provider keys	The Hacker News
May 2026	vm2 sandbox-escape cluster — 6 CVEs disclosed May 13	CVE-2026-43997 (CVSS 10.0 host `Object` exposure), CVE-2026-43999 (9.9 builtin bypass), CVE-2026-22709 (Promise callback), CVE-2026-26956 (Node.js 25), CVE-2026-45411 (property-lookup chain), CVE-2026-24781 (9.8 `util.inspect`) — every AI agent framework transitively loading vm2 inherits a prompt-to-host RCE primitive	Kodem
May 2026	PraisonAI CVE-2026-44338 (CVSS 7.3) — auth-disabled-by-default Flask API	Advisory at 13:56 UTC on May 11; scanner probing 3h44m later, targeted exploitation same day; affects 2.5.6–4.6.33, fixed in 4.6.34	Sysdig
May 2026	CVE-2026-41109 (CVSS 8.8) — VS Code / Copilot IPC-level safety-filter bypass	Disclosed May 12; logic flaw between extension host and Copilot streamed raw model output past the safety classifier and organization policy filters, suppressed approval workflows, leaked env vars and API keys; patched in VS Code 1.128.1 and Copilot extension v1.43.20260512	Windows News AI
May 2026	OpenClaw "Claw Chain" — CVE-2026-44112/44113/44115/44118	Disclosed May 15 by Cyera; TOCTOU sandbox escape (CVSS 9.6), heredoc credential leak (8.8), spoofable `senderIsOwner` privilege flag (7.8) chainable to data theft, privilege escalation, and persistence; patches in OpenClaw 2026.4.22 (April 23 release)	The Hacker News
Apr 2026	LMDeploy CVE-2026-33626 (CVSS 7.5) — multimodal SSRF via `load_image()`	Advisory April 21, 2026; first exploit in 12h31m before any public PoC; attacker reached AWS IMDS, Redis, MySQL, OOB DNS; fixed in 0.12.3	Sysdig
Mar 2026	MS-Agent / ModelScope CVE-2026-2256 — prompt-controlled shell command injection	Regex denylist `check_safe()` bypassed by prompt-injected shell metacharacters / encoding; CERT/CC VU#431821; no vendor patch at coordination	CERT/CC
Apr 2026	"Comment and Control" — PR-metadata prompt injection across three coding agents	Anthropic Claude Code Security Review, Google Gemini CLI Action, and GitHub Copilot Agent leaked `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GITHUB_TOKEN`, `COPILOT_JOB_NONCE`, and other runner secrets; vendors closed as "architectural limitation" rather than CVE	Aonan Guan
Apr 2026	Azure SRE Agent CVE-2026-32173 (CVSS 8.6) — multi-tenant agent telemetry exposure	SignalR Hub authenticated callers without checking tenant binding; any Entra ID identity could subscribe to another tenant's live command stream, LLM reasoning, tool calls, and credentials	Enclave AI
Apr 2026	Oregon federal $110,000 AI-hallucination sanction (largest US to date)	Two lawyers submitted 23 fabricated citations and eight invented quotations; case dismissed; Charlotin's database now tracks 1,300+ global cases	Fortune
May 2026	"Do Benchmarks Underestimate LLM Performance?" (arXiv 2605.08462)	LLM-first human-adjudicated re-evaluation of QAGS-C and SummEval — triple agreement improved 6.38%/7.62%; GPT and Gemini accuracy gains 2.3–8.5% — suggests fixed hallucination benchmarks may underweight modern models	arXiv
Apr 2026	Stanford HAI 2026 AI Index — incidents up 55%, hallucination 22–94% across 26 models	AA-Omniscience benchmark: best (Grok 4.20 Beta 0305) hallucinates 22%, worst (gpt-oss-20B) 94%; safety incidents up 55% YoY; Transparency Index down 18 points	Stanford HAI
May 2026	EU Digital Omnibus political agreement on AI Act amendments	May 7, 2026 trilogue agreement extended compliance deadlines for high-risk systems; postponed Article 50(2) machine-readable marking of generative outputs from August 2 to December 2, 2026 (four-month watermarking grace period); the other Article 50 transparency obligations (deepfake disclosure, public-interest text labelling) still apply on August 2, 2026; banned non-consensual "nudifier" deepfake apps and CSAM-generating AI systems with fines up to €35M or 7% of worldwide turnover	Latham & Watkins
Apr 2025	Panasonic joins CAI; all major makers now shipping C2PA hardware	Leica, Nikon, Canon, Fujifilm, Sony, Panasonic — Sony PXW-Z300 first camcorder with native C2PA signing (IBC 2025)	C2PA
May 2026	European Commission draft Article 50 transparency guidelines	Draft guidance opened for consultation through June 3, 2026; final code of practice expected June 2026 before Article 50 applies on August 2, 2026	European Commission
2023-03	Indirect prompt injection via Bing Chat (Greshake et al.)	Model exfiltrated data by following injected instructions in retrieved content	arXiv
Jan 2026	AgentHallu: hallucination attribution benchmark for agent trajectories (arXiv 2601.06818)	693 human-curated trajectories across 7 agent frameworks; 5 categories (Planning/Retrieval/Reasoning/Human-Interaction/Tool-Use) and 14 sub-categories; even GPT-5 and Gemini-2.5-Pro hit only 41.1% step-localization accuracy and 11.6% on tool-use hallucinations — multi-step agent reasoning is a substantially harder hallucination-evaluation surface than single-turn output	arXiv
Jan 2026	n8n "Ni8mare" CVE-2026-21858 (CVSS 10.0) — unauthenticated RCE in workflow automation platform	Content-Type confusion in webhook handler overwrote internal variables → arbitrary file read → full RCE; 24,700+ exposed instances on Shadowserver; CISA flagged actively exploited March 2026; companion CVE-2026-21877 authenticated RCE plus CVE-2026-44789/44790/44791 in HTTP Request, Git, and XML nodes; fixed in 1.121.0	The Hacker News
Mar 2026	nginx-ui "MCPwn" CVE-2026-33032 (CVSS 9.8) — unauthenticated MCP-tool takeover	`/mcp_message` SSE endpoint shipped without `AuthRequired()`; empty default IP whitelist treated as "allow all" so any caller on port 9000 could invoke every MCP tool, rewrite nginx config, and reload the service; ~2,689 instances publicly reachable; CISA-flagged exploited; upstream advisory lists all versions affected and no patched version	Upstream advisory
Apr 2026	Splunk MCP Server CVE-2026-20205 (CVSS 7.2) — clear-text session/auth tokens in `_internal` index	MCP Server app failed to mask session and Authorization tokens before logging to `_internal`; any role with that index plus the `mcp_tool_admin` capability could hijack sessions and pivot inside Splunk; CWE-532; fixed in 1.0.3 — the same observability surfaces C12.1 logging controls expect to capture for audit are themselves credential-bearing material if not masked	Splunk Advisory
May 2026	EU Digital Omnibus — Article 50(2) watermarking deadline officially moved to December 2, 2026	Provisional agreement on May 7 publishes provider obligations to mark generative outputs in a machine-readable format on December 2, 2026 (four-month grace from the original August 2 date); other Article 50 transparency duties (deepfake disclosure, public-interest text labelling) keep their August 2 application date; awaiting formal endorsement and Official Journal publication	White & Case
Jun 2026	vLLM auth-bypass CVE-2026-48746 (CVSS 9.1) + Docker dependency-confusion GHSA-jrf6-vqxq-pjv2 (CVSS 8.8)	Host-header desync between Starlette auth middleware and FastAPI routing exposed API-key-protected inference endpoints (fixed 0.22.0); global `unsafe-best-match` index strategy let a malicious PyPI package backdoor the official image at build time (fixed 0.22.1)	vLLM Advisory
Jun 2026	LiteLLM CVE-2026-42271 (CVSS 8.7) added to CISA KEV	MCP-preview endpoints accept a full stdio server config and run host commands; chained with Starlette CVE-2026-48710 host-header bypass for unauthenticated RCE; actively exploited, KEV-listed June 8; fixed in LiteLLM 1.83.7 / Starlette 1.0.1	The Hacker News
May 2026	VIPER-MCP — 106 MCP-server zero-days, 67 CVEs assigned (arXiv 2605.21392)	Taint-analysis framework audited 39,884 MCP server repos and confirmed 106 zero-day input-to-sink flaws with PoC traces (FP 4.6%, FN 7.7%); quantifies the MCP ecosystem's systemic unsanitized-input problem	arXiv
Jun 2026	Ninth Circuit published sanctions order (No. 24-4790) — AI-hallucinated citations	First precedential circuit-level AI-hallucination sanction: two immigration attorneys suspended six months and fined $2,500 each, with two-year firm-wide GenAI disclosure certifications, after briefs cited nonexistent opinions and fabricated quotes	Ninth Circuit Order (No. 24-4790)
May 2026	MITRE ATLAS "Secure AI v2" expansion	Cumulative 45+ new techniques/sub-techniques, 10+ mitigations, 20+ case studies, expanded agentic-AI coverage, a Technique Maturity filter and ATLAS Knowledge Graph; ATLAS moves to a monthly release cadence	MITRE CTID
May 2026	FTC begins TAKE IT DOWN Act enforcement	Section 3 NCII 48-hour-removal duties became enforceable May 19, 2026; FTC sent warning letters to 15+ platforms (Alphabet, Amazon, Apple, Meta, Microsoft, Reddit, TikTok, X, and others); penalties up to ~$53,088 per violation	IAPP
Jun 2026	EU final Code of Practice on marking and labelling AI-generated content	Presented at the Closing Plenary on June 10, 2026; Section 1 covers Art. 50(2) machine-readable provider marking, Section 2 covers Art. 50(4) deployer labelling of deepfakes and public-interest AI text; voluntary compliance-demonstration tool ahead of August 2, 2026 transparency rules	European Commission
Jun 2026	AutoJack — localhost MCP control-plane RCE in AutoGen Studio development builds	Untrusted web content rendered by a browsing agent reached a localhost MCP WebSocket and supplied `StdioServerParams` that spawned arbitrary processes; fixed before PyPI release, but the pattern generalizes to agent-accessible local control planes	Microsoft
Jun 2026	GitInject — real GitHub workflow prompt-injection evaluation	Eleven attacks across four AI-powered CI/CD workflow families; config-file injection through provider guidance files loaded from PR branches shows why agent output and repo edits need workflow-level controls outside the model context	arXiv
Jun 2026	LegalHalluLens — typed hallucination auditing for legal workflows	Aggregate legal hallucination rates around 52% hide 38-40 percentage-point gaps between claim types; typed debate reduced fabricated detections by 45%, supporting claim-category-specific verification gates for high-risk domains	arXiv

Tooling & Implementation

Current tools, frameworks, and libraries that help implement these controls:

Output validation frameworks: Guardrails AI (Pydantic-style validators with a ~70-validator hub; v0.10.2 on June 4, 2026 was a maintenance release adding a security-advisory doc set, a GitHub Actions template-injection fix, and a litellm>=1.83.0 pin), NeMo Guardrails (NVIDIA, Colang DSL for programmable rails — v0.22.0 on May 22, 2026 advanced IORails to milestone 2 with streaming support, OpenTelemetry integration, reasoning-model support, and speculative generation that runs input rails in parallel with main-LLM generation, and made LangChain optional via a built-in OpenAI-compatible httpx client; v0.21.0 (March 12, 2026) introduced the opt-in IORails engine (NEMO_GUARDRAILS_IORAILS_ENGINE=1) running NemoGuard content-safety, topic-safety, and jailbreak rails in parallel with per-request logging and unique request IDs, a check_async method on LLMRails for standalone I/O validation, an OpenAI-compatible server with a /v1/models endpoint, and a GuardrailsMiddleware that wraps LangChain agents; v0.20.0 (January 22, 2026) added reasoning-capable content safety models such as Nemotron-Content-Safety-Reasoning-4B with a configurable /think mode and GLiNER as an open PII detector; BotThinking events have been available since v0.18.0), LLM Guard (Protect AI, input/output sanitization scanners; note the project is dormant — latest release 0.3.16 dates to May 2025), Rebuff (multi-layer prompt injection detection with canary tokens)
Encoding & representation smuggling detection (C7.3.4): no single tool covers all three artifact classes (invisible/tag characters, homoglyphs, and bidi text), so a practical output-side stack composes several. AID — ASCII/Invisible Character Detector (Johann Rehberger) scans text for Unicode-tag characters (U+E0000–E007F) and auto-decodes them to ASCII, zero-width characters (ZWSP/ZWJ/ZWNJ), bidi/directional marks (U+200E/F, U+202A–E, U+2066–2069), variation selectors, and invisible operators, emitting graded suspicion levels; the companion ASCII Smuggler web tool and the "Scary Agent Skills" writeup document the encoding families. LLM Guard's InvisibleText scanner strips non-printable Unicode by category (Cf/Cc/Co/Cn plus Private Use Areas), satisfying the rejection-or-sanitization need — but it ships as an input scanner, so it must be invoked manually on responses to cover output. The homoglyph leg, which AID and InvisibleText leave open, is best handled with confusable_homoglyphs (Unicode confusables matrix) or a UTS #39 skeleton check. For assurance, exercise the sanitizer with offensive probes: garak's ascii_smuggling probes (NVIDIA — Tag Smuggling, Variant Smuggling, and "Sneaky Bits" binary encoding) and promptfoo's ASCII-smuggling red-team plugin. AWS also published prescriptive guidance on defending LLM apps against Unicode character smuggling using Bedrock Guardrails-based tag-character filtering.
Hallucination detection: Vectara HHEM-2.3 + FaithJudge (Nov 2025–Mar 2026 refresh — cross-encoder scorer plus an LLM-as-judge approach trained on labelled hallucinations; FaithJudge with o3-mini-high reports 84.0% balanced accuracy and 82.1% F1 on the new corpus of 7,700+ articles spanning ten domains and lengths up to 32K tokens — Gemini-2.5-flash-lite leads at 3.3% hallucination while Claude Sonnet 4.5 and GPT-5 remain above 10%, and high-complexity inputs hallucinate substantially more), RAGAS (RAG evaluation — faithfulness, context precision/recall), DeepEval (14+ metrics, pytest-native CI/CD integration; v4.0.x in late May 2026 added RetrievedContextData retrieval-source tracking for RAG faithfulness, multimodal trace support, and day-0 Claude Opus 4.8 structured-output support), TruLens (groundedness and relevance feedback), Lynx (Patronus AI, RAG faithfulness evaluation), LettuceDetect (KRLabs, ModernBERT/EuroBERT token-classification — flags exact hallucinated spans inside answers, 79.22% example-level F1 on RAGTruth at 30–60 examples/sec on a single GPU, supports 4K–8K context windows and multilingual models, pip-installable Python API), MetaQA (ACM FSE 2025, metamorphic prompt mutation, works on closed-source models without token probabilities), RT4CHART (March 2026, claim-level retromorphic verification for RAG answers), W&B Weave HallucinationFree (LLM-as-judge scorer, 91% accuracy / 86% recall in benchmarks), Galileo Luna-2 (sub-200ms latency, ChainPoll multi-model consensus, ~$0.02/M tokens)
Fine-grained grounding checks (June–July 2026): CORTEX compares a model's token representations with and without retrieved documents to localize unsupported spans; its authors report improvements across two RAG benchmarks and three models, but the method needs access to internal model states and is not a drop-in check for closed APIs. Beyond Document Grounding broadens span-level evaluation to code, tool output, Markdown, tables, and repository metadata: its Qwen3.5-2B detector reached 0.60 span-F1 on the code-agent source, versus 0.17 for LettuceDetect-large and at most 0.22 for the tested zero-shot judges. Auditors evaluating C7.2 should stratify tests by evidence type and require exact-span labels; a detector validated only on prose RAG should not be assumed to cover tool traces or code.
Structured output / constrained decoding: OpenAI Structured Outputs and Anthropic strict tool use now provide provider-native schema enforcement for JSON/tool inputs, while PydanticAI builds JSON Schema from Pydantic types and validates returned model data. Self-hosted stacks can use Outlines (FSM token masking, 100% schema compliance), Instructor (post-generation validation with retries), Guidance, XGrammar/llguidance (near-zero overhead grammar-guided generation; llguidance reports roughly 50 microseconds of CPU mask computation per token for JSON-schema-derived grammars), and SGLang structured outputs (JSON Schema, regex, EBNF, and structural tags; XGrammar is the default backend). Auditors should still validate semantic safety after schema conformance: a schema-valid string can be a malicious URL, shell argument, citation, or HTML fragment.
Content safety classifiers: LlamaGuard 4 (Meta, 12B multimodal, MLCommons taxonomy), ShieldGemma (Google, 2B/9B/27B sizes, exceeds GPT-4 F1 by 6.4%), GPT-OSS-Safeguard (OpenAI, Oct 2025; Apache 2.0 open-weight in 120B and 20B variants; interprets developer-provided policy at inference time with visible reasoning traces; outperforms gpt-5-thinking and gpt-oss baselines on multi-policy accuracy per technical report), OpenAI Moderation API, Azure AI Content Safety, Perspective API
Media provenance & watermarking: C2PA v2.3 (February 2026 — current spec release, adds live-streaming/CMAF segment signing; advancing through ISO as ISO/DIS 22144; a Conformance Program with a public Conforming Products List is live, and Pixel 10 is the first certified smartphone), SynthID (Google, 10B+ watermarks, text watermarking open-sourced; OpenAI joined the C2PA steering committee in May 2024 and now embeds SynthID signals in its generated images alongside C2PA Content Credentials, with a public verification tool that checks uploads for both), Meta Video Seal (Dec 2024, open-source video watermarking)
PII detection in outputs: Microsoft Presidio (open-source, pluggable NER, 10+ languages), Nightfall AI (2x precision over AWS Comprehend/Google DLP), AWS Comprehend PII (30+ entity types), Google Cloud DLP (150+ infoTypes)
Rendered-output and outbound-request controls: For chat UIs that render Markdown or HTML, pair a sanitizer such as DOMPurify with Trusted Types, context-aware output encoding from the OWASP XSS Prevention Cheat Sheet, and explicit CSP fetch directives such as img-src and connect-src. Auditors should verify that model-generated links, images, iframes, link previews, prefetches, and pre-filled prompt URLs cannot trigger outbound requests except through reviewed, logged, allowlisted origins.
RAG attribution & verification: Google Vertex AI Check Grounding API (verifies response grounding, returns unsupported claims), GaRAGe Benchmark (2,366 questions, 35K+ annotations for citation accuracy evaluation), "Fast and Faithful" (arXiv 2603.23508) (March 2026, real-time verification for documents up to 32K tokens — 63ms/sample with +817% hallucination recall over 8K-truncated models; early-exit inference at 24ms/sample for latency-sensitive deployments)
Citation-record verification: CiteCheck retrieves candidate scholarly records, compares citation metadata with a structured verifier, and labels matches Exact, Minor, or Major. On its 982-citation physics benchmark it reports 88.7 macro-F1 and 88.9% accuracy, outperforming the paper's tested general-model baselines. This is useful for a C7.4 negative test that mutates a real title, author, year, or identifier and injects a nonexistent reference, but it does not prove that a cited source supports the generated claim; chunk-level entailment and retrieval-metadata provenance still need separate checks.

Implementation Maturity

Control Area	Tooling Maturity	Notes
C7.1 Output Format Enforcement	Mature	Outlines achieves 100% schema compliance via FSM token masking. JSON mode available in all major providers. XGrammar/llguidance and SGLang make grammar-constrained decoding practical for self-hosted stacks. The residual gap is semantic: syntactically valid JSON can still contain unsafe strings, fabricated values, or malicious tool arguments.
C7.2 Hallucination Detection & Mitigation	Maturing	HHEM-2.3 plus FaithJudge (Vectara, refreshed through March 2026 — 7,700+ articles, ten domains, lengths up to 32K tokens; FaithJudge reports 84.0% balanced accuracy and 82.1% F1), RAGAS, DeepEval provide production-ready metrics. Vertex AI Check Grounding offers API-level verification. MetaQA (ACM FSE 2025) enables detection on closed-source models via prompt mutation. RT4CHART (Mar 2026) adds claim-level hierarchical verification and suggests older RAG benchmarks undercount hallucinations. LettuceDetect (ModernBERT/EuroBERT token classification) provides span-level flags at 30–60 examples/sec, useful when an auditor needs to point at exactly which sentence is unsupported. Galileo Luna-2 hits sub-200ms for real-time guardrails. W&B Weave reaches 91% accuracy. "Fast and Faithful" extends verification to 32K-token documents at 63ms/sample with +817% hallucination recall. The agent-trajectory frontier is harder than the single-turn one: AgentHallu (arXiv 2601.06818, January 2026) released a 693-trajectory benchmark across 7 frameworks with a 5-category taxonomy (Planning, Retrieval, Reasoning, Human-Interaction, Tool-Use), and the best of 13 evaluated models (GPT-5, Gemini-2.5-Pro tier) reached only 41.1% step-localization accuracy with tool-use hallucinations at 11.6% — so auditors validating confidence scoring and high-risk verification on agent outputs should expect the underlying detection to lag behind single-turn faithfulness scoring by a wide margin. Note: reasoning models may perform worse at grounded summarization — DeepSeek-R1 scores 14.3% hallucination rate vs. V3's 6.1% on Vectara benchmarks, and even the current leader still hallucinates several percent of the time. AIMultiple's 2026 benchmark also caps the best detectors at roughly 90–91% recall, meaning ~1 in 10 hallucinated outputs still slip past automated checking. Still no standardized cross-vendor benchmark.
C7.3 Output Safety	Mature	LlamaGuard 4, ShieldGemma, GPT-OSS-Safeguard (open-weight, Apache 2.0) offer multiple classifier options. Presidio handles PII well. NeMo Guardrails v0.21.0 (March 12, 2026) ships IORails as an opt-in parallel content-safety/topic-safety/jailbreak engine with logging and unique request IDs, a `check_async` method for standalone I/O validation, an OpenAI-compatible server, and a `GuardrailsMiddleware` for LangChain agents; v0.20.0 (January 22, 2026) added reasoning-capable safety models (Nemotron-Content-Safety-Reasoning-4B with `/think`) and GLiNER PII detection, while `BotThinking` events for guardrailing reasoning traces have been available since v0.18.0 — together these narrow the gap between "we have a classifier" and "we have observability over the rail's decisions." The harder C7.3 gap is not classifier coverage but browser and client behavior: auto-rendered Markdown images, iframes, link previews, URL-prefilled prompts, and repeated fetches need CSP-backed allowlists, Trusted Types, sanitizer regression tests, and egress logging. The 2026 Semantic Kernel, Flowise, LangChain, CrewAI, and LiteLLM advisories show that safety filtering must be paired with ASVS-style validation of model-influenced parameters before they reach tools, and CVE-2026-2256 in ModelScope's MS-Agent shows that regex-denylist filters on shell-tool arguments will be bypassed by prompt-injected metacharacters — allowlist parameterization is the only stable defense. Server-side fetchers are now in scope too: LMDeploy CVE-2026-33626 turned `load_image()` in a vision-language inference server into a full SSRF primitive against AWS IMDS, Redis, and MySQL, so any model-driven URL fetch (image loader, link preview, RAG fetcher, tool input) needs hostname resolution, scheme restriction, and link-local/private-IP blocking. AutoJack extends the same point to localhost: a browsing agent that renders untrusted web content can cross into developer control planes unless every WebSocket/API path is authenticated, authorized, and isolated. And the May 13, 2026 vm2 cluster (CVE-2026-43997 CVSS 10.0, CVE-2026-43999, plus four siblings) shows that the code-interpreter sandbox itself is part of C7.3's attack surface whenever an agent runs LLM-authored JavaScript — treat any deprecated sandbox (vm2 in particular) as an unpatched RCE and migrate to isolated-vm or OS-level isolation. The May 12, 2026 VS Code / Copilot CVE-2026-41109 added the IDE delivery layer to the same threat model: a classifier that runs in a separate process from the editor stream can be sidestepped at the IPC boundary, so output-safety filtering needs to live on the same trust boundary as output generation and any organization-policy filter has to be enforced end-to-end, not at a downstream surface that an extension host can detach. The May 15, 2026 OpenClaw "Claw Chain" cluster (CVE-2026-44112/44113/44115/44118) reinforces the underlying point — shell-tool guardrails and sandbox boundaries fail when validation-time and execution-time semantics disagree (unquoted-heredoc shell expansion leaked environment-variable credentials through commands that passed the validator) and authorization derived from a caller-supplied flag (`senderIsOwner`) is not authorization at all. C7.3.4's hidden/encoded/misleading-output clause remains a distinct gap: detection tooling here is emerging and fragmented — AID covers invisible/tag/bidi characters, LLM Guard's `InvisibleText` strips non-printable Unicode by category but only as an input scanner, and homoglyph detection needs a separate confusables/UTS-39 check, so a production output filter has to be assembled from multiple libraries and regression-tested with garak/promptfoo smuggling probes rather than bought off the shelf.
Explainability & Transparency (C7.3)	Emerging	SHAP/LIME work for classification but CoT faithfulness remains unverifiable for LLMs. Mechanistic interpretability (sparse autoencoders) is research-stage. EU AI Act pressure driving adoption.
Generative Media Safeguards	Maturing	C2PA v2.3 is the current specification release and adds work such as live-streaming/CMAF segment signing and durable Content Credentials through soft bindings. Hardware and newsroom adoption continues across Leica, Nikon, Canon, Fujifilm, Panasonic, Sony, BBC, AP, Reuters, AFP, and The New York Times. OpenAI signs Sora 2 video and DALL·E 3 images with Content Credentials plus invisible watermarking; Google does the same across Imagen 4, Veo 3, and Lyria 2 using SynthID; Adobe Firefly, Microsoft Designer, Stability AI, and Black Forest Labs FLUX have followed. Detection evasion remains an arms race — detectors lose up to 50% accuracy on novel deepfakes; deepfake incidents grew from ~500K in 2023 to 8M+ in 2025. The biggest distribution-layer gap: effectively 100% of major social platforms (Instagram, X, LinkedIn, TikTok, Facebook) strip C2PA manifests during upload via re-encoding, privacy stripping, and size-optimization pipelines, so durable provenance now depends on pairing the signed manifest with an invisible watermark and a perceptual-hash recovery service rather than relying on metadata alone. Regulatory pressure intensified with the EU Transparency Code of Practice published June 10, 2026; Article 50 transparency duties apply August 2, 2026, while Article 50(2) machine-readable watermarking for generative outputs has been pushed to December 2, 2026 by the May 7, 2026 Digital Omnibus political agreement. Important caveat: C2PA does not detect or classify content as real or fake — it only records provenance. Detection remains a separate unsolved problem.
C7.4 Source Attribution & Citation Integrity	Emerging	Vertex AI Check Grounding API exists but up to 57% of RAG citations may be post-rationalized. Citation accuracy averages 65-70% without explicit attribution training. GaRAGe benchmark (2025) standardizing evaluation.

OWASP LLM05:2025 Improper Output Handling — covers downstream XSS, SQL injection, RCE from untrusted LLM outputs; linked to LLM01 and LLM06 attack chains
OWASP LLM06:2025 Excessive Agency — three root causes: poorly designed integrations, unchecked tool access, lack of usage monitoring
OWASP LLM07:2025 System Prompt Leakage — new in 2025; all frontier models tested leak prompt approximations
MITRE ATLAS — AML.T0054 (LLM Jailbreak), AML.T0051 (Prompt Injection — .000 Direct, .001 Indirect), AML.T0062 (Discover LLM Hallucinations), AML.T0048 (External Harms), AML.T0015 (Evade ML Model)
NIST AI 600-1 (July 2024) — 12 GenAI risk categories including Confabulation, Dangerous Content, Information Integrity, Harmful Bias, and Obscene Content. Confabulation is defined as "confidently stated but false or internally inconsistent outputs" — a narrower term than "hallucination" that emphasizes the confident presentation. NIST specifically flags the risk that GAI "outputs may also include confabulated logic or citations that purport to justify or explain the system's answer," which directly motivates C7.2 hallucination controls and C7.4 citation provenance controls. Over 400 recommended actions map to the AI RMF's Govern, Map, Measure, and Manage functions.
EU AI Act Article 50 — Transparency obligations for AI-generated content become applicable August 2, 2026, with one carve-out finalised by the May 7, 2026 Digital Omnibus political agreement: the Article 50(2) provider obligation to ensure that generative outputs are marked in a machine-readable format is deferred from August 2 to December 2, 2026 (White & Case). Providers must mark generated or manipulated audio, image, video, and text in a machine-readable format where technically feasible; deployers must disclose deepfakes and certain AI-generated public-interest text, with exceptions for law-enforcement use and human editorial responsibility. The Commission's Article 50 work now includes the June 10, 2026 Transparency Code of Practice and forthcoming guidelines; the Commission says the code is voluntary but can reduce administrative burden for signatories after a positive adequacy assessment.
C2PA Specification v2.3 — cryptographic Content Credentials for media provenance; v2.3 is the current specification release, adds durable Content Credentials and manifest discovery through soft bindings, and is adopted by Adobe, Google, Nikon, Canon, Sony, and other CAI participants
CSA AI Controls Matrix (July 2025) — 243 controls across 18 domains including Model Security and Application & Interface Security
ISO/IEC 42001:2023 — AI management systems; transparency and explainability provisions
EU AI Code of Practice on Transparency — published June 10, 2026 and under Commission / AI Board adequacy assessment. Section 1 sets provider obligations for machine-readable marking and detection of AI-generated or manipulated content; Section 2 sets deployer obligations for labelling deepfakes and public-interest AI-generated text. The Commission describes the code as voluntary but legally useful evidence for Article 50 compliance, with signatory taskforces expected to share implementation practice.
MITRE ATLAS — The Center for Threat-Informed Defense's "Secure AI v2" wrap-up (May 6, 2026) announced cumulative additions of 45+ techniques and sub-techniques, 10+ mitigations, and 20+ case studies, with expanded agentic-AI threat coverage, a new Technique Maturity filter, an ATLAS Knowledge Graph, a Rapid Response Report model for emerging incidents, and a move to a monthly release cadence — relevant to output-handling and agent-action techniques referenced throughout C7.
TAKE IT DOWN Act — FTC enforcement of the Section 3 platform duties (48-hour removal of non-consensual intimate imagery, including AI-generated NCII) began May 19, 2026; the FTC sent warning letters to 15+ major platforms, with penalties up to roughly $53,088 per violation. Relevant to generative-media safeguards and C7.3 output filtering for synthetic NCII.
India IT Rules 2026 Amendment (Feb 2026) — World's first binding synthetic content provenance mandate. Requires irremovable metadata, unique identifiers, mandatory labeling. Three-hour takedown window for prohibited content. Platforms cannot suppress provenance markers.
California SB 942 — AI Transparency Act; operative date aligned to Aug 2026 (via AB 853). Requires latent disclosure in AI-generated media. Fines up to $5,000 per daily violation.
DEFIANCE Act (S.1837) — Passed U.S. Senate unanimously Jan 2026 (awaiting House). Civil liability up to $150,000 for non-consensual sexually explicit deepfakes, 10-year statute of limitations.
COPIED Act (S.1396) — Bipartisan Content Origin Protection and Integrity from Edited and Deepfaked Media Act (April 2025). Mandates NIST-developed content provenance and watermarking standards, prohibits tampering with provenance metadata, and authorizes FTC and state AG enforcement.
OWASP Top 10 for Agentic Applications (December 2025) — First standardized risk taxonomy for autonomous AI agents. Covers tool misuse, delegated trust, inter-agent communication exploits, goal misalignment, and cascading failures. The December 2025 release formalized "ASI" (Agentic Security Issue) identifier prefixes for individual risks.

AISVS Cross-Chapter Links

Related Chapter	Overlap Area	Notes
C02 User Input Validation	Input/output attack chain	OWASP LLM05 explicitly chains with LLM01 (Prompt Injection). MITRE AML.T0051 (Prompt Injection) directly causes output manipulation. Input validation (C02) is the first defense; output controls (C07) are the last.
C06 Supply Chain	Behavioral impact of compromised models	MITRE AML.T0043.004 (Insert Backdoor Trigger) shows supply chain compromise altering model behavior. NIST AI 600-1 "Value Chain & Component Integration" risk category.
C09 Orchestration & Agents	Action limiting, tool use	OWASP LLM06 (Excessive Agency) is the direct bridge. LLM05 + LLM06 combined create attack chains where malicious output triggers unauthorized agent actions.
C11 Adversarial Robustness	Safety filter robustness	MITRE AML.T0054 (LLM Jailbreak) and AML.T0015 (Evade ML Model) target output filter robustness. JBFuzz achieves ~99% bypass rates against current models.
C12 Monitoring and Logging	Safety violation monitoring	OWASP LLM06 root cause 3 is "Lack of Usage Monitoring." Output-safety telemetry from C7 feeds into C12 centralized logging. MITRE AML.T0065 involves reconnaissance of model behaviors.
C09 Orchestration and Agentic Action	Human-in-the-loop	C7.2.3 high-risk response verification and C7.3 output safety gates relate to C09 oversight patterns. EU AI Act Art 50 allows text disclosure exemption when human review with editorial control is applied. NIST AI 600-1 "Human-AI Configuration" risk category.

Open Research Questions

Can hallucination detection scale to real-time production use? — HHEM-2.1 runs in 0.6s on RTX 3090 vs ~35s for RAGAS with GPT-4. As of early 2026, Galileo Luna-2 achieves sub-200ms latency at ~$0.02/M tokens, making real-time guardrails economically viable. MetaQA (ACM FSE 2025) works on closed-source models via prompt mutation. W&B Weave reaches 91% accuracy. "Fast and Faithful" extends verification to 32K-token documents at 63ms/sample with early-exit at 24ms — a significant step for enterprise documents (contracts, regulatory filings) that were previously truncated. RT4CHART (Mar 2026) pushes the field toward claim-level, evidence-linked labels rather than answer-level scores, and its re-annotation work suggests older RAG datasets may substantially undercount hallucinations. The picture is not one-directional, though: Atasoy et al. (arXiv 2605.08462, May 8, 2026) re-evaluated QAGS-C and SummEval with an LLM-first human-adjudicated protocol and found that triple agreement between humans, GPT-5 Mini, and Gemini 2.5 Flash improved by 6.38% (QAGS-C) and 7.62% (SummEval), with GPT and Gemini accuracy gains of 2.3–8.5% — meaning fixed benchmark annotations may also underestimate current models when annotators were less consistent than the model. The combined lesson is that any chosen hallucination benchmark needs periodic adjudicated re-annotation, not just a one-time label set. Vectara's HHEM-2.3 plus FaithJudge (refreshed through March 2026) expanded the leaderboard from 1,000 to 7,700+ articles across ten domains and lengths up to 32K tokens — even the leader (Gemini-2.5-flash-lite at 3.3%) hallucinates non-trivially, and frontier models like Claude Sonnet 4.5 and GPT-5 stay above 10%, with high-complexity inputs consistently producing more hallucinations. Importantly, reasoning models may actually increase hallucination risk: DeepSeek-R1 scores 14.3% hallucination on grounded summarization vs. V3's 6.1%, suggesting chain-of-thought reasoning can harm factual grounding. Cross-encoder models remain fast but miss complex multi-hop reasoning errors. AgentHallu (arXiv 2601.06818, January 2026) made the multi-step picture explicit: 13 frontier models including GPT-5 and Gemini-2.5-Pro topped out at 41.1% step-localization accuracy across 693 agent trajectories, with tool-use hallucinations at just 11.6% — so detection on agent workflows lags well behind single-turn faithfulness scoring, and tool-invocation-history signals are necessary but not sufficient. May-June 2026 work keeps narrowing the problem rather than closing it: K-FinHallu shows multi-turn financial RAG detectors still struggle with fine-grained diagnostics and justified abstention, TRIVIA+ shows long-context RAG benchmarks need realistic label-noise stress tests, and LegalHalluLens shows aggregate legal hallucination rates hide materially different risk directions by claim type. Still no standardized cross-vendor benchmark.
How effective are current jailbreak defenses? — JBFuzz achieves ~99% success across frontier models with ~7 queries. Multi-turn jailbreaks exceed 70% success rates. Constitutional AI and RLHF improve single-turn safety but multi-turn and compositional attacks remain largely unsolved.
Will C2PA and watermarking survive adversarial manipulation? — C2PA metadata can be stripped from files, and in 2026 effectively 100% of major social platforms (Instagram, X, LinkedIn, TikTok, Facebook) strip manifests during re-encoding and privacy stripping — so a signed manifest alone no longer carries provenance across the consumer-distribution layer. Newsroom adoption (BBC, AP, Reuters, AFP, NYT requiring signed wire imagery) and provider-side signing (OpenAI Sora 2 and DALL·E 3, Google Imagen 4 / Veo 3 / Lyria 2, Adobe Firefly, Stability AI, Black Forest Labs FLUX) raise the floor at origin, but the post-publication recovery story still depends on durable invisible watermarks plus perceptual-hash fingerprinting. Deepfake detectors lose up to 50% accuracy on novel content not in training data. The EU AI Code of Practice (second draft Mar 2026) and the Commission's May 8, 2026 draft Article 50 guidelines both explicitly acknowledge no single technique is sufficient and recommend layered approaches; the guidelines also confirm that "technically feasible" is judged against currently available technology, not provider preference. India's IT Rules 2026 go further by mandating irremovable metadata — but technical enforceability against determined adversaries remains untested. The COPIED Act (S.1396), if passed, would add U.S. federal enforcement via FTC and state AGs. C2PA is advancing toward ISO international standardization, which would strengthen global adoption. Regulatory convergence (EU Aug 2026, California Jan 2026, India Feb 2026, TAKE IT DOWN Act compliance May 2026) is creating compliance pressure even if individual techniques remain breakable.
Can RAG citation verification become reliable? — Up to 57% of RAG citations are post-rationalized. Citation accuracy averages 65-70% without explicit attribution training. Vertex AI Check Grounding API is a start, but claim-level verification (C7.4.3) remains a hard problem.
How should explainability requirements apply to LLMs? — CoT explanations cannot be verified by asking the LLM itself. Mechanistic interpretability (sparse autoencoders) is promising but pre-production. EU AI Act explainability obligations for high-risk systems take effect Dec 2, 2027 (deferred from Aug 2026 by the May 2026 Digital Omnibus).
Are AI frameworks themselves a hidden output-safety attack surface? — The March 27, 2026 LangChain/LangGraph CVE cluster made this question operational. CVE-2025-68664 (LangGrinch) allowed LLM outputs flowing through dumps()/dumpd() to trigger object rehydration and secret extraction; CVE-2026-34070 opened path traversal via prompt loading; CVE-2025-67644 enabled SQL injection through LangGraph SQLite checkpoint metadata filter keys. The May 2026 Semantic Kernel research then showed prompt-controlled tool parameters becoming eval() payloads and host file-write primitives (CVE-2026-25592 reached CVSS 10.0 — the exposed DownloadFileAsync had been silently tagged [KernelFunction]), Flowise CVE-2026-41264 showed CSV analysis turning into Python execution, and LangChain CVE-2026-44843 showed that deserialization allowlists remain hard to narrow correctly. The April 2026 OX Security MCP advisory expanded the surface again — Windsurf CVE-2026-30615 turned out to be zero-click (rewriting mcp.json and registering an attacker STDIO server) and Cursor / Claude Code / Gemini-CLI were vulnerable variants of the same class. Pillar Security's Antigravity finding (fixed February 2026) showed the same pattern from .agent/ Markdown rather than MCP. Langflow CVE-2026-33017 (CVSS 9.8) added a different lesson — unauthenticated public endpoints exposed by AI builder UIs were exploited at internet scale within 20 hours of disclosure, no prompt injection required. The LiteLLM trajectory tightened the screw: a March 2026 PyPI compromise, then CVE-2026-30623 (MCP stdio command injection through StdioServerParameters), then CVE-2026-42208 (pre-auth SQL injection, exploited within 36 hours, CISA KEV May 8, 2026). And the May 13, 2026 vm2 cluster added an underneath layer — any agent framework transitively loading vm2 as its code-interpreter sandbox now inherits CVE-2026-43997 (CVSS 10.0) and five siblings as a prompt-to-host RCE primitive, regardless of how careful the prompt-injection defenses above it are. Security audits find prompt injection in 73% of production deployments but only 34.7% have defenses. PraisonAI CVE-2026-44338 (May 11, 2026, scanned within 3h44m of disclosure) reinforced a different version of the same lesson — the agent-builder platform's control plane (a Flask API server with AUTH_ENABLED = False by default) is just as exploitable as the model's data plane, regardless of how careful the prompt-injection defenses on top are. n8n's "Ni8mare" CVE-2026-21858 (CVSS 10.0, ~24,700 instances exposed on Shodan, CISA-flagged actively exploited) and nginx-ui's "MCPwn" CVE-2026-33032 (CVSS 9.8, empty-by-default IP whitelist on /mcp_message treated as "allow all," ~2,689 publicly reachable instances) extend the same pattern across the broader workflow-automation and MCP-tool-server stack — unauthenticated public endpoints in AI/workflow infrastructure are an attacker's preferred entry point, no prompt injection required. And LMDeploy CVE-2026-33626 (12h31m to first exploit) extended the surface into the inference server's own data-fetch paths: a vision-language load_image() without hostname resolution became a complete SSRF primitive against AWS IMDS, Redis, and MySQL. Splunk MCP Server CVE-2026-20205 added an observability-side lesson: the very _internal logs that C12.1 logging controls expect to capture for audit shipped session and Authorization tokens in clear text, so any role with that index plus the mcp_tool_admin capability could harvest credentials — making "log everything" a credential-handling problem, not just a storage problem. AutoJack adds the localhost boundary to the list: loopback-only control planes are reachable to agents that can browse or execute code on the same host. GitInject adds CI/CD runner state: config files, workflow permissions, and token storage can turn model-output manipulation into repository write impact. The gap between output generation and downstream framework / package-manager / builder-platform / IDE-configuration / sandbox-runtime / inference-server / observability / CI processing is now the leading attack surface, not the model.
Can filter-patch responses ever match prompt-injection attack classes? — The ShareLeak episode (CVE-2026-21520) is instructive: Microsoft patched the specific payload, and researchers immediately demonstrated that data still exfiltrated via variant payloads operating on the same structural weakness (form input concatenated into system context with no trust boundary). Reprompt showed the same pattern from a different entry point: patching the first request is not enough if regenerated requests, deep links, and outbound fetches keep operating with the user's ambient authority. Until frameworks enforce a hard separation between control-plane instructions and data-plane inputs — something current LLM architectures do not natively support — blocklist and filter patches will remain a treadmill. Capsule Security's "PipeLeak" in Salesforce Agentforce (same pattern, parallel discovery) suggests this is a whole-class problem, not a vendor-specific bug.

C07-04 Source Attribution & Citation Integrity — Expands the RAG attribution, claim support, and provenance controls summarized in the chapter hub.
C02-01 Prompt Injection Defense — Covers the upstream injection patterns that often drive unsafe outputs, tool calls, and rendered-content exfiltration.
C02-02 Content Policy Screening — Connects pre-model content screening with the response-side classifiers and fallback gates used in C7.
C09-03 Tool and Plugin Isolation — Tracks how agent-to-agent messages, tool outputs, and shared memory should be validated before later components trust them.
C11-01 Model Alignment & Safety — Pairs C7 output controls with pre-execution review and adversarial checks for autonomous agents.

Community Notes

Space for contributor observations, discussion, and context that doesn't fit elsewhere.