Awesome LLM Jailbreak Papers

Defense

• Automatic Prompt Optimization with "Gradient Descent" and Beam Search

• Jailbreaker in Jail: Moving Target Defense for Large Language Models

• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

• Self-Guard: Empower the LLM to Safeguard Itself

• Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM

• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

• BASELINE DEFENSES FOR ADVERSARIAL ATTACKS AGAINST ALIGNED LANGUAGE MODELS

Benchmark

• Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

• Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models

• LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI’s ChatGPT Plugins

Prompt Injection

• Prompt Injection attack against LLM-integrated Applications

• Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• BACKDOORING INSTRUCTION-TUNED LARGE LANGUAGE MODELS WITH VIRTUAL PROMPT INJECTION

Fuzzing

• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Model

Role Play

• Quack: Automatic Jailbreaking Large Language Models via Role-playing

• Jailbreaking Language Models at Scale via Persona Modulation

• Role-Play with Large Language Models

Empirical Study

• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

• Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks

• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study (NDSS 2024)

• Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

LLM-based (Finetune a LLM to ask)

• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Prompt Engineering

• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs
• AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models

• Defending ChatGPT against Jailbreak Attack via Self-Reminder

• Shield and Spear: Jailbreaking Aligned LLMs with Generative Prompting

• Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models

• Open Sesame! Universal Black Box Jailbreaking of Large Language Models

• Jailbreaking Black Box Large Language Models in Twenty Queries

• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

Visual Adversarial Examples

• Misusing Tools in Large Language Models With Visual Adversarial Examples

• Visual Adversarial Examples Jailbreak Aligned Large Language Models

• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models

• Image Hijacks: Adversarial Images can Control Generative Models at Runtime

• Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs

Backdoor

• Universal Jailbreak Backdoors from Poisoned Human Feedback

• Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models

Cross-lingual

• Multilingual Jailbreak Challenges in Large Language Models

• Low-Resource Languages Jailbreak GPT-4

Others

• Jailbroken: How Does LLM Safety Training Fail?

• Multi-step Jailbreaking Privacy Attacks on ChatGPT

• Prompt Injection Attacks and Defenses in LLM-Integrated Applications

• Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots

• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation