Jailbroken: How Does LLM Safety Training Fail?

Authors: Alexander Wei, Nika Haghtalab, Jacob Steinhardt (2023)

arXiv: 2307.02483

Domains

Safety

TLDR (English)

Systematically classifies jailbreak methods (out-of-distribution, goal conflict) and explains why RLHF struggles to eradicate them. Reference material for jailbreak research "taxonomy".

TLDR（中文）

系统化分类越狱手段（在分布外、目标冲突），并解释为什么 RLHF 难以根治。是越狱研究"taxonomy"参考资料。

Appears in These Articles

• 安全与对抗：模型的防护与攻击
• Safety and Adversarial: Protecting and Attacking Models

Co-cited Papers

These papers appear in the same articles as this one

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai et al. (2022)
• Extracting Training Data from Large Language Models Nicholas Carlini et al. (2021)
• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake et al. (2023)
• Red Teaming Language Models with Language Models Ethan Perez et al. (2022)

Related Papers

Other papers in the same domain

• OpenAI o1 System Card OpenAI (2024)
  ReasoningSafety
• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake et al. (2023)
  Safety
• Universal and Transferable Adversarial Attacks on Aligned Language Models Andy Zou et al. (2023)
  Safety
• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
  AlignmentSafety
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai et al. (2022)
  AlignmentSafety
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli et al. (2022)
  SafetyEvaluation