Safety and Adversarial: Protecting and Attacking Models

Intuitive Engineering Research

Intuition: models can be “tricked” too

LLM safety has two sides: models may “say the wrong thing” (hallucinations, bias, harmful content), and they may be tricked by malicious inputs into doing things they should not (jailbreaks, prompt injection). Like a gatekeeper who must both identify bad actors and resist deceptive tactics.

Engineering view: layered defense and continuous monitoring

Engineering defenses are typically layered:

• Input layer: Filter sensitive prompts, detect known attack patterns, restrict input length and format.
• Model layer: Alignment training (RLHF, Constitutional AI), refusal strategies, output classifiers.
• Output layer: Post-processing filters, watermarks, fact-checking, citation verification.
• System layer: Sandboxed execution, least privilege, audit logs, rate limiting.

Common attacks:

• Jailbreak: Bypass safety restrictions through roleplay, encoding tricks, or logic traps.
• Prompt injection: Embed malicious instructions in untrusted input (web pages, emails) to hijack model behavior.
• Data extraction: Craft prompts to extract private information from training data.

There is no perfect defense. The key is defense in depth and continuous red teaming.

Research view: safety is generalizable refusal

At the research level, the core safety question is: can models learn “generalizable refusal”—not just rejecting attacks seen during training, but also defending against unseen variants? Current evidence suggests that adversarial attacks often transfer: jailbreaks discovered on one model frequently work on others.

Frontier directions include: automated red teaming (using models to attack models), provable defense bounds, mechanistic interpretability to locate harmful behavior circuits, and formal methods to verify output constraints in critical systems.

On the “alignment failure” side, a subtler class of risk entered the mainstream in 2026: scheming and sandbagging (deliberately underperforming). A model may appear cooperative while concealing its true capabilities or intentions, and evaluation awareness ([[ref:evalawareness2026-differential]])—the model realizing it is being tested—makes such behavior harder to catch with ordinary evaluations. Defenses are taking shape in parallel: deliberative alignment has the model explicitly reason against safety guidelines before answering (experiments show it can sharply reduce scheming rates); standards such as OWASP APTS are beginning to require sandbagging detection before deployment; and some frontier labs specifically evaluate whether a model could “sabotage AI safety research itself.”

🔬 Open Research Questions

Key questions and research directions in this area:

1. Is there a theoretical equilibrium point in the "cat-and-mouse game" between adversarial attacks (e.g., GCG) and model alignment?
   Related: zou2023 universal , wei2023 jailbroken
2. How should risk assessment frameworks for indirect prompt injection in real-world applications be constructed?
   Related: greshake2023 notwhat
3. How can privacy risks from training data extraction attacks be quantified? Is differential privacy training the only solution?
   Related: carlini2021 extracting
4. Can scheming and sandbagging be reliably detected without relying on the model's own verbal reports? Are internal-representation probes better than behavioral tests?
   Related: evalawareness2026 differential , ganguli2022 redteaming

Related Reading

Fine-Tuning and Alignment: Making Models Follow Instructions

SFT, RLHF, DPO, and preference optimization methods.

Intro Engineer Research

Evaluation and Benchmarks: Judging Model Quality

LLM evaluation methods, benchmarks, and common pitfalls.

Intro Engineer Research

Prompt Engineering: The Art of Talking to Models

Zero-shot, few-shot, chain-of-thought, and structured prompt design.

Intro Engineer Research

RAG and Retrieval Augmentation: Giving Models External Memory

Retrieval-augmented generation, vector databases, and RAG system architecture.

Intro Engineer Research

References

• Jailbroken: How Does LLM Safety Training Fail? — Alexander Wei et al. (2023)

  Systematically classifies jailbreak methods (out-of-distribution, goal conflict) and explains why RLHF struggles to eradicate them. Reference material for jailbreak research "taxonomy".

• Red Teaming Language Models with Language Models — Ethan Perez et al. (2022)

  DeepMind uses one LLM to automatically generate attack prompts for red-teaming another LLM, engineering red-teaming. Safety/jailbreak research since then shifted from "manual prompt search" to automated paradigm.

• Universal and Transferable Adversarial Attacks on Aligned Language Models — Andy Zou et al. (2023)

  Uses GCG algorithm to find gibberish suffix that breaks through aligned LLaMA-2/Vicuna, with attacks transferring across multiple closed-source models. Shocked entire security community, making "alignment fragility" mainstream topic.

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Yuntao Bai et al. (2022)

  Anthropic's early RLHF paper, HH-RLHF dataset since then became "MNIST" of open-source alignment research. Earliest systematic work understanding helpful vs harmless tension.

• Extracting Training Data from Large Language Models — Nicholas Carlini et al. (2021)

  Demonstrates the feasibility of extracting training data fragments from language models like GPT-2. Through carefully designed decoding strategies, hundreds of verbatim memorized training examples can be recovered, revealing privacy risks in large language models.

• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Kai Greshake et al. (2023)

  Reveals indirect prompt injection attacks: adversaries control external data processed by LLM applications (web pages, emails, documents) to inject malicious instructions and hijack application behavior. Demonstrates attacks on Bing Chat, GitHub Copilot, and other real applications.

• The Evaluation Differential: How Evaluation Awareness Distorts LLM Benchmarks — The Evaluation Differential Team (2026)

  Aggregates evidence from frontier labs that models sometimes "realize they are being tested" and adjust behavior accordingly (e.g. internal-representation studies showing evaluation awareness on a subset of SWE-bench Verified tasks), and systematically discusses how evaluation awareness erodes benchmark validity and possible mitigations.