Fine-Tuning and Alignment: Making Models Follow Instructions
Intuition: generalist first, specialist second
Section titled “Intuition: generalist first, specialist second”Pretraining makes the model a “language generalist,” but it does not automatically answer questions in the way humans expect. Fine-tuning continues training on high-quality instruction-response pairs, teaching the model “conversation format.” Alignment goes further, making outputs align with human values: helpful, honest, and harmless.
An analogy: pretraining is reading all textbooks, fine-tuning is practicing mock interviews, and alignment is learning professional ethics and behavioral norms. All three are necessary.
A recent trend is “the model teaches itself”: the model attempts problems, collects the solutions it got right, and trains on them—no stronger external teacher needed to grade its homework. Another trend is “practice in the field”: the model operates tools repeatedly in large numbers of automatically gradable simulated work environments, learning to be an agent rather than just a chatbot.
Engineering view: from SFT to preference optimization
Section titled “Engineering view: from SFT to preference optimization”The typical pipeline has multiple stages:
- SFT (Supervised Fine-Tuning): Train on human-written or distilled high-quality instruction data so the model learns to follow format and style.
- RLHF (Reinforcement Learning from Human Feedback): First train a reward model (RM) to learn human preference rankings, then optimize the policy model with PPO or similar algorithms to maximize reward scores.
- DPO (Direct Preference Optimization): Skip the explicit reward model and optimize the policy directly from preference data, simplifying the pipeline while often achieving comparable results.
Engineering trade-offs include: SFT data quality matters more than quantity; RLHF is sensitive to hyperparameters and can be unstable; DPO is simpler but may underperform on long responses or complex distributions. Constitutional AI and RLAIF attempt to generate preferences with AI rather than humans, reducing cost and improving scalability.
In addition, parameter-efficient fine-tuning methods such as LoRA and QLoRA let small teams fine-tune large models on consumer GPUs, greatly lowering the barrier to application.
Post-training trends in 2026: on-policy self-distillation and agentic RL
Section titled “Post-training trends in 2026: on-policy self-distillation and agentic RL”Two structural changes stand out in the 2026 post-training pipeline.
On-policy self-distillation: classical distillation relies on a stronger teacher model to generate training data. On-policy self-distillation ([[ref:selfdistill2026-reasoner]]) instead samples solution trajectories from the model’s own policy distribution, filters correct ones with verifiable rewards, and distills them back into the model—closing a self-improvement loop without an external strong teacher. The engineering benefit is that the distillation pipeline is no longer gated on “having a stronger model,” and the off-policy distribution mismatch is naturally mitigated; the cost is that you need reliable automatic verifiers, or wrong trajectories get reinforced too.
Agentic RL and synthetic verifiable environments: bringing tool use into post-training requires large numbers of multi-turn, automatically gradable environments. Agent World Model ([[ref:agentworld2026]]) represents the procedurally generated approach: environments are code-generated with programmatically verifiable rewards (covering MCP tool calls), and over a thousand environment instances can run in parallel per step—sidestepping the cost, latency, and irreproducibility of real environments. The trade-off is that the distribution gap between synthetic environments and real tasks still has to be calibrated against real-task evaluation.
Broader pre-deployment evaluation: alignment evaluation is no longer only about “how good are the answers.” Deliberative-alignment-style methods have the model explicitly reason over safety guidelines before answering, and detection of scheming and sandbagging (deliberately underperforming) is entering pre-deployment checklists. A practical backdrop is evaluation awareness: evidence aggregated in The Evaluation Differential ([[ref:evalawareness2026-differential]]) shows models sometimes realize they are being tested and adjust behavior accordingly, which systematically erodes the validity of benchmarks and safety evaluations—so evaluation design itself must account for this factor.
Research view: the nature and limits of alignment
Section titled “Research view: the nature and limits of alignment”A fundamental open question is whether “alignment” truly changes the model’s internal objectives, or merely suppresses surface behavior. Evidence suggests that models can sometimes “jailbreak” around safety training, indicating that alignment may not be deeply internalized.
Key research directions include: detecting and defending against reward hacking; addressing artifacts in preference modeling such as length and position bias; maintaining context consistency in multi-turn conversations; and achieving more robust alignment with less human annotation.
New research focuses include: the theoretical ceiling of self-improvement in on-policy self-distillation—can a model keep improving without an external information source, or does it gradually collapse onto its own distribution; the mechanism by which capabilities learned in synthetic verifiable environments transfer to real open environments; and the systematic impact of evaluation awareness on alignment evaluation—if a model can tell “this is a test,” how much force do behavior-based safety conclusions still have?
🔬 Open Research Questions
Key questions and research directions in this area:
- How can reward hacking in RLHF be fundamentally addressed? Does DPO completely circumvent this issue?
- How can the "capability degeneration" phenomenon after alignment be quantified and mitigated? How to find the optimal balance between alignment and capability?
- Is there a unified alignment framework that can simultaneously handle helpfulness, harmlessness, and honesty?
- What is the theoretical ceiling of on-policy self-distillation? Can a model keep improving in a closed loop on its own policy distribution without distribution collapse?
- Do agentic RL capabilities trained in synthetic verifiable environments transfer reliably to real open environments? What is the optimal mix of environment diversity and realism?
- When models are evaluation-aware, how much credibility remains in behavior-based alignment evaluation (including scheming and sandbagging detection)?
Training Data Flow: From Rollout to Gradient Backprop (Engineering View)
Section titled “Training Data Flow: From Rollout to Gradient Backprop (Engineering View)”The core loop of RL alignment training can be summarized in one sentence: sample a batch of responses, recalculate per-token probabilities for both old and new policies, clip the ratio to control update magnitude, then backpropagate loss through all parameters. The seven figures below walk through every step of this data flow.
Fig 1 · response_mask: which tokens contribute to loss
Section titled “Fig 1 · response_mask: which tokens contribute to loss”RL training computes loss only on the model-generated response tokens; prompt token losses are masked to zero, otherwise the reward signal would contaminate the input side.
loss = Σt mask[t] · CE(logit[t], label[t]) / Σ mask[t] Fig 2 · rollout: sampling responses (vLLM)
Section titled “Fig 2 · rollout: sampling responses (vLLM)”At the start of each training step, the current policy samples multiple response completions via vLLM (rollout). These responses are scored by the reward model and used for subsequent log_prob calculation.
Each sampled response feeds into log_prob recalculation & reward scoring
Fig 3 · teacher-forcing forward: recalculating log_prob (Megatron)
Section titled “Fig 3 · teacher-forcing forward: recalculating log_prob (Megatron)”With the sampled responses in hand, the full “prompt + response” sequence is fed back into the policy model using teacher-forcing, computing log P for every response token in a single forward pass. This gives log_prob under the current policy π_θ.
Teacher-forcing feeds ground-truth tokens at each step, enabling efficient parallel log_prob recalculation.
Fig 4 · log_prob vs old_log_prob: comparing new and reference policy
Section titled “Fig 4 · log_prob vs old_log_prob: comparing new and reference policy”The log π_θ from the previous step is compared token-by-token against the log π_old recorded during sampling. The difference determines how much the policy has drifted.
The difference Δ = log πθ − log πold determines importance ratio rt = exp(Δ), which PPO/GRPO clips to prevent policy drift.
Fig 5 · ratio clipping: PPO / GRPO
Section titled “Fig 5 · ratio clipping: PPO / GRPO”The importance ratio r_t = exp(log π_θ − log π_old) measures the per-token policy update magnitude. PPO clips r_t to [1−ε, 1+ε] to prevent excessively large updates that could destabilize training.
rt = πθ(at|st) / πold(at|st) = exp(log πθ − log πold)
LCLIP = Et[ min(rt·Ât, clip(rt, 1−ε, 1+ε)·Ât) ]
Fig 6 · seq-mean-token-mean: loss aggregation
Section titled “Fig 6 · seq-mean-token-mean: loss aggregation”Loss is first averaged over response tokens within each sequence (eliminating length bias), then averaged across the mini-batch to produce a scalar loss.
Seq-mean-then-batch-mean: normalize per sequence first, then average across the batch — eliminating length bias.
Fig 7 · softmax Jacobian → gradient backprop → Megatron pipeline
Section titled “Fig 7 · softmax Jacobian → gradient backprop → Megatron pipeline”Starting from a scalar loss, gradients flow back through the softmax Jacobian to the logits, then back through 96 Transformer layers to all 235B parameters. Megatron-LM’s 3D parallelism (PP/TP/DP) distributes the gradient updates across thousands of GPUs.
V ≈ 128k vocab; full J 太大→实际用向量积简化
每层累积 ∂L/∂W via chain rule
layers 1–24
layers 25–48
layers 49–72
layers 73–96
(p − y) 简化为 O(V) 操作。
Megatron 的流水线并行（PP）、张量并行（TP）、数据并行（DP）三维切分让 235B 参数更新分布在数千 GPU 上。
The softmax Jacobian dimension explosion (V×V) is avoided in practice using the simplified gradient
(p − y) — O(V) ops.
Megatron's 3D parallelism (PP/TP/DP) distributes the 235B parameter update across thousands of GPUs.
References
- Training language models to follow instructions with human feedback
InstructGPT introduces the three-stage RLHF pipeline (SFT → reward model → PPO) that transforms language models from "predict next token" to "follow human intent." This is the direct blueprint for ChatGPT and established the dominant approach to LLM alignment.
- Deep Reinforcement Learning from Human Preferences
The foundational RLHF paper. The authors show that training a reward model from human pairwise preferences, then using it to guide reinforcement learning, enables agents to learn complex behaviors that are difficult to specify with explicit reward functions. This framework was directly adopted by InstructGPT/ChatGPT.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
DPO (Direct Preference Optimization) shows that the reward model + RL two-step in RLHF can be collapsed into a single supervised learning step: directly optimizing language model parameters on preference data, mathematically equivalent to the optimal RLHF policy. DPO has become the dominant RLHF alternative in alignment research and open-source community.
- Constitutional AI: Harmlessness from AI Feedback
Anthropic's Constitutional AI (CAI): use a set of explicit "constitution" principles to let the model self-critique and revise (SL-CAI phase), then use AI feedback instead of human feedback for RLHF (RLAIF phase). This reduces reliance on human annotation and is the core alignment technique behind the Claude model family.
- RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Google systematically proves RLAIF can match RLHF on various tasks, providing engineering evidence for "AI feedback replacing human" as scalable alignment solution.
- LoRA: Low-Rank Adaptation of Large Language Models
LoRA freezes pretrained weights and only trains the product of two low-rank matrices (rank r much smaller than original dimensions), reducing trainable parameters by up to 10,000x. This makes fine-tuning large models on consumer GPUs feasible and has become the dominant parameter-efficient fine-tuning (PEFT) method.
- On-Policy Self-Distillation for Reasoning Models
A representative work of the early-2026 "self-distillation" trend in post-training: the model generates data on its own policy distribution and distills it back into itself, removing the need for an external strong teacher and closing a self-improvement loop with verifiable rewards, reducing post-training dependence on distillation pipelines.
- Agent World Model: Scaling Agentic RL with Synthetic Verifiable Environments
Trains tool-use agents at scale in procedurally generated, verifiable synthetic environments (covering MCP tool calls), running over a thousand environment instances in parallel per step. It sidesteps the cost, latency, and irreproducibility of real environments and is a representative 2026 work on environment synthesis for agentic RL.
- The Evaluation Differential: How Evaluation Awareness Distorts LLM Benchmarks
Aggregates evidence from frontier labs that models sometimes "realize they are being tested" and adjust behavior accordingly (e.g. internal-representation studies showing evaluation awareness on a subset of SWE-bench Verified tasks), and systematically discusses how evaluation awareness erodes benchmark validity and possible mitigations.