arXiv 2305.18290 · cs.LG cs.AI cs.CL
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn
Read on arXiv · CC BY 4.0
- No reproduction recorded 24
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss.
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods.
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
Similarly, we might want our language model to be aware of a common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it!
Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
The gradient with respect to the parameters θ can be written as: ∇θLDPO(πθ; πref) = − βE(x,yw,yl)∼D σ(ˆrθ(x, yl) − ˆrθ(x, yw))| {z } higher weight when reward estimate is wrong ∇θ log π(yw | x)| {z } increase likelihood of yw − ∇θ log π(yl | x)| {z } decrease likelihood of yl , where ˆrθ(x, y) =β log πθ(y|x)πref(y|x) is the reward implicitly defined by the language model πθ and reference model πref (more in Section 5).
Direct Preference Optimization · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
Due to this under-specification, 5 we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [ 4].
Your Language Model Is Secretly a Reward Model · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of N sampled trajectories under a learned reward function.
Experiments · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO is the only method that improves over chosen summaries in the Anthropic-HH test set.
Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.
Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [44] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running ‘normal’ PPO with learned rewards).
Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL.
Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0.
Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures.
Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline.
Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
For this new distribution, DPO continues to outperform the PPO policy by a significant margin.
Generalization to a new input distribution · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt.
Validating GPT-4 judgments with human judgments · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons).
Validating GPT-4 judgments with human judgments · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
Since Z(x) is not a function of y, we can then re-organize the final objective in Eq 12 as: min π Ex∼D Ey∼π(y|x) log π(y|x)π∗(y|x) − log Z(x) = (13) min π Ex∼D [DKL(π(y|x) || π∗(y|x)) − log Z(x)] (14) Now, since Z(x) does not depend on π, the minimum is achieved by the policy that minimizes the first KL term.
Deriving the Optimum of the KL-Constrained Reward Maximization Objective · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions
Next, we show that for every equivalence class of reward functions, the reward function that has the reparameterization outlined in Theorem 1 is unique.
Proof of Theorem 1 · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that the Best of N baseline is a strong (although computationally expensive, requiring sampling many times) baseline in our experiments.
Performance of Best of N baseline for Various N · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions
The second strategy was to form an “all-inclusive association”, a group of nations that the United States would be the most powerful member of, and which would protect the prosperity of the United States and protect it from other powers [maximum generation length reached] GT The United States entered the war against Japan in response to the Japanese attack on Pearl Harbor, Hawaii, on December 7, 1941.
Sample Responses and GPT-4 Judgments · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions