MearcAll papers

arXiv 2305.18290 · cs.LG cs.AI cs.CL

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_eb71a39b
In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cfe7bc76
Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0b7b75b1
Similarly, we might want our language model to be aware of a common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it!

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5a572929
In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c2b7e10d
We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3a7c251e
Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7c74465e
Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_56a4459f
The gradient with respect to the parameters θ can be written as: ∇θLDPO(πθ; πref) = − βE(x,yw,yl)∼D  σ(ˆrθ(x, yl) − ˆrθ(x, yw))| {z } higher weight when reward estimate is wrong  ∇θ log π(yw | x)| {z } increase likelihood of yw − ∇θ log π(yl | x)| {z } decrease likelihood of yl  , where ˆrθ(x, y) =β log πθ(y|x)πref(y|x) is the reward implicitly defined by the language model πθ and reference model πref (more in Section 5).

Direct Preference Optimization · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5d21154a
Due to this under-specification, 5 we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [ 4].

Your Language Model Is Secretly a Reward Model · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cca53997
We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of N sampled trajectories under a learned reward function.

Experiments · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9eca5ef8
Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO is the only method that improves over chosen summaries in the Anthropic-HH test set.

Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d38f69c4
We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.

Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0b2cca34
In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [44] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running ‘normal’ PPO with learned rewards).

Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2edd9ebb
We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL.

Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_436dcc92
We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0.

Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_58dca497
Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures.

Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d7d40f95
Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline.

Experiments · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6be8ba43
For this new distribution, DPO continues to outperform the PPO policy by a significant margin.

Generalization to a new input distribution · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ece3ca5a
The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt.

Validating GPT-4 judgments with human judgments · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9462f4ce
We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons).

Validating GPT-4 judgments with human judgments · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_70ca8a51
Since Z(x) is not a function of y, we can then re-organize the final objective in Eq 12 as: min π Ex∼D  Ey∼π(y|x)  log π(y|x)π∗(y|x)  − log Z(x) = (13) min π Ex∼D [DKL(π(y|x) || π∗(y|x)) − log Z(x)] (14) Now, since Z(x) does not depend on π, the minimum is achieved by the policy that minimizes the first KL term.

Deriving the Optimum of the KL-Constrained Reward Maximization Objective · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fb85ecd0
Next, we show that for every equivalence class of reward functions, the reward function that has the reparameterization outlined in Theorem 1 is unique.

Proof of Theorem 1 · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a86738eb
We find that the Best of N baseline is a strong (although computationally expensive, requiring sampling many times) baseline in our experiments.

Performance of Best of N baseline for Various N · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6fc959e4
The second strategy was to form an “all-inclusive association”, a group of nations that the United States would be the most powerful member of, and which would protect the prosperity of the United States and protect it from other powers [maximum generation length reached] GT The United States entered the war against Japan in response to the Japanese attack on Pearl Harbor, Hawaii, on December 7, 1941.

Sample Responses and GPT-4 Judgments · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2305.18290) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.