MearcAll papers

arXiv 2203.02155 · cs.CL cs.AI cs.LG

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, and 14 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_1b890ac9
Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3b479d4d
Our InstructGPT models (PPO-ptx) as well as its variant trained without pretraining mix (PPO) significantly outperform the GPT-3 baselines (GPT, GPT prompted); outputs from our 1.3B PPO-ptx model are preferred … (excerpt)

Model · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a6059301
InstructGPT models show improvements in truthfulness over GPT-3.

Model · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a53e9bba
InstructGPT shows small improvements in toxicity over GPT-3, but not bias.

Model · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d292cecc
InstructGPT models generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful.

Model · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_63601566
On our API prompt distribution, our FLAN and T0 models perform slightly worse than our SFT baseline, and labelers significantly prefer InstructGPT to these models (InstructGPT has a 73.4±2% winrate … (excerpt)

Model · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ac06d8e0
Overall, our results indicate that fine-tuning large language models using human preferences significantly improves their behavior on a wide range of tasks, though much work remains to be done to … (excerpt)

Model · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_da4a417d
Similarly to Wu et al. (2021), we find that our SFT models overfit on validation loss after 1 epoch; however, we find that training for more epochs helps both the … (excerpt)

Models · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8cba3891
This is much more computationally efficient because it only requires a single forward pass of the RM for each completion (rather than (K 2 ) forward passes forK completions) and, … (excerpt)

Models · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b53fc8f0
We find that GPT-3 outputs perform the worst, and one can obtain significant step-size improvements by using a well-crafted few-shot prompt (GPT-3 (prompted)), then by training on demonstrations using supervised … (excerpt)

Results on the API distribution · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8a8b7d58
We also found that our results do not change significantly when evaluated on prompts submitted to GPT-3 models on the API (see Figure 3), though our PPO-ptx models perform slightly … (excerpt)

Results on the API distribution · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f4425d77
Compared to GPT-3, the PPO models are more appropriate in the context of a customer assistant, are better at following explicit constraints in the instruction and attempting the correct instruction, … (excerpt)

Results on the API distribution · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_92ebe89a
In particular, according to held-out workers, all of our InstructGPT models still greatly outperform the GPT-3 baselines.

Model · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7ed241fe
This indicates that these datasets are not sufficiently diverse to improve performance on our API prompt 12 distribution.

Model · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_910a8f15
We believe our InstructGPT model outperforms FLAN and T0 for two reasons.

Model · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dcbc5616
As measured by human evaluatoins on the TruthfulQA dataset, our PPO models show small but significant improvements in generating truthful and informative outputs compared to GPT-3 (see Figure 6).

Results on public NLP datasets · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d8aa246f
Interestingly, the exception is our 1.3B PPO-ptx model, which performs slightly worse than a GPT-3 model of the same size.

Results on public NLP datasets · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4aaca356
When evaluated only on prompts that were not adversarially selected against GPT-3, our PPO models are still significantly more truthful and informative than GPT-3 (although the absolute improvement decreases by … (excerpt)

Results on public NLP datasets · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_001a7f4f
Our improvements in truthfulness are also evidenced by the fact that our PPO models hallucinate (i.e. fabricate information) less often on closed-domain tasks from our API distribution, which we’ve shown … (excerpt)

Model · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9fde3b51
To summarize: all of our models are rated as less toxic than expected given the prompt (they get a negative score on a scale from -1 to 1, where 0 … (excerpt)

Model · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c8ccefc7
In Figure 29 we show that adding pretraining updates to our PPO fine-tuning (PPO-ptx) mitigates these performance regressions on all datasets, and even surpasses GPT-3 on HellaSwag.

Model · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f3c36f7e
To give a few examples: (1) when given an instruction with a false premise, the model sometimes incorrectly assumes the premise is true, (2) the model can overly hedge; when … (excerpt)

Qualitative results · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fdb11eb6
Empirically, around 96% of our dataset (110k datapoints) is classified as English, although we estimate that the actual fraction may be 99% or higher, due to classifier inaccuracies.

Data diversity · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3fb173bf
That is, if Output A and Output B are both plausible (i.e. would get a continuity score of 2), and Output A is a bit more plausible but also a … (excerpt)

Labeler demographic data · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_23c47671
Larger 175B RMs had the potential to achieve lower validation loss, but (1) their training was more unstable which made them less suitable for use as initializations for the PPO … (excerpt)

Details of RM training · page 41 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_69092657
Label: [positive / negative] Dataset features • 872 binary multiple choice sentiment analysis questions • In the few-shot setting, there are 15 additional text / label pairs.

Prompt structure and evaluation features for each eval dataset · page 48 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c346de3c
InstructGPT doesn’t significantly improve over GPT-3 on these datasets.

Measuring bias · page 53 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d9e2eefc
As can be seen, on DROP and SquadV2, the model starts out with better performance than the GPT-3 model.

Model · page 56 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8f5f2862
In the RealToxicityPrompts task, we measure toxicity via the Perspective API and find that the toxicity of our model outputs is highly correlated with the toxicity of the input prompt, … (excerpt)

RealToxicityPrompts results as a function of input toxicity · page 58 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ee53a70f
In order to better capture our models’ behavior in unsafe regimes, we draw 5000 examples from the RealToxicityPrompts dataset with an approximately uniform distribution over prompt toxicity and report average … (excerpt)

RealToxicityPrompts results as a function of input toxicity · page 58 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21677822
Some preliminary experiments show better human Likert scores can be achieved with a pretraining data ratio of 32.

Additional ablations · page 58 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b14f39d3
InstructGPT 175B completion: - One bullet point of evidence of a positive trend in the employment market could be that recent job gains have been greater than in prior months, … (excerpt)

Model samples · page 65 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2203.02155) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.