clm_1b890ac9Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.
arXiv 2203.02155 · cs.CL cs.AI cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_1b890ac9Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.
res_3b479d4dOur InstructGPT models (PPO-ptx) as well as its variant trained without pretraining mix (PPO) significantly outperform the GPT-3 baselines (GPT, GPT prompted); outputs from our 1.3B PPO-ptx model are preferred … (excerpt)
res_a6059301InstructGPT models show improvements in truthfulness over GPT-3.
res_a53e9bbaInstructGPT shows small improvements in toxicity over GPT-3, but not bias.
res_d292ceccInstructGPT models generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful.
res_63601566On our API prompt distribution, our FLAN and T0 models perform slightly worse than our SFT baseline, and labelers significantly prefer InstructGPT to these models (InstructGPT has a 73.4±2% winrate … (excerpt)
clm_ac06d8e0Overall, our results indicate that fine-tuning large language models using human preferences significantly improves their behavior on a wide range of tasks, though much work remains to be done to … (excerpt)
res_da4a417dSimilarly to Wu et al. (2021), we find that our SFT models overfit on validation loss after 1 epoch; however, we find that training for more epochs helps both the … (excerpt)
res_8cba3891This is much more computationally efficient because it only requires a single forward pass of the RM for each completion (rather than (K 2 ) forward passes forK completions) and, … (excerpt)
res_b53fc8f0We find that GPT-3 outputs perform the worst, and one can obtain significant step-size improvements by using a well-crafted few-shot prompt (GPT-3 (prompted)), then by training on demonstrations using supervised … (excerpt)
res_8a8b7d58We also found that our results do not change significantly when evaluated on prompts submitted to GPT-3 models on the API (see Figure 3), though our PPO-ptx models perform slightly … (excerpt)
res_f4425d77Compared to GPT-3, the PPO models are more appropriate in the context of a customer assistant, are better at following explicit constraints in the instruction and attempting the correct instruction, … (excerpt)
res_92ebe89aIn particular, according to held-out workers, all of our InstructGPT models still greatly outperform the GPT-3 baselines.
res_7ed241feThis indicates that these datasets are not sufficiently diverse to improve performance on our API prompt 12 distribution.
clm_910a8f15We believe our InstructGPT model outperforms FLAN and T0 for two reasons.
res_dcbc5616As measured by human evaluatoins on the TruthfulQA dataset, our PPO models show small but significant improvements in generating truthful and informative outputs compared to GPT-3 (see Figure 6).
res_d8aa246fInterestingly, the exception is our 1.3B PPO-ptx model, which performs slightly worse than a GPT-3 model of the same size.
res_4aaca356When evaluated only on prompts that were not adversarially selected against GPT-3, our PPO models are still significantly more truthful and informative than GPT-3 (although the absolute improvement decreases by … (excerpt)
res_001a7f4fOur improvements in truthfulness are also evidenced by the fact that our PPO models hallucinate (i.e. fabricate information) less often on closed-domain tasks from our API distribution, which we’ve shown … (excerpt)
res_9fde3b51To summarize: all of our models are rated as less toxic than expected given the prompt (they get a negative score on a scale from -1 to 1, where 0 … (excerpt)
res_c8ccefc7In Figure 29 we show that adding pretraining updates to our PPO fine-tuning (PPO-ptx) mitigates these performance regressions on all datasets, and even surpasses GPT-3 on HellaSwag.
res_f3c36f7eTo give a few examples: (1) when given an instruction with a false premise, the model sometimes incorrectly assumes the premise is true, (2) the model can overly hedge; when … (excerpt)
res_fdb11eb6Empirically, around 96% of our dataset (110k datapoints) is classified as English, although we estimate that the actual fraction may be 99% or higher, due to classifier inaccuracies.
res_3fb173bfThat is, if Output A and Output B are both plausible (i.e. would get a continuity score of 2), and Output A is a bit more plausible but also a … (excerpt)
res_23c47671Larger 175B RMs had the potential to achieve lower validation loss, but (1) their training was more unstable which made them less suitable for use as initializations for the PPO … (excerpt)
res_69092657Label: [positive / negative] Dataset features • 872 binary multiple choice sentiment analysis questions • In the few-shot setting, there are 15 additional text / label pairs.
res_c346de3cInstructGPT doesn’t significantly improve over GPT-3 on these datasets.
res_d9e2eefcAs can be seen, on DROP and SquadV2, the model starts out with better performance than the GPT-3 model.
res_8f5f2862In the RealToxicityPrompts task, we measure toxicity via the Perspective API and find that the toxicity of our model outputs is highly correlated with the toxicity of the input prompt, … (excerpt)
res_ee53a70fIn order to better capture our models’ behavior in unsafe regimes, we draw 5000 examples from the RealToxicityPrompts dataset with an approximately uniform distribution over prompt toxicity and report average … (excerpt)
res_21677822Some preliminary experiments show better human Likert scores can be achieved with a pretraining data ratio of 32.
res_b14f39d3InstructGPT 175B completion: - One bullet point of evidence of a positive trend in the employment market could be that recent job gains have been greater than in prior months, … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2203.02155) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.