clm_f42aa174Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models.
arXiv 2307.09288 · cs.CL cs.AI
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_f42aa174Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models.
clm_433d23fdWe also share novel observations we made during the development ofLlama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge.
res_434a3cb9We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023).
res_4a1c3fa3Specifically, we performed more robust data cleaning, updated our data mixes, trained on 40% more total tokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability for our larger models.
res_6d57d2f2We trained on 2 trillion tokens of data as this provides a good performance–cost trade-off, up-sampling the most factual sources in an effort to increase knowledge and dampen hallucinations.
res_cc196905Bigger models — 34B and 70B — use Grouped-Query Attention (GQA) for improved inference scalability.
res_17e3f85bAs shown in Table 3,Llama 2 models outperformLlama 1 models.
res_2d6e7b88Forthe Falcon models,Llama 2 7B and 34B outperform Falcon 7B and 40B models on all categories of benchmarks.
res_f069f860Additionally,Llama 2 70B model outperforms all open-source models.
clm_9634eb65We also share a new technique, Ghost Attention (GAtt), which we find helps control dialogue flow over multiple turns (Section 3.3).
res_505e85dbAs we collected more preference data, our reward models improved, and we were able to train progressively better versions forLlama 2-Chat (see the results in Section 5, Figure 20).Llama 2-Chat … (excerpt)
res_28d3e1a7Leveraging such response scores as rewards, we can optimizeLlama 2-Chat during RLHF for better human preference alignment and improved helpfulness and safety.
clm_4fa9c7d4Given that our preference ratings is decomposed as a scale of four points (e.g., significantly better), as presented in Section 3.2.1, it can be useful to leverage this information to … (excerpt)
res_5834dc54Thus, we have decided to keep them in our data mixture, as they could enable better generalization for the reward model and prevent reward hacking, i.e.Llama 2-Chat taking advantage of … (excerpt)
clm_96ec3946The reward models show superior accuracy on more distinct responses (e.g., significantly better) and lower accuracy on similar responses (e.g., negligibly better).
res_c6723ca2Overall, our reward models outperform all of the baselines, including GPT-4.
res_5c6fb38bWhen we group the scores by preference rating in Table 8, we can see that the accuracy is superior for the “significantlybetter”testsetanddegradesgraduallyascomparisonpairsbecomemoresimilar(e.g.,“slightly better”).
res_b775ea48Therefore, everything else being equal, an improvement of the reward model can be directly translated into an improvement forLlama 2-Chat.
res_f1b459c6However, despite continuous improvement, this method led to a 14 regression in some capabilities.
clm_d9a2a231As was observed in other works (Stiennon et al., 2020; Ouyang et al., 2022), we find this constraint is useful for training stability, and to reduce reward hackingwherebywewouldachievehighscoresfromtherewardmodelbutlowscoresfromhumanevaluation.
res_9d79f9a8We report a quantitative analysis indicating that GAtt is consistent up to 20+ turns, until the maximum context length is reached (see Appendix A.3.5).
res_a4f6686cHow Far Can Model-Based Evaluation Go?To measure the robustness of our reward model, we collected a test set of prompts for both helpfulness and safety, and asked three annotators to … (excerpt)
res_37b51dc3On this set of evaluations, we outperform ChatGPT on both axes after RLHF-V3 (harmlessness and helpfulness >50%).
res_734645afAs shown in Figure 12,Llama 2-Chat models outperform open-source models by a significant margin on both single turn and multi-turn prompts.
res_b4a6405eParticularly,Llama 2-Chat 7B model outperforms MPT-7B-chat on 60% of the prompts.Llama 2-Chat 34B has an overall win rate of more than 75% against equivalently sized Vicuna-33B and Falcon 40B models.
res_3e7970a5The largestLlama 2-Chat model is competitive with ChatGPT.Llama 2-Chat 70B model has a win rate of 36% and a tie rate of 31.5% relative to ChatGPT.Llama 2-Chat 70B model outperforms … (excerpt)
res_1617d565High IRR scores (closer to 1.0) are typically seen as better from a data quality perspective, however, context is important.
res_7260d179We see scores on the lower end of that range for ratings from model comparisons with similar win rates to each other (like theLlama 2-Chat-70B-chat vs. ChatGPT comparison).
res_11df126fWe see scores on the higher end of that range for ratings from model comparisons with a more clear winner (like theLlama 2-Chat-34b-chat vs. Falcon-40b-instruct).
res_95d56051The HateBERT classifier assigns a toxicity likelihood of 0.5 or higher to about 0.2% of documents in our pretraining corpus.
res_d5bcbc47About 0.2% of documents evaluated are assigned a likelihood score of 0.5 or higher, meaning there is a small amount of toxicity in our pretraining data.
clm_57fc70bcFor TruthfulQA, we present the percentage of generations that are both truthful and informative (the higher, the better).
clm_98929b4aFor ToxiGen, we present the percentage of generations that are deemed toxic by the metric (the lower, the better).
res_e0907280Llama 2 does not outperform other models on toxicity metrics, and we speculate that this may be because we refrained from aggressively filtering the pretraining data.
clm_e314ff61For ToxiGen, we present the percentage of toxic generations (the smaller, the better).
res_3f56eecdThe clustering of samples in the top left corner suggests the improvements of model safety.Right: helpfulness reward model scores of generations on the Meta Helpfulness test set. ➤Prompt: Write a … (excerpt)
res_3240cff7We see lower IRR scoresonbatcheswherethemodelshaveahighviolationrate(e.g.,Vicuna)andhigherIRRscoresonbatches where the models have relatively low violation rates (e.g.,Llama 2-Chat, Falcon, and ChatGPT).
res_a5165e56As a result, we note that in Figure 17b the average rating of Falcon is much lower thanLlama 2-Chat(34B) although their violation percentages look similar (3.88 vs 4.45).
res_c90c2196For the other two categories,Llama 2-Chat achieves comparable or lower violation percentage consistently regardless of model sizes.
res_e0a4065aTruthfulness, Toxicity, and Bias.In Table 14, fine-tunedLlama 2-Chat shows great improvement over the pretrainedLlama 2 in terms of truthfulness (50.18 → 64.14 for 70B) and toxicity (24.60 → 0.01 for … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2307.09288) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.