MearcAll papers

arXiv 2307.09288 · cs.CL cs.AI

Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, and 62 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_f42aa174
Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_433d23fd
We also share novel observations we made during the development ofLlama 2 and Llama 2-Chat, such as the emergence of tool usage and temporal organization of knowledge.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_434a3cb9
We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023).

Introduction · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4a1c3fa3
Specifically, we performed more robust data cleaning, updated our data mixes, trained on 40% more total tokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability for our larger models.

Pretraining · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6d57d2f2
We trained on 2 trillion tokens of data as this provides a good performance–cost trade-off, up-sampling the most factual sources in an effort to increase knowledge and dampen hallucinations.

Pretraining Data · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cc196905
Bigger models — 34B and 70B — use Grouped-Query Attention (GQA) for improved inference scalability.

Training Details · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_17e3f85b
As shown in Table 3,Llama 2 models outperformLlama 1 models.

Llama 2 Pretrained Model Evaluation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2d6e7b88
Forthe Falcon models,Llama 2 7B and 34B outperform Falcon 7B and 40B models on all categories of benchmarks.

Llama 2 Pretrained Model Evaluation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f069f860
Additionally,Llama 2 70B model outperforms all open-source models.

Llama 2 Pretrained Model Evaluation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9634eb65
We also share a new technique, Ghost Attention (GAtt), which we find helps control dialogue flow over multiple turns (Section 3.3).

Fine-tuning · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_505e85db
As we collected more preference data, our reward models improved, and we were able to train progressively better versions forLlama 2-Chat (see the results in Section 5, Figure 20).Llama 2-Chat … (excerpt)

Human Preference Data Collection · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_28d3e1a7
Leveraging such response scores as rewards, we can optimizeLlama 2-Chat during RLHF for better human preference alignment and improved helpfulness and safety.

Reward Modeling · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4fa9c7d4
Given that our preference ratings is decomposed as a scale of four points (e.g., significantly better), as presented in Section 3.2.1, it can be useful to leverage this information to … (excerpt)

Reward Modeling · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5834dc54
Thus, we have decided to keep them in our data mixture, as they could enable better generalization for the reward model and prevent reward hacking, i.e.Llama 2-Chat taking advantage of … (excerpt)

Reward Modeling · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_96ec3946
The reward models show superior accuracy on more distinct responses (e.g., significantly better) and lower accuracy on similar responses (e.g., negligibly better).

Reward Modeling · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c6723ca2
Overall, our reward models outperform all of the baselines, including GPT-4.

Reward Modeling · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5c6fb38b
When we group the scores by preference rating in Table 8, we can see that the accuracy is superior for the “significantlybetter”testsetanddegradesgraduallyascomparisonpairsbecomemoresimilar(e.g.,“slightly better”).

Reward Modeling · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b775ea48
Therefore, everything else being equal, an improvement of the reward model can be directly translated into an improvement forLlama 2-Chat.

Reward Modeling · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f1b459c6
However, despite continuous improvement, this method led to a 14 regression in some capabilities.

Iterative Fine-Tuning · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d9a2a231
As was observed in other works (Stiennon et al., 2020; Ouyang et al., 2022), we find this constraint is useful for training stability, and to reduce reward hackingwherebywewouldachievehighscoresfromtherewardmodelbutlowscoresfromhumanevaluation.

Iterative Fine-Tuning · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d79f9a8
We report a quantitative analysis indicating that GAtt is consistent up to 20+ turns, until the maximum context length is reached (see Appendix A.3.5).

System Message for Multi-Turn Consistency · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a4f6686c
How Far Can Model-Based Evaluation Go?To measure the robustness of our reward model, we collected a test set of prompts for both helpfulness and safety, and asked three annotators to … (excerpt)

Model-Based Evaluation · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_37b51dc3
On this set of evaluations, we outperform ChatGPT on both axes after RLHF-V3 (harmlessness and helpfulness >50%).

Model-Based Evaluation · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_734645af
As shown in Figure 12,Llama 2-Chat models outperform open-source models by a significant margin on both single turn and multi-turn prompts.

Human Evaluation · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b4a6405e
Particularly,Llama 2-Chat 7B model outperforms MPT-7B-chat on 60% of the prompts.Llama 2-Chat 34B has an overall win rate of more than 75% against equivalently sized Vicuna-33B and Falcon 40B models.

Human Evaluation · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3e7970a5
The largestLlama 2-Chat model is competitive with ChatGPT.Llama 2-Chat 70B model has a win rate of 36% and a tie rate of 31.5% relative to ChatGPT.Llama 2-Chat 70B model outperforms … (excerpt)

Human Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1617d565
High IRR scores (closer to 1.0) are typically seen as better from a data quality perspective, however, context is important.

Human Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7260d179
We see scores on the lower end of that range for ratings from model comparisons with similar win rates to each other (like theLlama 2-Chat-70B-chat vs. ChatGPT comparison).

Human Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_11df126f
We see scores on the higher end of that range for ratings from model comparisons with a more clear winner (like theLlama 2-Chat-34b-chat vs. Falcon-40b-instruct).

Human Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_95d56051
The HateBERT classifier assigns a toxicity likelihood of 0.5 or higher to about 0.2% of documents in our pretraining corpus.

Safety in Pretraining · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d5bcbc47
About 0.2% of documents evaluated are assigned a likelihood score of 0.5 or higher, meaning there is a small amount of toxicity in our pretraining data.

Safety in Pretraining · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_57fc70bc
For TruthfulQA, we present the percentage of generations that are both truthful and informative (the higher, the better).

Safety in Pretraining · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_98929b4a
For ToxiGen, we present the percentage of generations that are deemed toxic by the metric (the lower, the better).

Safety in Pretraining · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e0907280
Llama 2 does not outperform other models on toxicity metrics, and we speculate that this may be because we refrained from aggressively filtering the pretraining data.

Safety in Pretraining · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e314ff61
For ToxiGen, we present the percentage of toxic generations (the smaller, the better).

Safety in Pretraining · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3f56eecd
The clustering of samples in the top left corner suggests the improvements of model safety.Right: helpfulness reward model scores of generations on the Meta Helpfulness test set. ➤Prompt: Write a … (excerpt)

Safety RLHF · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3240cff7
We see lower IRR scoresonbatcheswherethemodelshaveahighviolationrate(e.g.,Vicuna)andhigherIRRscoresonbatches where the models have relatively low violation rates (e.g.,Llama 2-Chat, Falcon, and ChatGPT).

Safety Evaluation of Llama 2-Chat · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a5165e56
As a result, we note that in Figure 17b the average rating of Falcon is much lower thanLlama 2-Chat(34B) although their violation percentages look similar (3.88 vs 4.45).

Safety Evaluation of Llama 2-Chat · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c90c2196
For the other two categories,Llama 2-Chat achieves comparable or lower violation percentage consistently regardless of model sizes.

Safety Evaluation of Llama 2-Chat · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e0a4065a
Truthfulness, Toxicity, and Bias.In Table 14, fine-tunedLlama 2-Chat shows great improvement over the pretrainedLlama 2 in terms of truthfulness (50.18 → 64.14 for 70B) and toxicity (24.60 → 0.01 for … (excerpt)

Safety Evaluation of Llama 2-Chat · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2307.09288) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.