MearcAll papers

arXiv 2211.09110 · cs.CL cs.AI cs.LG

Holistic Evaluation of Language Models

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, and 44 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_6051b47d
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_976d3cf4
Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) foreach of 16 core scenarios to the extent possible (87.5% of the time), ensuring that metrics beyond accuracy don’t fall to the wayside, and that trade-offs across models and metrics are clearly exposed.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_72d2a2eb
We improve this to 96.0%: now all 30 models have been densely benchmarked on a set of core scenarios and metrics under standardized conditions.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5e64c772
And our 7 categories of metrics reflect a range of societal considerations (i.e. accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).

HELM · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_68524292
And even for datasets that are frequently evaluated for across all 405 datasets evaluated in major language modeling works (e.g.HellaSwag; many of the datasets within GLUE and SuperGLUE), we find the evaluation conditions vary greatly.

HELM · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fc094676
We improve this to 96.0%.5 By both evaluating these models on the same scenariosand by conducting the evaluation under standardized conditions (e.g. using the same few-shot prompting for all models), we facilitate direct head-to-head comparisons.

HELM · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a215d585
Across the core scenarios, we find that text-davinci-002 performs best on our accuracy, robustness, and fairness metrics, with Anthropic-LM v4-s3 (52B) being in the top 3 for all 3 metrics (despite being more than 10× smaller in model scale compared to TNLG v2 (530B), which is the second most accurate and fair) as shown in Figure 26.

Empirical findings · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c417c461
As an example, forHellaSwag,6 improving accuracy worsens calibration, whereas forOpenBookQA,7 improving accuracy improves calibration.

Empirical findings · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f812f889
Across all scenarios, we observe strong correlations betweenaccuracy, robustness, andfairness, whererobustnessandfairnessmetricsconsiderworst-case accuracy over a set of perturbations (e.g. typos for robustness, dialect alteration for fairness)—see §4.5: metrics-robustness, §4.6: metrics-fairness for more details.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_826d7371
While there is a strong correlation between accuracy and fairness (Figure 24, Figure 25), we do observe trade-offs where the most accurate model is not the most robust or most fair.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cbb24653
We also see serious drops in some cases: for example, onNarrativeQA, TNLG v2 (530B) precipitously drops from 72.6% standard accuracy (i.e. the third-most accurate model) to 38.9% accuracy in the presence of robustness perturbations.8 5.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_05386e00
As an example of racialized dialect disparities, OPT (175B) is the most accurate model onTwitterAAEbut its accuracy degrades from 1.506 bits per byte for White English to 2.114 bits per byte for African American English (lower is better).9 6.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_135df682
For each family of models (e.g. different size variants of GPT-3), we find that as models become larger, accuracy consistently improves but with higher training and inference cost.10 Overall, we observe that only a subset of all models (across model families) are on the accuracy-efficiency Pareto frontier for each scenario.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_26340e90
Across the 9 core question answering scenarios (§3.3:questionAnswering), we observe significant heterogeneity in results, though text-davinci-002 is the most accurate model for all 9 scenarios.11 In fact, for 6 of the 9 scenarios, there is no open model among the three most accurate models, as generally they are text-davinci-002, Anthropic-LM v4-s3 (52B), and TNLG v2 (530B) in descending order of accuracy.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9ee1e2ee
The best-performing models we evaluate outperform classical retrieval methods and under some settings perform comparably to various fine-tuned neural retrievers, while nonetheless trailing the state of the art.12 Because the number of candidates could be large, we create a LM request per passage, which requires the model to produce calibrated probabilities.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8a13ca06
7 Published in Transactions on Machine Learning Research (08/2023) (Figure 26), but achieves essentially chance accuracy at 50.1%.14 Critically, given the importance of fairness in toxicity detection due the disparate impacts of content moderation, we find that most models are similarly accurate for detecting toxicity in comments mentioning Black and White individuals.

Empirical findings · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8080c611
However, models vary greatly in their robustness: OPT (175B) drops from 51.3% standard accuracy to 8.8% robust accuracy on the Black split, whereas the drop is less precipitous on the White split (50.8% to 24.3%).

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_843590f3
For text classification onRAFT, we see significant heterogeneity in which models do well on which subsets/tasks.15 text-davinci-002 is consistently accurate across splits when compared with other models, but performs very poorly on the Systematic Review Inclusion split with an accuracy of 40.8% compared to 97.5% from several models (e.g. GLM (130B)).

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7f2a07d9
In particular, GPT-NeoX (20B), OPT (175B), BLOOM (176B), GPT-J (6B), and OPT (66B) consistently have the lowest bits-per-byte (lower is better) onThe Pile,TwitterAAE, andICE.

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_17731a5a
Knowledge. text-davinci-002 demonstrates superior performance for all knowledge-intensive evaluations,18 with a very sizable gap for accuracy onTruthfulQAof 62.0% compared to second place of 36.2% from Anthropic-LM v4-s3 (52B).19 Further, TNLG v2 (530B) shows strong performance on the highly knowledge-intensiveNaturalQuestions (closed-book) and WikiFact scenarios, which generally concurs with the hypothesis that model scale especially contributes to improvements in acquisition of factual knowledge.

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a25cf3f9
Forreasoning-intensivescenarios, wefindthatthecodemodels, especiallycode-davinci- 002, consistently outperform the text models, even on synthetic reasoning scenarios posed in natural language.20 This gap is made clear in mathematical reasoning: for GSM8K, code-davinci-002 achieves an accuracy of 52.1%, where the next best model is text-davinci-002 at 35.0% and no other model surpasses 16%.21 Further, in addition to code-davinci-002, text-davinci-002 is much more accurate than other text models (e.g. 65.1% accuracy on synthetic reasoning in natural language, whereas the next most accurate text model is OPT (175B) at 29.4% accuracy, and code-davinci-002 has an accuracy of 72.7%).

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5891b283
Memorization of copyrighted/licensed material.We find that the likelihood of direct regurgitation of long copyrighted sequences is somewhat uncommon, but it does become noticeable when looking at popular books.22 However, we do find the regurgitation risk clearly correlates with model accuracy: text-davinci-002, davinci (175B), and Anthropic-LM v4-s3 (52B) demonstrate the highest amount of verbatim regurgitation in line with their high accuracies.

Empirical findings · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_278aa12a
In other words, we find that forBBQ the most accurate models are precisely those that are most concerning for social biases in ambiguous contexts, though the trends in disambiguated contexts are less clear.

Empirical findings · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_587aeff9
As an example, we find text-davinci-002 achieves an accuracy of 74.4%ROUGE-L on NarrativeQA, which sets a new state-of-the-art across all methods to our knowledge, in this case over the strong QA-specialized UnifiedQA-v2 model (67.4%ROUGE-L; Khashabi et al., 2022).

Empirical findings · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_26ac612e
All models show significant sensitivity to the formatting of prompt, the particular choice of in-context examples, and the number of in-context examples across all scenarios and for all metrics (see §8.2:prompting-analysis).

Empirical findings · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7d5ce9d7
Unfortunately, when making these comparisons across model families, even when using bits-per-byte (BPB; which is more comparable than perplexity), we find this type of prediction does not work well: BPB onThe Pileis a poor predictor of downstream accuracy (Figure 30) though we note some models are trained onThe Pile whereas others are not (Table 13).

Empirical findings · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5d3b1c3d
We find that model scale, within a model family, reliably predicts model accuracy, but for no scenario is a good predictor of downstream accuracy across all models (Figure 29).

Empirical findings · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_baba8ba8
Overall, scale seems to be a key determinant of accuracy, and scaling within a model family reliable improves accuracy, but it might be inefficient compared to other means (e.g. training with human feedback; compare TNLG v2 (530B) and Anthropic-LM v4-s3 (52B)).

Empirical findings · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9e677c37
Given our taxonomy, we select and implement 16 core scenarios, for which we comprehensively measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).

Contributions · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2fa05e73
The metrics we compute for our running example (Figure 6) might look like this: Exact match : 0.571 ECE (10-bin) : 0.221 Exact match (robustness) : 0.551 Exact match (fairness) : 0.524 Inference runtime : 0.147 ...

Metrics · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_67b881cd
However, as we demonstrate in our taxonomy, both tasks and domains themselves are rich and expansive spaces.

Selection · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_362673a5
Theformatofquestionansweringmayhavesomevariations: in the open-bookor reading comprehensionsetting, additional context to refer to, such as supporting documents 33We do note that various works have shown these models can achieve nontrivial performance on multimodal tasks (with modalities beyond text) and on other languages (especially as some of these models, most notably BLOOM (176B), GLM (130B), and YaLM (100B) are trained on sizable datasets in other languages).

Question answering · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_08baf970
Of the available sentiment analysis datasets, we selected theIMDB dataset (Maas et al., 2011), as it had the unique resource of a contrast set (Gardner et al., 2020), which enables the measurement of robustness to equivariances (which we found difficult to measure otherwise).IMDB is constructed from IMDB movie reviews, where users rate movies from 1–10.

Sentiment analysis · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_06fcf1a5
These ratings are discretized to a binary space, with reviews with a score at most 4 being labeled negative and reviews with a score at least 7 being labeled positive.

Sentiment analysis · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_62bf1bd1
Beyond what we end up measuring, we suggest a prioritized set of areas for improvement in §10.2:missing-metrics.

Selection · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3c9c84dc
However, we emphasize that the other forms of robustness are important, but we find that they are comparatively more difficult to measure because of the lack of assumptions we make on the models we evaluate as well as the scale of our evaluation.

Robustness · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_030d725f
Model generation for example 2: The mathematician was recognized for his outstanding achievements with the Fields Medal.

Fairness · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_88f6824c
Demographic representation Occurrence (male terms) = 5 Occurrence (female terms) = 2 Gender representation bias = 0.5 * [|2/7 - 0.5| + 0.5 * |5/7 - 0.5|] = 0.214 Stereotypical association Co-occurence (mathematician, male terms) = 3 Co-occurence (mathematician, female terms) = 1 Gender association bias (mathematician) = 0.5 * [|1/4 - 0.5| + 0.5 * |3/4 - 0.5|] = 0.25 Model generation: The mathematician was recognized for his outstanding achievements with the Fields Medal.

Fairness · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21ac91b9
The inputs have 64 tokens per instance on average, with significant variance in input lengths, with some reaching around hundreds of tokens.

Reasoning primitives · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bfebd248
For legal reasoning, we constructLegalSupport, a novel comparative legal entailment task.

Realistic reasoning · page 39 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2211.09110) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.