clm_6051b47dWe present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
arXiv 2211.09110 · cs.CL cs.AI cs.LG
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
clm_6051b47dWe present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
res_976d3cf4Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) foreach of 16 core scenarios to the extent possible (87.5% of the time), ensuring that metrics beyond accuracy don’t fall to the wayside, and that trade-offs across models and metrics are clearly exposed.
res_72d2a2ebWe improve this to 96.0%: now all 30 models have been densely benchmarked on a set of core scenarios and metrics under standardized conditions.
res_5e64c772And our 7 categories of metrics reflect a range of societal considerations (i.e. accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).
clm_68524292And even for datasets that are frequently evaluated for across all 405 datasets evaluated in major language modeling works (e.g.HellaSwag; many of the datasets within GLUE and SuperGLUE), we find the evaluation conditions vary greatly.
res_fc094676We improve this to 96.0%.5 By both evaluating these models on the same scenariosand by conducting the evaluation under standardized conditions (e.g. using the same few-shot prompting for all models), we facilitate direct head-to-head comparisons.
res_a215d585Across the core scenarios, we find that text-davinci-002 performs best on our accuracy, robustness, and fairness metrics, with Anthropic-LM v4-s3 (52B) being in the top 3 for all 3 metrics (despite being more than 10× smaller in model scale compared to TNLG v2 (530B), which is the second most accurate and fair) as shown in Figure 26.
res_c417c461As an example, forHellaSwag,6 improving accuracy worsens calibration, whereas forOpenBookQA,7 improving accuracy improves calibration.
res_f812f889Across all scenarios, we observe strong correlations betweenaccuracy, robustness, andfairness, whererobustnessandfairnessmetricsconsiderworst-case accuracy over a set of perturbations (e.g. typos for robustness, dialect alteration for fairness)—see §4.5: metrics-robustness, §4.6: metrics-fairness for more details.
res_826d7371While there is a strong correlation between accuracy and fairness (Figure 24, Figure 25), we do observe trade-offs where the most accurate model is not the most robust or most fair.
res_cbb24653We also see serious drops in some cases: for example, onNarrativeQA, TNLG v2 (530B) precipitously drops from 72.6% standard accuracy (i.e. the third-most accurate model) to 38.9% accuracy in the presence of robustness perturbations.8 5.
res_05386e00As an example of racialized dialect disparities, OPT (175B) is the most accurate model onTwitterAAEbut its accuracy degrades from 1.506 bits per byte for White English to 2.114 bits per byte for African American English (lower is better).9 6.
res_135df682For each family of models (e.g. different size variants of GPT-3), we find that as models become larger, accuracy consistently improves but with higher training and inference cost.10 Overall, we observe that only a subset of all models (across model families) are on the accuracy-efficiency Pareto frontier for each scenario.
res_26340e90Across the 9 core question answering scenarios (§3.3:questionAnswering), we observe significant heterogeneity in results, though text-davinci-002 is the most accurate model for all 9 scenarios.11 In fact, for 6 of the 9 scenarios, there is no open model among the three most accurate models, as generally they are text-davinci-002, Anthropic-LM v4-s3 (52B), and TNLG v2 (530B) in descending order of accuracy.
clm_9ee1e2eeThe best-performing models we evaluate outperform classical retrieval methods and under some settings perform comparably to various fine-tuned neural retrievers, while nonetheless trailing the state of the art.12 Because the number of candidates could be large, we create a LM request per passage, which requires the model to produce calibrated probabilities.
res_8a13ca067 Published in Transactions on Machine Learning Research (08/2023) (Figure 26), but achieves essentially chance accuracy at 50.1%.14 Critically, given the importance of fairness in toxicity detection due the disparate impacts of content moderation, we find that most models are similarly accurate for detecting toxicity in comments mentioning Black and White individuals.
res_8080c611However, models vary greatly in their robustness: OPT (175B) drops from 51.3% standard accuracy to 8.8% robust accuracy on the Black split, whereas the drop is less precipitous on the White split (50.8% to 24.3%).
res_843590f3For text classification onRAFT, we see significant heterogeneity in which models do well on which subsets/tasks.15 text-davinci-002 is consistently accurate across splits when compared with other models, but performs very poorly on the Systematic Review Inclusion split with an accuracy of 40.8% compared to 97.5% from several models (e.g. GLM (130B)).
res_7f2a07d9In particular, GPT-NeoX (20B), OPT (175B), BLOOM (176B), GPT-J (6B), and OPT (66B) consistently have the lowest bits-per-byte (lower is better) onThe Pile,TwitterAAE, andICE.
res_17731a5aKnowledge. text-davinci-002 demonstrates superior performance for all knowledge-intensive evaluations,18 with a very sizable gap for accuracy onTruthfulQAof 62.0% compared to second place of 36.2% from Anthropic-LM v4-s3 (52B).19 Further, TNLG v2 (530B) shows strong performance on the highly knowledge-intensiveNaturalQuestions (closed-book) and WikiFact scenarios, which generally concurs with the hypothesis that model scale especially contributes to improvements in acquisition of factual knowledge.
res_a25cf3f9Forreasoning-intensivescenarios, wefindthatthecodemodels, especiallycode-davinci- 002, consistently outperform the text models, even on synthetic reasoning scenarios posed in natural language.20 This gap is made clear in mathematical reasoning: for GSM8K, code-davinci-002 achieves an accuracy of 52.1%, where the next best model is text-davinci-002 at 35.0% and no other model surpasses 16%.21 Further, in addition to code-davinci-002, text-davinci-002 is much more accurate than other text models (e.g. 65.1% accuracy on synthetic reasoning in natural language, whereas the next most accurate text model is OPT (175B) at 29.4% accuracy, and code-davinci-002 has an accuracy of 72.7%).
res_5891b283Memorization of copyrighted/licensed material.We find that the likelihood of direct regurgitation of long copyrighted sequences is somewhat uncommon, but it does become noticeable when looking at popular books.22 However, we do find the regurgitation risk clearly correlates with model accuracy: text-davinci-002, davinci (175B), and Anthropic-LM v4-s3 (52B) demonstrate the highest amount of verbatim regurgitation in line with their high accuracies.
clm_278aa12aIn other words, we find that forBBQ the most accurate models are precisely those that are most concerning for social biases in ambiguous contexts, though the trends in disambiguated contexts are less clear.
res_587aeff9As an example, we find text-davinci-002 achieves an accuracy of 74.4%ROUGE-L on NarrativeQA, which sets a new state-of-the-art across all methods to our knowledge, in this case over the strong QA-specialized UnifiedQA-v2 model (67.4%ROUGE-L; Khashabi et al., 2022).
res_26ac612eAll models show significant sensitivity to the formatting of prompt, the particular choice of in-context examples, and the number of in-context examples across all scenarios and for all metrics (see §8.2:prompting-analysis).
res_7d5ce9d7Unfortunately, when making these comparisons across model families, even when using bits-per-byte (BPB; which is more comparable than perplexity), we find this type of prediction does not work well: BPB onThe Pileis a poor predictor of downstream accuracy (Figure 30) though we note some models are trained onThe Pile whereas others are not (Table 13).
res_5d3b1c3dWe find that model scale, within a model family, reliably predicts model accuracy, but for no scenario is a good predictor of downstream accuracy across all models (Figure 29).
res_baba8ba8Overall, scale seems to be a key determinant of accuracy, and scaling within a model family reliable improves accuracy, but it might be inefficient compared to other means (e.g. training with human feedback; compare TNLG v2 (530B) and Anthropic-LM v4-s3 (52B)).
res_9e677c37Given our taxonomy, we select and implement 16 core scenarios, for which we comprehensively measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).
res_2fa05e73The metrics we compute for our running example (Figure 6) might look like this: Exact match : 0.571 ECE (10-bin) : 0.221 Exact match (robustness) : 0.551 Exact match (fairness) : 0.524 Inference runtime : 0.147 ...
clm_67b881cdHowever, as we demonstrate in our taxonomy, both tasks and domains themselves are rich and expansive spaces.
res_362673a5Theformatofquestionansweringmayhavesomevariations: in the open-bookor reading comprehensionsetting, additional context to refer to, such as supporting documents 33We do note that various works have shown these models can achieve nontrivial performance on multimodal tasks (with modalities beyond text) and on other languages (especially as some of these models, most notably BLOOM (176B), GLM (130B), and YaLM (100B) are trained on sizable datasets in other languages).
res_08baf970Of the available sentiment analysis datasets, we selected theIMDB dataset (Maas et al., 2011), as it had the unique resource of a contrast set (Gardner et al., 2020), which enables the measurement of robustness to equivariances (which we found difficult to measure otherwise).IMDB is constructed from IMDB movie reviews, where users rate movies from 1–10.
res_06fcf1a5These ratings are discretized to a binary space, with reviews with a score at most 4 being labeled negative and reviews with a score at least 7 being labeled positive.
res_62bf1bd1Beyond what we end up measuring, we suggest a prioritized set of areas for improvement in §10.2:missing-metrics.
clm_3c9c84dcHowever, we emphasize that the other forms of robustness are important, but we find that they are comparatively more difficult to measure because of the lack of assumptions we make on the models we evaluate as well as the scale of our evaluation.
res_030d725fModel generation for example 2: The mathematician was recognized for his outstanding achievements with the Fields Medal.
res_88f6824cDemographic representation Occurrence (male terms) = 5 Occurrence (female terms) = 2 Gender representation bias = 0.5 * [|2/7 - 0.5| + 0.5 * |5/7 - 0.5|] = 0.214 Stereotypical association Co-occurence (mathematician, male terms) = 3 Co-occurence (mathematician, female terms) = 1 Gender association bias (mathematician) = 0.5 * [|1/4 - 0.5| + 0.5 * |3/4 - 0.5|] = 0.25 Model generation: The mathematician was recognized for his outstanding achievements with the Fields Medal.
res_21ac91b9The inputs have 64 tokens per instance on average, with significant variance in input lengths, with some reaching around hundreds of tokens.
clm_bfebd248For legal reasoning, we constructLegalSupport, a novel comparative legal entailment task.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2211.09110) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.