MearcAll papers

arXiv 1804.07461 · cs.CL

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_61a22a8a
To better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cabe761f
We find that unified multi-task trained models slightly outperform comparable models trained on each task separately.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_829fded8
As in MRPC, the class distribution in QQP is unbalanced (63% negative), so we report both accuracy and F1 score.

S IMILARITY AND PARAPHRASE TASKS · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_064cc308
The average R3 score among the annotators is 0.80, much higher than any of the baseline systems described in Section 5.

D IAGNOSTIC DATASET · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0157e07f
We use the macro-average score as the validation metric and stop training when the learning rate drops below 10−5 or performance does not improve after 5 validation checks.

B ASELINES · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b5c87022
Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best.

B ENCHMARK RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_54ce23de
On WNLI, no model exceeds most-frequent-class guessing (65.1%) and we substitute the model predictions for the most-frequent baseline.

B ENCHMARK RESULTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2ccc10e4
Coarse Categories Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance.

A NALYSIS · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_92bcf09d
Unlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts.

A NALYSIS · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_923f6ce9
The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen.

A NALYSIS · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_709d6aeb
We confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, … (excerpt)

C ONCLUSION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad6e8f6f
We divide the learning rate by 5 whenever validation performance does not improve.

T RAINING · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fd020d68
We stop training when the learning rate drops below 10−5 or performance does not improve after 5 validation checks.

T RAINING · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1804.07461) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.