MearcAll papers

arXiv 1907.11692 · cs.CL

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, and 4 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_d982bf49
Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c25d115f
When trained for longer over additional data, our model achieves a score of 88.5 on the public GLUE leaderboard, matching the 88.4 reported by Y ang et al. (2019).

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2badf6e1
Our model establishes a new state-of-the-art on 4/9 of the GLUE tasks: MNLI, QNLI, RTE and STS-B.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d41a7e3c
We also match state-of-the-art results on SQuAD and RACE.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_427f39e5
Overall, we re-establish that BERT’s masked language model training objective is competitive with other recently proposed training objectives such as perturbed autoregressive language modeling (Y ang et al., 2019).2 In … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d316b39
Similarly, we found setting β2 = 0.98 to improve stability when training with large batch sizes.

Implementation · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3ba8c714
Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of … (excerpt)

Model Input Format and Next Sentence · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_12e5bb84
We find that this setting outperforms the originally published BERTBASE results and that removing the NSP loss matches or slightly improves downstream task performance, in contrast to Devlin et al. … (excerpt)

Model Input Format and Next Sentence · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2134c869
8Large batch training can improve training efficiency even without large scale parallel hardware through gradient accumulation, whereby gradients from multiple mini-batches are accumulated locally before each optimization step.

Text Encoding · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d97d83de
In the previous section we propose modifications to the BERT pretraining procedure that improve end-task performance.

RoBERTa · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7fed2b34
When controlling for training data, we observe that RoBERTa provides a large improvement over the originally reported BERTLARGE results, reaffirming the importance of the design choices we explored in Section … (excerpt)

RoBERTa · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_570566c0
We again observe significant gains in downstream task performance, and the 300K and 500K step models outperform XLNet LARGE across most tasks.

RoBERTa · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c87a117a
In the first setting ( single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets.

GLUE Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_48f8e32a
Crucially, RoBERTa uses the same masked language modeling pretraining objective and architecture as BERT LARGE , yet consistently outperforms both BERT LARGE and XLNetLARGE .

GLUE Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_65bca05b
In the second setting ( ensembles, test ), we submit RoBERTa to the GLUE leaderboard and achieve state-of-the-art results on 4 out of 9 tasks and the highest average score … (excerpt)

GLUE Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cdcfeae3
On the SQuAD v1.1 development set, RoBERTa matches the state-of-the-art set by XLNet.

SQuAD Results · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2e3d7948
On the SQuAD v2.0 development set, RoBERTa sets a new state-of-the-art, improving over XLNet by 0.4 points (EM) and 0.6 points (F1).

SQuAD Results · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1f1af971
Our single RoBERTa model outperforms all but one of the single model submissions, and is the top scoring system among those that do not rely on data augmentation.

SQuAD Results · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4ed44695
RoBERTa achieves state-of-the-art results on both middle-school and high-school settings.

RACE Results · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aed5257e
We find that performance can be substantially improved by training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and … (excerpt)

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5c87b07d
Our improved pretraining procedure, which we call RoBERTa, achieves state-of-the-art results on GLUE, RACE and SQuAD, without multi-task finetuning for GLUE or additional data for SQuAD.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0cbd0287
We additionally use a novel dataset, CC-N EWS , and release our models and code for pretraining and finetuning at: https://github.com/pytorch/fairseq.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1907.11692) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.