clm_d982bf49Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD.
arXiv 1907.11692 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_d982bf49Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD.
res_c25d115fWhen trained for longer over additional data, our model achieves a score of 88.5 on the public GLUE leaderboard, matching the 88.4 reported by Y ang et al. (2019).
clm_2badf6e1Our model establishes a new state-of-the-art on 4/9 of the GLUE tasks: MNLI, QNLI, RTE and STS-B.
clm_d41a7e3cWe also match state-of-the-art results on SQuAD and RACE.
res_427f39e5Overall, we re-establish that BERT’s masked language model training objective is competitive with other recently proposed training objectives such as perturbed autoregressive language modeling (Y ang et al., 2019).2 In … (excerpt)
res_8d316b39Similarly, we found setting β2 = 0.98 to improve stability when training with large batch sizes.
res_3ba8c714Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of … (excerpt)
clm_12e5bb84We find that this setting outperforms the originally published BERTBASE results and that removing the NSP loss matches or slightly improves downstream task performance, in contrast to Devlin et al. … (excerpt)
res_2134c8698Large batch training can improve training efficiency even without large scale parallel hardware through gradient accumulation, whereby gradients from multiple mini-batches are accumulated locally before each optimization step.
clm_d97d83deIn the previous section we propose modifications to the BERT pretraining procedure that improve end-task performance.
res_7fed2b34When controlling for training data, we observe that RoBERTa provides a large improvement over the originally reported BERTLARGE results, reaffirming the importance of the design choices we explored in Section … (excerpt)
res_570566c0We again observe significant gains in downstream task performance, and the 300K and 500K step models outperform XLNet LARGE across most tasks.
res_c87a117aIn the first setting ( single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets.
clm_48f8e32aCrucially, RoBERTa uses the same masked language modeling pretraining objective and architecture as BERT LARGE , yet consistently outperforms both BERT LARGE and XLNetLARGE .
res_65bca05bIn the second setting ( ensembles, test ), we submit RoBERTa to the GLUE leaderboard and achieve state-of-the-art results on 4 out of 9 tasks and the highest average score … (excerpt)
clm_cdcfeae3On the SQuAD v1.1 development set, RoBERTa matches the state-of-the-art set by XLNet.
res_2e3d7948On the SQuAD v2.0 development set, RoBERTa sets a new state-of-the-art, improving over XLNet by 0.4 points (EM) and 0.6 points (F1).
clm_1f1af971Our single RoBERTa model outperforms all but one of the single model submissions, and is the top scoring system among those that do not rely on data augmentation.
clm_4ed44695RoBERTa achieves state-of-the-art results on both middle-school and high-school settings.
clm_aed5257eWe find that performance can be substantially improved by training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and … (excerpt)
clm_5c87b07dOur improved pretraining procedure, which we call RoBERTa, achieves state-of-the-art results on GLUE, RACE and SQuAD, without multi-task finetuning for GLUE or additional data for SQuAD.
clm_0cbd0287We additionally use a novel dataset, CC-N EWS , and release our models and code for pretraining and finetuning at: https://github.com/pytorch/fairseq.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1907.11692) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.