MearcAll papers

arXiv 1910.10683 · cs.LG cs.CL stat.ML

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, and 3 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_e522446c
By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_499c1466
For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics (Lin, 2004) to be highly correlated so we report the ROUGE-2-F score alone under … (excerpt)

Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ab0e59e1
Similarly, for SQuAD we find the performance of the “exact match” and “F1” scores to be highly correlated so we report the “exact match” score alone.

Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_db021675
We provide every score achieved on every task for all experiments in Table 16, Appendix E.

Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_04970c34
For example, BERTBASE achieved an exact match score of80.8 on SQuAD and an accuracy of 84.4 on MNLI-matched, whereas we achieve 80.88 and 84.24, respectively (see Table 16).

Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a925307c
Unsurprisingly, we find that pre-training provides significant gains across almost all benchmarks.

Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1ea71038
As for inter-run variance, we find that for most tasks the standard deviation across runs is smaller than1% of the task’s baseline score.

Baseline Performance · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_44e2522e
The scores achieved by each of the architectures we compare are shown in Table 2.

Results · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7f5f4fd5
We evaluate each architectural variant using a denoising objective (described in Section 3.1.4) and an autoregressive objective (as is commonly used to train language models). the encoder and decoder stacks … (excerpt)

Results · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ab205b8f
We also note that the shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial.

Results · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2520cc7f
Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks.

Disparate High-Level Approaches · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_489a8b6c
Indeed, the motivation for the BERT objective was to outperform language model-based pre-training.

Disparate High-Level Approaches · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d52d971f
We find that in our setting, all of these variants perform similarly.

Simplifying the BERT Objective · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_34f55d7c
Overall, we find that the corruption rate had a limited effect on the model’s performance.

V arying the Corruption Rate · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_70c436ad
The only exception is that the largest corruption rate we consider (50%) results in a significant degradation of performance on GLUE and SQuAD.

V arying the Corruption Rate · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0f2a20e5
Again, we find a limited difference between these objectives, though the version with an average span length of10 slightly underperforms the other values in some cases.

Corrupting Spans · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d80c7afa
We also find in particular that using an average span length of3 slightly (but significantly) outperforms the i.i.d. objective on most non-translation benchmarks.

Corrupting Spans · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e8ddd8c8
Overall, the most significant difference in performance we observed was that denoising objectives outperformed language modeling and deshuffling for pre-training.

Discussion · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_16c0b56b
The results achieved after pre-training on each of these data sets is shown in Table 8.

Unlabeled Data Sets · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d060904b
Beyond this, we found that in some cases a pre-training data set with a more constrained domain outperformed the diverse C4 data set.

Unlabeled Data Sets · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ae73f682
Pursuant with past results (Houlsby et al., 2019; Bapna et al., 2019) we find that lower-resource tasks like SQuAD work well with a small value ofd whereas higher resource tasks … (excerpt)

Fine-tuning Methods · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4925910a
In general, we find that multi-task training underperforms pre-training followed by fine-tuning on most tasks.

Multi-task Learning · page 32 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_580b8834
For examples-proportional mixing, we find that for most tasks there is a “sweet spot” forK where the model obtains the best performance, and larger or smaller values ofK tend to … (excerpt)

Multi-task Learning · page 32 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_028e6ca5
We find that fine-tuning after multi-task pre-training results in comparable performance to our baseline.

Combining Multi-Task Learning with Fine-Tuning · page 33 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7d162580
The performance achieved after applying these various scaling methods is shown in Table 13.

Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e8f848fa
In some tasks (CNN/DM, WMT English to German, and WMT English to Romanian), ensembling4 completely separately trained models significantly outperformed every other scaling approach.

Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d3ceb5c3
The only exception was SuperGLUE, where neither ensembling approach significantly improved over the baseline.

Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_298d0d0f
In Section 3.4.1, we showed that pre-training on the RealNews-like, WebText-like, and Wikipedia + TBC data sets outperformed pre-training on C4 on a few downstream tasks.

Putting It All Together · page 36 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6d7b340a
Model sizes In Section 3.6 we also showed how scaling up the baseline model size improved performance.

Putting It All Together · page 36 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ae65005d
Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set, so we can still compare to what is ostensibly the state-of-the-art.

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f13bfc0e
Overall, we achieved state-of-the-art performance on18 out of the24 tasks we consider.

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d7ffe46
Our T5-3B model variant did beat the previous state of the art in a few tasks, but scaling the model size to11 billion parameters was the most important ingredient for … (excerpt)

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_72ee6600
We achieved a state-of-the-art average GLUE score of90.3.

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9c1ed497
Notably, our performance was substantially better than the previous state-of-the-art for the natural language inference tasks MNLI, RTE, and WNLI.

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bf1d01c8
For SQuAD, we outperformed the previous state-of-the-art (ALBERT (Lan et al., 2019)) by over one point on the Exact Match score.

Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_51d40878
All results are reported on the test set except for SQuAD where we use the validation set. a(Lan et al., 2019) b(Wang et al., 2019c)c(Zhu et al., 2019)d(Liu et al., … (excerpt)

Putting It All Together · page 39 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_90fd6cc4
For SuperGLUE, we improved upon the state-of-the-art by a large margin (from an average score of 84.6 (Liu et al., 2019c) to88.9).

Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5ab8f5ef
On the other hand, humans achieve100% accuracy on both COPA and WSC, which is significantly better than our model’s performance.

Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2dd8b17e
Our results suggest that scale and English-language pre-training may be insufficient to match the performance of these more sophisticated methods.

Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5f63d771
Finally, on CNN/Daily Mail we attain state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score.

Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1910.10683) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.