arXiv 1910.10683 · cs.LG cs.CL stat.ML
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, and 3 more
Read on arXiv · arXiv non-exclusive license
- No reproduction recorded 40
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and … (excerpt)
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
For CNN/Daily Mail, we find the performance of models on the ROUGE-1-F, ROUGE-2-F, and ROUGE-L-F metrics (Lin, 2004) to be highly correlated so we report the ROUGE-2-F score alone under … (excerpt)
Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions
Similarly, for SQuAD we find the performance of the “exact match” and “F1” scores to be highly correlated so we report the “exact match” score alone.
Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions
We provide every score achieved on every task for all experiments in Table 16, Appendix E.
Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions
For example, BERTBASE achieved an exact match score of80.8 on SQuAD and an accuracy of 84.4 on MNLI-matched, whereas we achieve 80.88 and 84.24, respectively (see Table 16).
Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions
Unsurprisingly, we find that pre-training provides significant gains across almost all benchmarks.
Baseline Performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions
As for inter-run variance, we find that for most tasks the standard deviation across runs is smaller than1% of the task’s baseline score.
Baseline Performance · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions
The scores achieved by each of the architectures we compare are shown in Table 2.
Results · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions
We evaluate each architectural variant using a denoising objective (described in Section 3.1.4) and an autoregressive objective (as is commonly used to train language models). the encoder and decoder stacks … (excerpt)
Results · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions
We also note that the shared parameter encoder-decoder outperforms the decoder-only prefix LM, suggesting that the addition of an explicit encoder-decoder attention is beneficial.
Results · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions
Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks.
Disparate High-Level Approaches · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions
Indeed, the motivation for the BERT objective was to outperform language model-based pre-training.
Disparate High-Level Approaches · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that in our setting, all of these variants perform similarly.
Simplifying the BERT Objective · page 22 · Automated extraction, not reviewed · 0 reviewed reproductions
Overall, we find that the corruption rate had a limited effect on the model’s performance.
V arying the Corruption Rate · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions
The only exception is that the largest corruption rate we consider (50%) results in a significant degradation of performance on GLUE and SQuAD.
V arying the Corruption Rate · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions
Again, we find a limited difference between these objectives, though the version with an average span length of10 slightly underperforms the other values in some cases.
Corrupting Spans · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions
We also find in particular that using an average span length of3 slightly (but significantly) outperforms the i.i.d. objective on most non-translation benchmarks.
Corrupting Spans · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions
Overall, the most significant difference in performance we observed was that denoising objectives outperformed language modeling and deshuffling for pre-training.
Discussion · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions
The results achieved after pre-training on each of these data sets is shown in Table 8.
Unlabeled Data Sets · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions
Beyond this, we found that in some cases a pre-training data set with a more constrained domain outperformed the diverse C4 data set.
Unlabeled Data Sets · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions
Pursuant with past results (Houlsby et al., 2019; Bapna et al., 2019) we find that lower-resource tasks like SQuAD work well with a small value ofd whereas higher resource tasks … (excerpt)
Fine-tuning Methods · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions
In general, we find that multi-task training underperforms pre-training followed by fine-tuning on most tasks.
Multi-task Learning · page 32 · Automated extraction, not reviewed · 0 reviewed reproductions
For examples-proportional mixing, we find that for most tasks there is a “sweet spot” forK where the model obtains the best performance, and larger or smaller values ofK tend to … (excerpt)
Multi-task Learning · page 32 · Automated extraction, not reviewed · 0 reviewed reproductions
We find that fine-tuning after multi-task pre-training results in comparable performance to our baseline.
Combining Multi-Task Learning with Fine-Tuning · page 33 · Automated extraction, not reviewed · 0 reviewed reproductions
The performance achieved after applying these various scaling methods is shown in Table 13.
Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions
In some tasks (CNN/DM, WMT English to German, and WMT English to Romanian), ensembling4 completely separately trained models significantly outperformed every other scaling approach.
Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions
The only exception was SuperGLUE, where neither ensembling approach significantly improved over the baseline.
Scaling · page 35 · Automated extraction, not reviewed · 0 reviewed reproductions
In Section 3.4.1, we showed that pre-training on the RealNews-like, WebText-like, and Wikipedia + TBC data sets outperformed pre-training on C4 on a few downstream tasks.
Putting It All Together · page 36 · Automated extraction, not reviewed · 0 reviewed reproductions
Model sizes In Section 3.6 we also showed how scaling up the baseline model size improved performance.
Putting It All Together · page 36 · Automated extraction, not reviewed · 0 reviewed reproductions
Fortunately, the model with the highest performance on the SQuAD test set also reported results on the validation set, so we can still compare to what is ostensibly the state-of-the-art.
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
Overall, we achieved state-of-the-art performance on18 out of the24 tasks we consider.
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
Our T5-3B model variant did beat the previous state of the art in a few tasks, but scaling the model size to11 billion parameters was the most important ingredient for … (excerpt)
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
We achieved a state-of-the-art average GLUE score of90.3.
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
Notably, our performance was substantially better than the previous state-of-the-art for the natural language inference tasks MNLI, RTE, and WNLI.
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
For SQuAD, we outperformed the previous state-of-the-art (ALBERT (Lan et al., 2019)) by over one point on the Exact Match score.
Putting It All Together · page 38 · Automated extraction, not reviewed · 0 reviewed reproductions
All results are reported on the test set except for SQuAD where we use the validation set. a(Lan et al., 2019) b(Wang et al., 2019c)c(Zhu et al., 2019)d(Liu et al., … (excerpt)
Putting It All Together · page 39 · Automated extraction, not reviewed · 0 reviewed reproductions
For SuperGLUE, we improved upon the state-of-the-art by a large margin (from an average score of 84.6 (Liu et al., 2019c) to88.9).
Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions
On the other hand, humans achieve100% accuracy on both COPA and WSC, which is significantly better than our model’s performance.
Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions
Our results suggest that scale and English-language pre-training may be insufficient to match the performance of these more sophisticated methods.
Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions
Finally, on CNN/Daily Mail we attain state-of-the-art performance, though only by a significant amount on the ROUGE-2-F score.
Putting It All Together · page 40 · Automated extraction, not reviewed · 0 reviewed reproductions