clm_167777c7Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
arXiv 2005.11401 · cs.CL cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_167777c7Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
clm_ed50a2f4We fine-tune and evaluate our models on a wide range of knowledgeintensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq … (excerpt)
clm_bc1aaab8For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
res_6e3bce29Our RAG models achieve state-of-the-art results on open Natural Questions [29], WebQuestions [3] and CuratedTrec [2] and strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [24].
res_ece08d85For FEVER [56] fact verification, we achieve results within 4.3% of state-of-the-art pipeline models which use strong retrieval supervision.
clm_dfbaeed6Finally, we demonstrate that the non-parametric memory can be replaced to update the models’ knowledge as the world changes.1
clm_74f2592cWe propose two models that marginalize over the latent documents in different ways to produce a distribution over generated text.
clm_bbbe7cd8Rather than use questions from standard open-domain QA tasks, which typically consist of short, simple questions, we propose the more demanding task of generating Jeopardy questions.
clm_56ba7c43On all four open-domain QA tasks, RAG sets a new state of the art (only on the T5-comparable split for TQA).
res_57e9dbb6Furthermore, RAG can generate correct answers even when the correct answer is not in any retrieved document, achieving 11.8% accuracy in such cases for NQ, where an extractive model would … (excerpt)
res_a024f45aAs shown in Table 2, RAG-Sequence outperforms BART on Open MS-MARCO NLG by 2.6 Bleu points and 2.6 Rouge-L points.
res_cd79682dTable 2 shows that RAG-Token performs better than RAG-Sequence on Jeopardy question generation, with both models outperforming BART on Q-BLEU-1.
res_5f94924eRAG achieves an accuracy within 2.7% of this model, despite being supplied with only the claim and retrieving its own evidence.
res_67d24663As shown in Table 6, learned retrieval improves results for all tasks.
res_276c033dFigure 3 (left) shows that retrieving more documents at test time monotonically improves Open-domain QA results for RAG-Sequence, but performance peaks for RAG-Token at 10 retrieved documents.
res_376c5e5eFigure 3 (right) shows that retrieving more documents leads to higher Rouge-L for RAG-Token at the expense of Bleu-1, but the effect is less pronounced for RAG-Sequence.
clm_c01ca085We showed that our RAG models obtain state of the art results on open-domain QA.
res_cc2b7149After submission, We have ported our code to HuggingFace Transformers [66]3, which achieves equivalent performance to the previous version but is a cleaner and easier to use implementation.
res_ec63024dThe T5 model with the closest number of parameters to our models is T5-large (770M parameters), which achieves a score of 28.9 EM on Natural Questions [52], substantially below the … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2005.11401) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.