clm_b37956daOur pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on Wiki- Hop and TriviaQA.
arXiv 2004.05150 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_b37956daOur pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on Wiki- Hop and TriviaQA.
res_08255e49On the other hand, our proposed Longformer is able to build contextual representations of the entire context using multiple layers of attention, reducing the arXiv:2004.05150v2 [cs.CL] 2 Dec 2020 need … (excerpt)
clm_a1cc3bd3We achieve state-of-the-art results on text8 and enwik8 benchmark datasets, demonstrating the effectiveness of Longformer in long document modeling.
clm_3d6cd7cdAfter pretraining, we apply it to downstream language tasks through finetuning and demonstrate that Longformer consistently outperforms RoBERTa on a wide range of document-level natural language tasks including text classification, … (excerpt)
clm_487da822We demonstrate the effectiveness of LED on the arXiv summarization dataset (Cohan et al., 2018).
res_d6524673For the higher layers, we use a small amount of increasing dilation only on 2 heads.
res_bdae8bb9We achieve a new state-of-the-art on both text8 and enwik8 using the small models with BPC of 1.10 and 1.00 on text8 and enwik8 respectively, demonstrating the effectiveness of our … (excerpt)
res_98fcf458Traininig for 2K steps improves BPC from 1.957 to 1.753, which further decreases to 1.705 after 65K steps, demonstrating the model is learning to better utilize the sliding window attention … (excerpt)
clm_4ec35749We observe that Longformer consistently outperforms the RoBERTa baseline.
res_71a96281Tab. 8 shows that our Longformer-large achieves new state-of-the-art results9 on WikiHop and TriviaQA by large margins (3.6 and 4 points respectively), and for HotpotQA, it underperforms the current state-of-the-art … (excerpt)
res_9d9e50a3In addition, when configured as in RoBERTa-base (seqlen: 512, and n2 attention) Longformer performs slightly worse then RoBERTa-base, confirming that performance gains are not due to additional pretraining.
clm_63a6dc67To facilitate modeling long sequences for seq2seq learning, we propose a Longformer variant that has both the encoder and decoder Transformer stacks but instead of the full self-attention in the … (excerpt)
clm_f7a7152bWe observe that LED achieves stateof-the-art results on arXiv, slightly outperforming BigBird (Zaheer et al., 2020).
clm_8b7658b8With no pre-training or task-specific initialization, but with ability to process longer inputs, LED can slightly outperform BigBird.
res_e42b9a0fFig. 3 further illustrates the importance of sequence length showing the ablility to process longer input significantly improves the results.
clm_7819b5a0Longformer achieves state-of-the-art results on the character-level language modeling tasks of text8 and enwik8.
clm_d965ded1When pretrained, Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.
clm_119c3ccdWe further present LED, an encoder-decoder variant of Longformer for modeling sequence-to-sequence tasks, and achieve stateof-the-art results on the arXiv long document summarization task.
res_1e1dbda3We ran two hyperparameter trials for Longformer-large, lr=3e-5 and number epochs in [5, 15] (the 5 epoch model had higher dev accuracy of 77.6, and was the single model submitted … (excerpt)
res_e6c1df6d15The final dev performance of the two stage model improves over a single stage model by about 4.2 points on joint- F1 metric 15 ⟨t⟩ title2 ⟨/t⟩ sent2,1 [s] sent … (excerpt)
res_283e766eWe then keep up to 5 paragraphs whose raw score is higher than a pre-specified threshold (-3.0), and remove the other paragraphs from the context.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2004.05150) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.