MearcAll papers

arXiv 2004.05150 · cs.CL

Longformer: The Long-Document Transformer

Iz Beltagy, Matthew E. Peters, Arman Cohan

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_b37956da
Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on Wiki- Hop and TriviaQA.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_08255e49
On the other hand, our proposed Longformer is able to build contextual representations of the entire context using multiple layers of attention, reducing the arXiv:2004.05150v2 [cs.CL] 2 Dec 2020 need … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a1cc3bd3
We achieve state-of-the-art results on text8 and enwik8 benchmark datasets, demonstrating the effectiveness of Longformer in long document modeling.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3d6cd7cd
After pretraining, we apply it to downstream language tasks through finetuning and demonstrate that Longformer consistently outperforms RoBERTa on a wide range of document-level natural language tasks including text classification, … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_487da822
We demonstrate the effectiveness of LED on the arXiv summarization dataset (Cohan et al., 2018).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6524673
For the higher layers, we use a small amount of increasing dilation only on 2 heads.

Attention Pattern · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bdae8bb9
We achieve a new state-of-the-art on both text8 and enwik8 using the small models with BPC of 1.10 and 1.00 on text8 and enwik8 respectively, demonstrating the effectiveness of our … (excerpt)

Results · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_98fcf458
Traininig for 2K steps improves BPC from 1.957 to 1.753, which further decreases to 1.705 after 65K steps, demonstrating the model is learning to better utilize the sliding window attention … (excerpt)

Pretraining and Finetuning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4ec35749
We observe that Longformer consistently outperforms the RoBERTa baseline.

Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_71a96281
Tab. 8 shows that our Longformer-large achieves new state-of-the-art results9 on WikiHop and TriviaQA by large margins (3.6 and 4 points respectively), and for HotpotQA, it underperforms the current state-of-the-art … (excerpt)

Results · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d9e50a3
In addition, when configured as in RoBERTa-base (seqlen: 512, and n2 attention) Longformer performs slightly worse then RoBERTa-base, confirming that performance gains are not due to additional pretraining.

Ablations on WikiHop · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_63a6dc67
To facilitate modeling long sequences for seq2seq learning, we propose a Longformer variant that has both the encoder and decoder Transformer stacks but instead of the full self-attention in the … (excerpt)

Longformer-Encoder-Decoder (LED) · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f7a7152b
We observe that LED achieves stateof-the-art results on arXiv, slightly outperforming BigBird (Zaheer et al., 2020).

Longformer-Encoder-Decoder (LED) · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8b7658b8
With no pre-training or task-specific initialization, but with ability to process longer inputs, LED can slightly outperform BigBird.

Longformer-Encoder-Decoder (LED) · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e42b9a0f
Fig. 3 further illustrates the importance of sequence length showing the ablility to process longer input significantly improves the results.

Longformer-Encoder-Decoder (LED) · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7819b5a0
Longformer achieves state-of-the-art results on the character-level language modeling tasks of text8 and enwik8.

Conclusion and Future Work · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d965ded1
When pretrained, Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

Conclusion and Future Work · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_119c3ccd
We further present LED, an encoder-decoder variant of Longformer for modeling sequence-to-sequence tasks, and achieve stateof-the-art results on the arXiv long document summarization task.

Conclusion and Future Work · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1e1dbda3
We ran two hyperparameter trials for Longformer-large, lr=3e-5 and number epochs in [5, 15] (the 5 epoch model had higher dev accuracy of 77.6, and was the single model submitted … (excerpt)

Pretraining Data · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6c1df6d
15The final dev performance of the two stage model improves over a single stage model by about 4.2 points on joint- F1 metric 15 ⟨t⟩ title2 ⟨/t⟩ sent2,1 [s] sent … (excerpt)

Pretraining Data · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_283e766e
We then keep up to 5 paragraphs whose raw score is higher than a pre-specified threshold (-3.0), and remove the other paragraphs from the context.

Pretraining Data · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2004.05150) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.