MearcAll papers

arXiv 1706.03762 · cs.CL cs.LG

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and 2 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_07eb8978
We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4e525f82
Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_62ef5654
On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_add315ff
In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_44565f2f
The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3153e763
On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0 BLEU, establishing … (excerpt)

Machine Translation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_addb787d
On the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0, outperforming all of the previously published single models, at less than 1/4 the training … (excerpt)

Machine Translation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_be04bd39
While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.

Model Variations · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e62d28b1
In Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality.

Model Variations · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8cc402ad
For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_822da8d6
On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4851b6ec
In the former task our best model outperforms even all previously reported ensembles.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1706.03762) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.