clm_07eb8978We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
arXiv 1706.03762 · cs.CL cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_07eb8978We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
res_4e525f82Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU.
res_62ef5654On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of … (excerpt)
clm_add315ffIn this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
clm_44565f2fThe Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight … (excerpt)
res_3153e763On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0 BLEU, establishing … (excerpt)
res_addb787dOn the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0, outperforming all of the previously published single models, at less than 1/4 the training … (excerpt)
res_be04bd39While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.
res_e62d28b1In Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality.
clm_8cc402adFor translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers.
clm_822da8d6On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art.
clm_4851b6ecIn the former task our best model outperforms even all previously reported ensembles.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1706.03762) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.