Attention Is All You Need
arXiv 1706.03762Older translation software read a sentence one word at a time. The Transformer reads every word at once and uses to learn which words matter to each other.
That makes it faster to train, and it translated better than earlier systems.
The Transformer drops recurrence and uses , so every token attends to every other token in parallel.
and positional encodings stand in for the word order an RNN gets for free. It reaches 28.4 on WMT 2014 English–German.
An encoder–decoder built only from stacked multi-head self-attention and position-wise feed-forward layers, with sinusoidal positional encodings.
The big model reaches 28.4 BLEU on WMT14 EN–DE and 41.8 on EN–FR, trained in 3.5 days on eight P100 GPUs.
InferenceFull parallelism over tokens is what later made large-scale pretraining practical.