MearcAll papers

arXiv 2010.11929 · cs.CV cs.AI cs.LG

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, and 6 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_1a7585d7
We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6579cc86
When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_678f5cbc
When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_21ff5fd9
In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8048381c
This model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs.

R ELATED WORK · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a04700b6
The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy … (excerpt)

R ELATED WORK · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_38b6cb97
When considering the computational cost of pre-training the model, ViT performs very favourably, attaining state of the art on most recognition benchmarks at a lower pre-training cost.

E XPERIMENTS · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_53908eaa
We first compare our largest models – ViT-H/14 and ViT-L/16 – to state-of-the-art CNNs from the literature.

C OMPARISON TO STATE OF THE ART · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2448da7e
The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks, while requiring substantially less computational resources to train.

C OMPARISON TO STATE OF THE ART · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6cb8f77b
The larger model, ViT-H/14, further improves the performance, especially on the more challenging datasets – ImageNet, CIFAR-100, and the VTAB suite.

C OMPARISON TO STATE OF THE ART · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a28ee9b1
Vision Transformer models pre-trained on the JFT-300M dataset outperform ResNet-based baselines on all datasets, while taking substantially less computational resources to pre-train.

C OMPARISON TO STATE OF THE ART · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2cc7d1c0
ViT-H/14 outperforms BiT-R152x4, and other methods, on theNatural and Structured tasks.

C OMPARISON TO STATE OF THE ART · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cb172929
Vision Transformers generally outperform ResNets with the same computational budget.

ImageNet · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_dca691ff
The BiT CNNs outperform ViT on ImageNet, but with the larger datasets, ViT overtakes.

ImageNet · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c28029e7
Second, hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models.

S CALING STUDY · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_73bbe0c2
That the position embeddings learn to represent 2D image topology explains why hand-crafted 2D-aware embedding variants do not yield improvements (Appendix D.4).

I NSPECTING VISION TRANSFORMER · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb2227bb
With self-supervised pre-training, our smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant improvement of 2% to training from scratch, but still 4% behind supervised pre-training.

Input patch row · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_89994f89
Thus, Vision Transformer matches or exceeds the state of the art on many image classification datasets, whilst being relatively cheap to pre-train.

C ONCLUSION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5048f1a0
We also experimented with 15% corruption rate as used by Devlin et al. (2019) but results were also slightly worse on our few-shot metrics.

S ELF -SUPERVISION · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b390e345
Note that the ImageNet results are computed without additional techniques (Polyak averaging and 512 resolution images) used to achieve results in Table 2.

A DDITIONAL RESULTS · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d7727da8
Adam pre-training outperforms SGD pre-training on most datasets and on average.

A DDITIONAL ANALYSES · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4fcb0de4
We can see that scaling the depth results in the biggest improvements which are clearly visible up until 64 layers.

T RANSFORMER SHAPE · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d2c77a92
For the AxialResNet, although it looks reasonable in terms of accuracy/compute trade-off (Figure 13, left), the naive implementation is extremely slow on TPUs (Figure 13, right).

A XIAL ATTENTION · page 20 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2010.11929) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.