clm_1a7585d7We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.
arXiv 2010.11929 · cs.CV cs.AI cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_1a7585d7We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.
clm_6579cc86When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art … (excerpt)
clm_678f5cbcWhen pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks.
res_21ff5fd9In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks.
clm_8048381cThis model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs.
res_a04700b6The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy … (excerpt)
clm_38b6cb97When considering the computational cost of pre-training the model, ViT performs very favourably, attaining state of the art on most recognition benchmarks at a lower pre-training cost.
clm_53908eaaWe first compare our largest models – ViT-H/14 and ViT-L/16 – to state-of-the-art CNNs from the literature.
res_2448da7eThe smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks, while requiring substantially less computational resources to train.
res_6cb8f77bThe larger model, ViT-H/14, further improves the performance, especially on the more challenging datasets – ImageNet, CIFAR-100, and the VTAB suite.
res_a28ee9b1Vision Transformer models pre-trained on the JFT-300M dataset outperform ResNet-based baselines on all datasets, while taking substantially less computational resources to pre-train.
res_2cc7d1c0ViT-H/14 outperforms BiT-R152x4, and other methods, on theNatural and Structured tasks.
clm_cb172929Vision Transformers generally outperform ResNets with the same computational budget.
clm_dca691ffThe BiT CNNs outperform ViT on ImageNet, but with the larger datasets, ViT overtakes.
clm_c28029e7Second, hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models.
res_73bbe0c2That the position embeddings learn to represent 2D image topology explains why hand-crafted 2D-aware embedding variants do not yield improvements (Appendix D.4).
res_bb2227bbWith self-supervised pre-training, our smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant improvement of 2% to training from scratch, but still 4% behind supervised pre-training.
clm_89994f89Thus, Vision Transformer matches or exceeds the state of the art on many image classification datasets, whilst being relatively cheap to pre-train.
res_5048f1a0We also experimented with 15% corruption rate as used by Devlin et al. (2019) but results were also slightly worse on our few-shot metrics.
res_b390e345Note that the ImageNet results are computed without additional techniques (Polyak averaging and 512 resolution images) used to achieve results in Table 2.
clm_d7727da8Adam pre-training outperforms SGD pre-training on most datasets and on average.
res_4fcb0de4We can see that scaling the depth results in the biggest improvements which are clearly visible up until 64 layers.
res_d2c77a92For the AxialResNet, although it looks reasonable in terms of accuracy/compute trade-off (Figure 13, left), the naive implementation is extremely slow on TPUs (Figure 13, right).
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2010.11929) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.