res_a8c608eaWe show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. ∗Univ.
arXiv 2104.14294 · cs.CV
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
res_a8c608eaWe show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. ∗Univ.
res_e5036581This information is directly accessible in the self-attention modules of the last block. • Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN) without any finetuning, linear classifier nor data augmentation, achieving 78.3% top-1 accuracy on ImageNet.
res_d935930fWe further validate the synergy between DINO and ViT by outperforming previous self-supervised features on the ImageNet linear classification benchmark with 80.1% top-1 accuracy with a ViT-Base with small patches.
clm_a51a0f32We also confirm that DINO works with convnets by matching the state of the art with a ResNet-50 architecture.
clm_86743e85First, we observe that DINO performs on par with the state of the art on ResNet-50, validating that DINO works in the standard setting.
res_61e2b146When we switch to a ViT architecture, DINO outperforms BYOL, MoCov2 and SwA V by +3.5% with linear classification and by +7.9% withk-NN evaluation.
res_65dd803cWhile training a larger ViT with DINO improves the performance, reducing the size of the patches (“/8” variants) has a bigger impact on the performance.
clm_6e0629a1We observe that DINO features outperform those trained on ImageNet with labels.
res_0cb0c1b6Finally, for this dense recognition task, the variants with small patches (“/8”) perform much better (+9.1% (J &F )m for ViT-B).
clm_5e78817cIn Fig. 3, we show that different heads can attend to different semantic regions of an image, even when they are occluded (the bushes on the third row) or small (the flag on the second row).
clm_8419d22dIn Fig. 4, we show that a supervised ViT does not attend well to objects in presence of clutter both qualitatively and quantitatively.
res_3e875de4Finally, self-supervised pretraining greatly improves results on ImageNet (+1-2%).
clm_7899479dFor completeness, we propose in Appendix B an extended version of this ablation study.
res_433bca5fA key observation is that this teacher constantly outperforms the student during the training, and we observe the same behavior when training with a ResNet-50 (Appendix D).
clm_3d3a15d9We propose to interpret the momentum teacher in DINO as a form of Polyak-Ruppert averaging [ 51, 59] with an exponentially decay.
res_aeee16e1We observe in Tab. 8 that using multi-crop improves the accuracy / running-time tradeoff for DINO runs.
res_d609af2eThis is an improvement of+2% while requiring 2× less time, though the memory usage is higher (15.4G versus 9.3G).
res_67759979Overall, training DINO with Vision Transformers achieves 76.1 top-1 accuracy using two 8-GPU servers for 3 days.
clm_4fc6dbd4In this work, we have shown the potential of selfsupervised pretraining a standard ViT model, achieving performance that are comparable with the best convnets specifically designed for this setting.
res_7e35f177In Tab. 10 we see that ViT-S accuracies are better than accuracies obtained with RN50 both with a linear or a k-NN classifier.
res_b3cd3177For example on ImageNet 1%, ViT-S outperforms ResNet-50 by a large margin of +14.1% with k-NN evaluation.
clm_7db5be47When comparing models with a similar number of parameters and image/sec, we observe that our features are on par with state-of-the-art semi-supervised models.
res_e307fadcHowever, DINO unravels its potential with ViT, outperforming MoCo-v2, SwA V and BYOL by large margins (+4.3% with linear and +6.2% with k-NN evaluations).
clm_ca50523eAll numbers are run by us and match or outperform published results.
res_c800eb0fFinally, we observe that multi-crop works particularly well with DINO and MoCo-v2, removing it hurts performance by 2− 4% (1 versus 4 and, 5 versus 6).
res_0ce132aaFirst, these ablations show that using a momentum encoder significantly improves the performance for ViT (3 versus 6, and 2 versus 5).
res_b768ed0cWe observe in Tab. 13 that our reproduction of BYOL, MoCo-v2, SwA V matches or outperforms the corresponding published numbers with ResNet-50.
res_3e2e87d4Our improvement compared to the implementation of [ 15] can be explained by the use of a larger projection head (3-layer, use of batchnormalizations and projection dimension of 256).
res_b9c509d5Similarly to other self-supervised frameworks, using a projection head [12] improves greatly the accuracy of our method.
res_9d7d4c7eThe convergence is robust m 0 0.9 0.99 0.999 k-NN top-1 69.1 69.7 69.4 0.1 to a wide range of smoothing, and the model only collapses when the update is too slow, i.e.,m = 0.999.
res_5cd66db2In this table, we observe that a temperature lower than 0.06 is required to avoid collapse.
res_220a9b5fHowever, we have observed that using higher temperature than 0.06 does not collapse if we start the training from a smaller value and increase it during the first epochs.
res_2ad76a6cWe note that in our experiments with BYOL on ViT-S, training longer than 300 epochs has been leading to worse performance compare our 300 epochs run.
res_d6c8b269We have shown in Fig. 6 that the momentum teacher outperforms the student with ViT and we show in this Figure that it is also the case with ResNet-50.
res_79a4a778Without multi-crop, DINO has better accuracy than other frameworks, though by a moderate margin (1%).
clm_ecf5224aAs a final visualization, we propose to look at the distribution of ImageNet concepts in the feature space from DINO.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.14294) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.