MearcAll papers

arXiv 2104.14294 · cs.CV

Emerging Properties in Self-Supervised Vision Transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and 1 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_a8c608ea
We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. ∗Univ.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e5036581
This information is directly accessible in the self-attention modules of the last block. • Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN) without any finetuning, linear classifier nor data augmentation, achieving 78.3% top-1 accuracy on ImageNet.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d935930f
We further validate the synergy between DINO and ViT by outperforming previous self-supervised features on the ImageNet linear classification benchmark with 80.1% top-1 accuracy with a ViT-Base with small patches.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a51a0f32
We also confirm that DINO works with convnets by matching the state of the art with a ResNet-50 architecture.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_86743e85
First, we observe that DINO performs on par with the state of the art on ResNet-50, validating that DINO works in the standard setting.

Comparing with SSL frameworks on ImageNet · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_61e2b146
When we switch to a ViT architecture, DINO outperforms BYOL, MoCov2 and SwA V by +3.5% with linear classification and by +7.9% withk-NN evaluation.

Comparing with SSL frameworks on ImageNet · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_65dd803c
While training a larger ViT with DINO improves the performance, reducing the size of the patches (“/8” variants) has a bigger impact on the performance.

Comparing with SSL frameworks on ImageNet · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6e0629a1
We observe that DINO features outperform those trained on ImageNet with labels.

Nearest neighbor retrieval with DINO ViT · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0cb0c1b6
Finally, for this dense recognition task, the variants with small patches (“/8”) perform much better (+9.1% (J &F )m for ViT-B).

Discovering the semantic layout of scenes · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5e78817c
In Fig. 3, we show that different heads can attend to different semantic regions of an image, even when they are occluded (the bushes on the third row) or small (the flag on the second row).

Discovering the semantic layout of scenes · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8419d22d
In Fig. 4, we show that a supervised ViT does not attend well to objects in presence of clutter both qualitatively and quantitatively.

Discovering the semantic layout of scenes · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3e875de4
Finally, self-supervised pretraining greatly improves results on ImageNet (+1-2%).

Transfer learning on downstream tasks · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7899479d
For completeness, we propose in Appendix B an extended version of this ablation study.

Importance of the Different Components · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_433bca5f
A key observation is that this teacher constantly outperforms the student during the training, and we observe the same behavior when training with a ResNet-50 (Appendix D).

Impact of the choice of Teacher Network · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3d3a15d9
We propose to interpret the momentum teacher in DINO as a form of Polyak-Ruppert averaging [ 51, 59] with an exponentially decay.

Impact of the choice of Teacher Network · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_aeee16e1
We observe in Tab. 8 that using multi-crop improves the accuracy / running-time tradeoff for DINO runs.

Compute requirements · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d609af2e
This is an improvement of+2% while requiring 2× less time, though the memory usage is higher (15.4G versus 9.3G).

Compute requirements · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_67759979
Overall, training DINO with Vision Transformers achieves 76.1 top-1 accuracy using two 8-GPU servers for 3 days.

Compute requirements · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4fc6dbd4
In this work, we have shown the potential of selfsupervised pretraining a standard ViT model, achieving performance that are comparable with the best convnets specifically designed for this setting.

Conclusion · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7e35f177
In Tab. 10 we see that ViT-S accuracies are better than accuracies obtained with RN50 both with a linear or a k-NN classifier.

Additional Results · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b3cd3177
For example on ImageNet 1%, ViT-S outperforms ResNet-50 by a large margin of +14.1% with k-NN evaluation.

Additional Results · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7db5be47
When comparing models with a similar number of parameters and image/sec, we observe that our features are on par with state-of-the-art semi-supervised models.

Additional Results · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e307fadc
However, DINO unravels its potential with ViT, outperforming MoCo-v2, SwA V and BYOL by large margins (+4.3% with linear and +6.2% with k-NN evaluations).

Methodology Comparison · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ca50523e
All numbers are run by us and match or outperform published results.

Methodology Comparison · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c800eb0f
Finally, we observe that multi-crop works particularly well with DINO and MoCo-v2, removing it hurts performance by 2− 4% (1 versus 4 and, 5 versus 6).

Methodology Comparison · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0ce132aa
First, these ablations show that using a momentum encoder significantly improves the performance for ViT (3 versus 6, and 2 versus 5).

Methodology Comparison · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b768ed0c
We observe in Tab. 13 that our reproduction of BYOL, MoCo-v2, SwA V matches or outperforms the corresponding published numbers with ResNet-50.

Methodology Comparison · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3e2e87d4
Our improvement compared to the implementation of [ 15] can be explained by the use of a larger projection head (3-layer, use of batchnormalizations and projection dimension of 256).

Methodology Comparison · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b9c509d5
Similarly to other self-supervised frameworks, using a projection head [12] improves greatly the accuracy of our method.

Projection Head · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d7d4c7e
The convergence is robust m 0 0.9 0.99 0.999 k-NN top-1 69.1 69.7 69.4 0.1 to a wide range of smoothing, and the model only collapses when the update is too slow, i.e.,m = 0.999.

Additional Ablations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5cd66db2
In this table, we observe that a temperature lower than 0.06 is required to avoid collapse.

Additional Ablations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_220a9b5f
However, we have observed that using higher temperature than 0.06 does not collapse if we start the training from a smaller value and increase it during the first epochs.

Additional Ablations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2ad76a6c
We note that in our experiments with BYOL on ViT-S, training longer than 300 epochs has been leading to worse performance compare our 300 epochs run.

Additional Ablations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6c8b269
We have shown in Fig. 6 that the momentum teacher outperforms the student with ViT and we show in this Figure that it is also the case with ResNet-50.

Additional Ablations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_79a4a778
Without multi-crop, DINO has better accuracy than other frameworks, though by a moderate margin (1%).

Multi-crop · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ecf5224a
As a final visualization, we propose to look at the distribution of ImageNet concepts in the feature space from DINO.

Class Representation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.14294) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.