MearcAll papers

arXiv 2304.07193 · cs.CV

DINOv2: Learning Robust Visual Features without Supervision

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, and 20 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_2d8e52ba
In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f8f29f61
In terms of models, we train a ViT model (Dosovitskiy et al., 2021) with 1B parameters and distill it into a series of smaller models that surpass the best available … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2d3fb9dc
These improvements make our approach around 2× faster and require 3× less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5c9fd609
Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weaklysupervised features.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1f46702f
Our experiments did not show significant differences in final accuracy, and our ViT-g backbone counts 1.1B parameters.

Efficient implementation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f1d584f8
With high drop rates (d = 40% in this work), this allows a drastic improvement in compute efficiency and memory usage.

Efficient implementation · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f56c2912
Our approach improves over the iBOT method by combining it with several existing components described in Sec. 4.

Improved Training Recipe · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_67d09b71
Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.

Improved Training Recipe · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8d504b61
This confirms that training on a more diverse set of images improves the quality of the features in domains that are not covered by ImageNet-22k.

Pretraining Data Source · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cc82c7f6
The ViT-g trained on LVD-142M surpasses the ViT-g trained on ImageNet-22k on most benchmarks.

Pretraining Data Source · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_860db36d
For instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.

Model Size and Data · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e3b2ed7d
We validated the proposed technical improvements in Sec. 6.1 by adding them incrementally.

Loss Components · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ce150cbe
The distilled model outperforms the one trained from scratch on all 12 benchmarks, validating our pretraining approach for small models.

Impact of Knowledge Distillation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_eac6e681
We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.

Impact of Knowledge Distillation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_95003f68
First, we show that our self-supervised features outperform the current state of the art by a very large margin.

Results · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6b1d4b17
Second, we show that they match, or surpass the performance of weakly-supervised ones on a substantial number of tasks.

Results · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d65485dc
We see the components proposed in this work lead to a very significant improvement (+4.2%) over the previous state of the art (iBOT ViT-L/16 trained on ImageNet-22k) on linear evaluation.

ImageNet Classification · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_72c3a007
How far are we from weakly-supervised models?We also want to validate that our features are competitive with state-of-the-art open-source weakly supervised models.

ImageNet Classification · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dd212594
At the same time, we also observe that our performance on the ImageNet-V2 test set is significantly better (+1.1% versus EVA-CLIP), indicating better generalization.

ImageNet Classification · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2df32fec
In Table 5, we show that the Top-1 accuracy on the validation set of ImageNet-1k improves by more than+2% when the backbone is finetuned.

ImageNet Classification · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_75388fc0
When comparing with state-of-the-art SSL methods, our models shows drastically better robustness (+29.6% on A (Hendrycks et al., 2021b),+22.1% on R (Hendrycks et al., 2021a) and+23.0% on Sketch (Wang et … (excerpt)

ImageNet Classification · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a01f62c9
Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist (+8.6% and +9.7% for 2018 and 2021 respectively), and lags slightly behind on Places 205 (−2.3%).

Additional Image and Video classification Benchmarks · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7711b6cc
We see that amongst self-supervised approaches, our model clearly sets a new state of the art.

Additional Image and Video classification Benchmarks · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d1f59935
Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics (+0.1% and +0.5% respectively) and clearly outperforms them on SSv2 (+2.5%).

Additional Image and Video classification Benchmarks · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_71e37b11
Our model significantly outperforms state-of-the-art SSL models, with most notable differences on Stanford Cars (+14.8% versus DINO ViT-B/8) and FGVC Aircraft (+14.8% versus iBOT ViT-L/16).

Additional Image and Video classification Benchmarks · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2e1002e4
Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (−5.3%) and Cars … (excerpt)

Additional Image and Video classification Benchmarks · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2ca92a63
We see that our features significantly outperform both SSL (+41% mAP on Oxford-Hard), and weakly-supervised (+34% mAP on Oxford-Hard) ones.

Instance Recognition · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4623022f
We concatenate the patch tokens of the 4 last layers, use a larger image resolution of 640, and use multiscale test-time augmentations to improve the predictions.

Dense Recognition Tasks · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4c0abf18
Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art on Pascal VOC (86.2 versus 89.0 mIoU).

Dense Recognition Tasks · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5aafdcad
With this setup, we reach60.2 mIoU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU (Wang et al., 2022).

Dense Recognition Tasks · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e8e85fb2
It is interesting to see that iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G.

Dense Recognition Tasks · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b2548ee3
In this final section of the empirical evaluation of our features, we propose a few qualitative analyses.

Qualitative Results · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b83db96c
Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by Goyal et al. (2022a).

Geographical Fairness · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4c1b1617
However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe.

Geographical Fairness · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e1e700f4
In this work, we present DINOv2, a new series of image encoders pretrained on large curated data with no supervision.

Future work and Discussion · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_39f01fac
We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1),ii) a larger model … (excerpt)

Future work and Discussion · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_00b4aba1
To achieve this, we apply a similar procedure as for self-deduplication, with a stricter similarity >0.45, this time identifying the duplicate components (if any) to which each reference image belong … (excerpt)

Deduplication · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2304.07193) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.