clm_2d8e52baIn terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.
arXiv 2304.07193 · cs.CV
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_2d8e52baIn terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.
res_f8f29f61In terms of models, we train a ViT model (Dosovitskiy et al., 2021) with 1B parameters and distill it into a series of smaller models that surpass the best available … (excerpt)
res_2d3fb9dcThese improvements make our approach around 2× faster and require 3× less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.
clm_5c9fd609Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weaklysupervised features.
res_1f46702fOur experiments did not show significant differences in final accuracy, and our ViT-g backbone counts 1.1B parameters.
res_f1d584f8With high drop rates (d = 40% in this work), this allows a drastic improvement in compute efficiency and memory usage.
res_f56c2912Our approach improves over the iBOT method by combining it with several existing components described in Sec. 4.
clm_67d09b71Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.
res_8d504b61This confirms that training on a more diverse set of images improves the quality of the features in domains that are not covered by ImageNet-22k.
res_cc82c7f6The ViT-g trained on LVD-142M surpasses the ViT-g trained on ImageNet-22k on most benchmarks.
res_860db36dFor instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.
res_e3b2ed7dWe validated the proposed technical improvements in Sec. 6.1 by adding them incrementally.
res_ce150cbeThe distilled model outperforms the one trained from scratch on all 12 benchmarks, validating our pretraining approach for small models.
clm_eac6e681We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.
clm_95003f68First, we show that our self-supervised features outperform the current state of the art by a very large margin.
clm_6b1d4b17Second, we show that they match, or surpass the performance of weakly-supervised ones on a substantial number of tasks.
res_d65485dcWe see the components proposed in this work lead to a very significant improvement (+4.2%) over the previous state of the art (iBOT ViT-L/16 trained on ImageNet-22k) on linear evaluation.
clm_72c3a007How far are we from weakly-supervised models?We also want to validate that our features are competitive with state-of-the-art open-source weakly supervised models.
res_dd212594At the same time, we also observe that our performance on the ImageNet-V2 test set is significantly better (+1.1% versus EVA-CLIP), indicating better generalization.
res_2df32fecIn Table 5, we show that the Top-1 accuracy on the validation set of ImageNet-1k improves by more than+2% when the backbone is finetuned.
res_75388fc0When comparing with state-of-the-art SSL methods, our models shows drastically better robustness (+29.6% on A (Hendrycks et al., 2021b),+22.1% on R (Hendrycks et al., 2021a) and+23.0% on Sketch (Wang et … (excerpt)
res_a01f62c9Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist (+8.6% and +9.7% for 2018 and 2021 respectively), and lags slightly behind on Places 205 (−2.3%).
clm_7711b6ccWe see that amongst self-supervised approaches, our model clearly sets a new state of the art.
res_d1f59935Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics (+0.1% and +0.5% respectively) and clearly outperforms them on SSv2 (+2.5%).
res_71e37b11Our model significantly outperforms state-of-the-art SSL models, with most notable differences on Stanford Cars (+14.8% versus DINO ViT-B/8) and FGVC Aircraft (+14.8% versus iBOT ViT-L/16).
res_2e1002e4Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (−5.3%) and Cars … (excerpt)
res_2ca92a63We see that our features significantly outperform both SSL (+41% mAP on Oxford-Hard), and weakly-supervised (+34% mAP on Oxford-Hard) ones.
res_4623022fWe concatenate the patch tokens of the 4 last layers, use a larger image resolution of 640, and use multiscale test-time augmentations to improve the predictions.
clm_4c0abf18Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art on Pascal VOC (86.2 versus 89.0 mIoU).
res_5aafdcadWith this setup, we reach60.2 mIoU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU (Wang et al., 2022).
clm_e8e85fb2It is interesting to see that iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G.
clm_b2548ee3In this final section of the empirical evaluation of our features, we propose a few qualitative analyses.
clm_b83db96cOur model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by Goyal et al. (2022a).
res_4c1b1617However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe.
clm_e1e700f4In this work, we present DINOv2, a new series of image encoders pretrained on large curated data with no supervision.
res_39f01facWe can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1),ii) a larger model … (excerpt)
res_00b4aba1To achieve this, we apply a similar procedure as for self-deduplication, with a stricter similarity >0.45, this time identifying the duplicate components (if any) to which each reference image belong … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2304.07193) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.