clm_98e866c4We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a … (excerpt)
arXiv 2103.00020 · cs.CV cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_98e866c4We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a … (excerpt)
res_b39cbc2dFor instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on.
res_aa6de309Swapping the prediction objective for the contrastive objective of CLIP further improves efficiency another 4x. it can be competitive with prior task-specific supervised models.
clm_7778f489We also confirm these findings with linear-probe representation learning analysis and show that CLIP outperforms the best publicly available ImageNet model while also being more computationally efficient.
clm_c4a8eac6In Figure 2 we show that a 63 million parameter transformer language model, which already uses twice the compute of its ResNet-50 image encoder, learns to recognize ImageNet classes three … (excerpt)
res_ac00c50dStarting with the same bag-of-words encoding baseline, we swapped the predictive objective for a contrastive objective in Figure 2 and observed a further 4x efficiency improvement in the rate of … (excerpt)
res_a4c6de94We make several modifications to the original version using the ResNet- D improvements from He et al. (2019) and the antialiased rect-2 blur pooling from Zhang (2019).
clm_279c103cWhile previous computer vision research has often scaled models by increasing the width (Mahajan et al., 2018) or depth (He et al., 2016a) in isolation, for the ResNet image encoders … (excerpt)
res_f80fc821The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 … (excerpt)
res_79135e91CLIP also outperforms Visual N-Grams on the other 2 reported datasets.
res_50f117d7On aYahoo, CLIP achieves a 95% reduction in the number of errors, and on SUN, CLIP more than doubles the accuracy of Visual N-Grams.
res_2f3e963dThis improvement is similar to the gain from using 4 times more compute with the baseline zero-shot method but is “free” when amortized over many predictions. chosen somewhat haphazardly and … (excerpt)
res_5a894a6cFor instance, just using this prompt improves accuracy on ImageNet by 1.3%.
res_49df7848Learning Transferable Visual Models From Natural Language Supervision 8 Similar to the “prompt engineering” discussion around GPT- 3 (Brown et al., 2020; Gao et al., 2020), we have also observed … (excerpt)
res_6bc399e3On ImageNet, we ensemble 80 different context prompts and this improves performance by an additional 3.5% over the single default prompt discussed above.
res_4b680377When considered together, prompt engineering and ensembling improve ImageNet accuracy by almost 5%.
res_ad2ae871Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet. ten than not and wins … (excerpt)
res_e668f2c1On two of these datasets, Stanford Cars and Food101, zero-shot CLIP outperforms logistic regression on ResNet-50 features by over 20% while on two others, Flowers102 and FGVCAircraft, zero-shot CLIP underperforms … (excerpt)
res_1f6f32adOn STL10, CLIP achieves 99.3% overall which appears to be a new state of the art despite not using any training examples.
res_9d7ffb9fZeroshot CLIP significantly outperforms a ResNet-50 on two datasets measuring action recognition in videos.
res_9c6f4ec2Zeroshot CLIP also outperforms a ResNet-50’s features by 7.7% on UCF101.
clm_8afe5568Zero-shot CLIP outperforms few-shot linear probes.
res_c6244b1aThat a BiT-M ResNet-152x2 performs best in a 16-shot setting is somewhat surprising since, as analyzed in Section 3.2, the Noisy Student EfficientNet-L2 outperforms it in a fully supervised setting … (excerpt)
res_9c36ec71For most datasets, the performance of zero-shot classifiers still underperform fully supervised classifiers by 10% to 25%, suggesting that there is still plenty of headroom for improving CLIP’s task-learning and … (excerpt)
res_0577a764The slope of a linear regression model predicting zero-shot performance as a function of fully supervised performance estimates that for every 1% improvement in fully supervised performance, zero-shot performance improves … (excerpt)
res_456aea1dThe GPT family of models has so far demonstrated consistent improvements in zero-shot performance across a 1000x increase in training compute.
res_d7303e08While small CLIP models such as a ResNet-50 and ResNet-101 outperform other ResNets trained on ImageNet-1K (BiT-S and the originals), they underperform ResNets trained on ImageNet-21K (BiT- M).
res_b1732c3dHowever, models trained with CLIP scale very well and the largest model we trained (ResNet-50x64) slightly outperforms the best performing existing model (a Noisy Student EfficientNet-L2) on both overall score … (excerpt)
res_3f1cc84eWe also find that CLIP vision transformers are about 3x more compute efficient than CLIP ResNets, which allows us to reach higher overall performance within our compute budget.
res_7906d98eThis model outperforms the best existing model across this evaluation suite by an average of 2.6%.
clm_e194ecafAll CLIP models, regardless of scale, outperform all evaluated systems in terms of compute efficiency.
res_97f364f5The improvement in average score of the best model over previous systems increases from 2.6% to 5%.
res_ac3602b6For instance, while SimCLRv2 still underperforms BiT-M on average on the 12 datasets of Kornblith et al. (2019), SimCLRv2 outperforms BiT-M on our 27 dataset evaluation suite.
res_4faa2760CLIP outperforms the Noisy Student EfficientNet-L2 on 21 of the 27 datasets.
res_3de6a489A result such as the 14.7% improvement on GTSRB could be indicative of an issue with ImageNet-1K, which has only a single label for all traffic and street signs.
clm_72c75bb2The EffcientNet also slightly outperforms CLIP on low-resolution datasets such as CIFAR10 and CIFAR100.
clm_21b4c54eCLIP’s features outperform the features of the best ImageNet model on a wide variety of datasets.
res_80b22546Fitting a linear classifier on CLIP’s features outperforms using the Noisy Student EfficientNet-L2 on 21 out of 27 datasets. low for both approaches.
res_d864338dAll zero-shot CLIP models improve effective robustness by a large amount and reduce the size of the gap between ImageNet accuracy and accuracy under distribution shift by up to 75%.
res_e6e0d429Zero-shot CLIP models shrink this “robustness gap” by up to 75%.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2103.00020) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.