MearcAll papers

arXiv 2103.00020 · cs.CV cs.LG

Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, and 6 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_98e866c4
We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b39cbc2d
For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_aa6de309
Swapping the prediction objective for the contrastive objective of CLIP further improves efficiency another 4x. it can be competitive with prior task-specific supervised models.

Introduction and Motivating Work · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7778f489
We also confirm these findings with linear-probe representation learning analysis and show that CLIP outperforms the best publicly available ImageNet model while also being more computationally efficient.

Introduction and Motivating Work · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c4a8eac6
In Figure 2 we show that a 63 million parameter transformer language model, which already uses twice the compute of its ResNet-50 image encoder, learns to recognize ImageNet classes three … (excerpt)

Natural Language Supervision · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ac00c50d
Starting with the same bag-of-words encoding baseline, we swapped the predictive objective for a contrastive objective in Figure 2 and observed a further 4x efficiency improvement in the rate of … (excerpt)

Natural Language Supervision · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a4c6de94
We make several modifications to the original version using the ResNet- D improvements from He et al. (2019) and the antialiased rect-2 blur pooling from Zhang (2019).

Choosing and Scaling a Model · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_279c103c
While previous computer vision research has often scaled models by increasing the width (Mahajan et al., 2018) or depth (He et al., 2016a) in isolation, for the ResNet image encoders … (excerpt)

Choosing and Scaling a Model · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f80fc821
The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 … (excerpt)

I NITIAL COMPARISON TO VISUAL N-G RAMS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_79135e91
CLIP also outperforms Visual N-Grams on the other 2 reported datasets.

I NITIAL COMPARISON TO VISUAL N-G RAMS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_50f117d7
On aYahoo, CLIP achieves a 95% reduction in the number of errors, and on SUN, CLIP more than doubles the accuracy of Visual N-Grams.

I NITIAL COMPARISON TO VISUAL N-G RAMS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2f3e963d
This improvement is similar to the gain from using 4 times more compute with the baseline zero-shot method but is “free” when amortized over many predictions. chosen somewhat haphazardly and … (excerpt)

P ROMPT ENGINEERING AND ENSEMBLING · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5a894a6c
For instance, just using this prompt improves accuracy on ImageNet by 1.3%.

P ROMPT ENGINEERING AND ENSEMBLING · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49df7848
Learning Transferable Visual Models From Natural Language Supervision 8 Similar to the “prompt engineering” discussion around GPT- 3 (Brown et al., 2020; Gao et al., 2020), we have also observed … (excerpt)

P ROMPT ENGINEERING AND ENSEMBLING · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6bc399e3
On ImageNet, we ensemble 80 different context prompts and this improves performance by an additional 3.5% over the single default prompt discussed above.

P ROMPT ENGINEERING AND ENSEMBLING · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4b680377
When considered together, prompt engineering and ensembling improve ImageNet accuracy by almost 5%.

P ROMPT ENGINEERING AND ENSEMBLING · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ad2ae871
Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet. ten than not and wins … (excerpt)

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e668f2c1
On two of these datasets, Stanford Cars and Food101, zero-shot CLIP outperforms logistic regression on ResNet-50 features by over 20% while on two others, Flowers102 and FGVCAircraft, zero-shot CLIP underperforms … (excerpt)

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1f6f32ad
On STL10, CLIP achieves 99.3% overall which appears to be a new state of the art despite not using any training examples.

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d7ffb9f
Zeroshot CLIP significantly outperforms a ResNet-50 on two datasets measuring action recognition in videos.

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c6f4ec2
Zeroshot CLIP also outperforms a ResNet-50’s features by 7.7% on UCF101.

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8afe5568
Zero-shot CLIP outperforms few-shot linear probes.

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c6244b1a
That a BiT-M ResNet-152x2 performs best in a 16-shot setting is somewhat surprising since, as analyzed in Section 3.2, the Noisy Student EfficientNet-L2 outperforms it in a fully supervised setting … (excerpt)

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c36ec71
For most datasets, the performance of zero-shot classifiers still underperform fully supervised classifiers by 10% to 25%, suggesting that there is still plenty of headroom for improving CLIP’s task-learning and … (excerpt)

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0577a764
The slope of a linear regression model predicting zero-shot performance as a function of fully supervised performance estimates that for every 1% improvement in fully supervised performance, zero-shot performance improves … (excerpt)

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_456aea1d
The GPT family of models has so far demonstrated consistent improvements in zero-shot performance across a 1000x increase in training compute.

A NALYSIS OF ZERO-SHOT CLIP P ERFORMANCE · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d7303e08
While small CLIP models such as a ResNet-50 and ResNet-101 outperform other ResNets trained on ImageNet-1K (BiT-S and the originals), they underperform ResNets trained on ImageNet-21K (BiT- M).

Representation Learning · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b1732c3d
However, models trained with CLIP scale very well and the largest model we trained (ResNet-50x64) slightly outperforms the best performing existing model (a Noisy Student EfficientNet-L2) on both overall score … (excerpt)

Representation Learning · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3f1cc84e
We also find that CLIP vision transformers are about 3x more compute efficient than CLIP ResNets, which allows us to reach higher overall performance within our compute budget.

Representation Learning · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7906d98e
This model outperforms the best existing model across this evaluation suite by an average of 2.6%.

Representation Learning · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e194ecaf
All CLIP models, regardless of scale, outperform all evaluated systems in terms of compute efficiency.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_97f364f5
The improvement in average score of the best model over previous systems increases from 2.6% to 5%.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ac3602b6
For instance, while SimCLRv2 still underperforms BiT-M on average on the 12 datasets of Kornblith et al. (2019), SimCLRv2 outperforms BiT-M on our 27 dataset evaluation suite.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4faa2760
CLIP outperforms the Noisy Student EfficientNet-L2 on 21 of the 27 datasets.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3de6a489
A result such as the 14.7% improvement on GTSRB could be indicative of an issue with ImageNet-1K, which has only a single label for all traffic and street signs.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_72c75bb2
The EffcientNet also slightly outperforms CLIP on low-resolution datasets such as CIFAR10 and CIFAR100.

Representation Learning · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_21b4c54e
CLIP’s features outperform the features of the best ImageNet model on a wide variety of datasets.

Representation Learning · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_80b22546
Fitting a linear classifier on CLIP’s features outperforms using the Noisy Student EfficientNet-L2 on 21 out of 27 datasets. low for both approaches.

Representation Learning · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d864338d
All zero-shot CLIP models improve effective robustness by a large amount and reduce the size of the gap between ImageNet accuracy and accuracy under distribution shift by up to 75%.

ResNet-101 makes 5 times as many mistakes when eval- · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6e0d429
Zero-shot CLIP models shrink this “robustness gap” by up to 75%.

ResNet-101 makes 5 times as many mistakes when eval- · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2103.00020) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.