MearcAll papers

arXiv 2002.05709 · cs.LG cs.CV stat.ML

A Simple Framework for Contrastive Learning of Visual Representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_9255aea0
We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_971ec472
A linear classifier trained on self-supervised representations learned by Sim- CLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-ofthe-art, matching the performance of a supervised … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_541be6d3
When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100× fewer labels.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3ceb808c
In this work, we introduce a simple framework for contrastive learning of visual representations, which we call SimCLR.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6ba07e85
We combine these findings to achieve a new state-of-the-art in self-supervised and semi-supervised learning on ImageNet ILSVRC-2012 (Russakovsky et al., 2015).

Simple Framework for Contrastive Learning of Visual Representations · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_40690645
Under the linear evaluation protocol, SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art (Hénaff et al., 2019).

Simple Framework for Contrastive Learning of Visual Representations · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6e34687
When fine-tuned with only 1% of the ImageNet labels, SimCLR achieves 85.8% top-5 accuracy, a relative improvement of 10% (Hénaff et al., 2019).

Simple Framework for Contrastive Learning of Visual Representations · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b6bfe323
As shown in Section 3, the combination of random crop and color distortion is crucial to achieve a good performance. • A neural network base encoderf (·) that extracts representation … (excerpt)

The Contrastive Learning Framework · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9120d472
While data augmentation has been widely used in both supervised and unsupervised representation learning (Krizhevsky et al., 3Although max performance is not reached in 100 epochs, reasonable results are achieved, … (excerpt)

Data Augmentation for Contrastive · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f87f6d20
We show that this complexity can be avoided by performing simple random cropping (with resizing) of target images, which creates a family of predictive tasks subsuming the above mentioned two, … (excerpt)

Simple Framework for Contrastive Learning of Visual Representations · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3bca8c4b
augmentation than supervised learning To further demonstrate the importance of the color augmentation, we adjust the strength of color augmentation as 5Supervised models are trained for 90 epochs; longer training … (excerpt)

Contrastive learning needs stronger data · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fa007303
Thus, our experiments show that unsupervised contrastive learning benefits from stronger (color) data augmentation than supervised learning.

Contrastive learning needs stronger data · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2e3f8066
Although previous work has reported that data augmentation is useful for self-supervised learning (Doersch et al., 2015; Bachman et al., 2019; Hénaff et al., 2019; Asano et al., 2019), we … (excerpt)

Contrastive learning needs stronger data · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6aee4b09
Unsupervised contrastive learning benefits (more) from bigger models Figure 7 shows, perhaps unsurprisingly, that increasing depth and width both improve performance.

Architectures for Encoder and Head · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ae308692
7Training longer does not improve supervised ResNets (see Appendix B.3).

Architectures for Encoder and Head · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7a8c3555
We observe that a nonlinear projection is better than a linear projection (+3%), and much better than no projection (>10%).

A nonlinear projection head improves the · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b3197049
Furthermore, even when nonlinear projection is used, the layer before the projection head, h, is still much better (>10%) than the layer after, z =g(h), which shows that the hidden … (excerpt)

A nonlinear projection head improves the · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d279f6c6
Linear evaluation (top-1) for models trained with different loss functions. “sh” means using semi-hard negative mining. 𝓁2 norm? τ Entropy Contrastive acc.

Simple Framework for Contrastive Learning of Visual Representations · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7340321e
Again, our approach significantly improves over state-of-the-art with both 1% and 10% of the labels.

Simple Framework for Contrastive Learning of Visual Representations · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0820a274
Interestingly, fine-tuning our pretrained ResNet-50 (2×, 4×) on full ImageNet are also significantly better then training from scratch (up to 2%, see Appendix B.2).

Simple Framework for Contrastive Learning of Visual Representations · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_741ce704
When fine-tuned, our self-supervised model significantly outperforms the supervised baseline on 5 datasets, whereas the supervised baseline is superior on only 2 (i.e. Pets and Flowers).

Simple Framework for Contrastive Learning of Visual Representations · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_112ffb66
In this work, we present a simple framework and its instantiation for contrastive visual representation learning.

Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ff08db81
Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402, 2013.

Simple Framework for Contrastive Learning of Visual Representations · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a99d034c
We find it helpful, as it improves our ResNet-50 trained for 100 epochs from 63.2% to 64.5%.

Simple Framework for Contrastive Learning of Visual Representations · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f345370d
The result is bolded if it is more than 0.5% better.

Batch Size and Training Steps · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dd9dbd25
A shown in Figure B.2, the performance seems to saturate with a batch size of 8192, while training longer can still significantly improve the performance.

Batch Size and Training Steps · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e9150074
Interestingly, when fine-tuned on full (100%) ImageNet training set, our ResNet (4 ×) model achieves 80.4% top-1 / 95.4% top-5 13, which is significantly better than that (78.4% top-1 / … (excerpt)

Broader composition of data augmentations further improves performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_27e037da
There is no improvement from fine-tuning for ResNet-50.

Broader composition of data augmentations further improves performance · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_104b215e
Stronger data augmentation slightly improves the accuracy of ResNet-50 (4×) but does not help on ResNet-50.

Effects of Longer Training for Supervised Models · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_05cf55ae
Importantly, our random initialization baselines are trained for 40,000 steps, which is sufficiently long to achieve near-maximal accuracy, as demonstrated in Figure 8 of Kornblith et al. (2019).

Simple Framework for Contrastive Learning of Visual Representations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5ca8615f
We found that, although stronger data augmentation and longer training time do not benefit accuracy on ImageNet, these models performed significantly better than a supervised baseline trained for 90 epochs … (excerpt)

Simple Framework for Contrastive Learning of Visual Representations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cda4882a
The supervised ResNet-50 baseline achieves 76.3% top-1 accuracy on ImageNet, vs. 69.3% for the self-supervised counterpart, while the ResNet-50 (4×) baseline achieves 78.3%, vs. 76.5% for the self-supervised model.

Simple Framework for Contrastive Learning of Visual Representations · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a0a4ac24
The supervised ResNet-50 model outperforms the self-supervised model on all datasets with linear evaluation, and most (10 of 12) datasets with fine-tuning.

R ESULTS WITH STANDARD RESNET · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e8ae8730
The self-supervised ResNet gets 69.3% top-1 accuracy, 6.8% worse than the supervised model in absolute terms, whereas the self-supervised ResNet (4×) model gets 76.5%, which is only 1.8% worse than … (excerpt)

Simple Framework for Contrastive Learning of Visual Representations · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5721e1b5
Our best model trained with batch size 1024 can achieve a linear evaluation accuracy of 94.0%, compared to 95.1% from the supervised baseline using the same architecture and batch size.

CIFAR-10 · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6375dcbe
However, the performance withτ = 0.1 improves as batch size increases, which may suggest a small shift of optimal temperature towards 0.1.

Simple Framework for Contrastive Learning of Visual Representations · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2002.05709) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.