res_9255aea0We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss … (excerpt)
arXiv 2002.05709 · cs.LG cs.CV stat.ML
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_9255aea0We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss … (excerpt)
res_971ec472A linear classifier trained on self-supervised representations learned by Sim- CLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-ofthe-art, matching the performance of a supervised … (excerpt)
res_541be6d3When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100× fewer labels.
clm_3ceb808cIn this work, we introduce a simple framework for contrastive learning of visual representations, which we call SimCLR.
res_6ba07e85We combine these findings to achieve a new state-of-the-art in self-supervised and semi-supervised learning on ImageNet ILSVRC-2012 (Russakovsky et al., 2015).
res_40690645Under the linear evaluation protocol, SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art (Hénaff et al., 2019).
res_e6e34687When fine-tuned with only 1% of the ImageNet labels, SimCLR achieves 85.8% top-5 accuracy, a relative improvement of 10% (Hénaff et al., 2019).
res_b6bfe323As shown in Section 3, the combination of random crop and color distortion is crucial to achieve a good performance. • A neural network base encoderf (·) that extracts representation … (excerpt)
res_9120d472While data augmentation has been widely used in both supervised and unsupervised representation learning (Krizhevsky et al., 3Although max performance is not reached in 100 epochs, reasonable results are achieved, … (excerpt)
clm_f87f6d20We show that this complexity can be avoided by performing simple random cropping (with resizing) of target images, which creates a family of predictive tasks subsuming the above mentioned two, … (excerpt)
res_3bca8c4baugmentation than supervised learning To further demonstrate the importance of the color augmentation, we adjust the strength of color augmentation as 5Supervised models are trained for 90 epochs; longer training … (excerpt)
clm_fa007303Thus, our experiments show that unsupervised contrastive learning benefits from stronger (color) data augmentation than supervised learning.
clm_2e3f8066Although previous work has reported that data augmentation is useful for self-supervised learning (Doersch et al., 2015; Bachman et al., 2019; Hénaff et al., 2019; Asano et al., 2019), we … (excerpt)
res_6aee4b09Unsupervised contrastive learning benefits (more) from bigger models Figure 7 shows, perhaps unsurprisingly, that increasing depth and width both improve performance.
res_ae3086927Training longer does not improve supervised ResNets (see Appendix B.3).
res_7a8c3555We observe that a nonlinear projection is better than a linear projection (+3%), and much better than no projection (>10%).
res_b3197049Furthermore, even when nonlinear projection is used, the layer before the projection head, h, is still much better (>10%) than the layer after, z =g(h), which shows that the hidden … (excerpt)
res_d279f6c6Linear evaluation (top-1) for models trained with different loss functions. “sh” means using semi-hard negative mining. 𝓁2 norm? τ Entropy Contrastive acc.
res_7340321eAgain, our approach significantly improves over state-of-the-art with both 1% and 10% of the labels.
res_0820a274Interestingly, fine-tuning our pretrained ResNet-50 (2×, 4×) on full ImageNet are also significantly better then training from scratch (up to 2%, see Appendix B.2).
res_741ce704When fine-tuned, our self-supervised model significantly outperforms the supervised baseline on 5 datasets, whereas the supervised baseline is superior on only 2 (i.e. Pets and Flowers).
clm_112ffb66In this work, we present a simple framework and its instantiation for contrastive visual representation learning.
res_ff08db81Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402, 2013.
res_a99d034cWe find it helpful, as it improves our ResNet-50 trained for 100 epochs from 63.2% to 64.5%.
res_f345370dThe result is bolded if it is more than 0.5% better.
res_dd9dbd25A shown in Figure B.2, the performance seems to saturate with a batch size of 8192, while training longer can still significantly improve the performance.
res_e9150074Interestingly, when fine-tuned on full (100%) ImageNet training set, our ResNet (4 ×) model achieves 80.4% top-1 / 95.4% top-5 13, which is significantly better than that (78.4% top-1 / … (excerpt)
res_27e037daThere is no improvement from fine-tuning for ResNet-50.
res_104b215eStronger data augmentation slightly improves the accuracy of ResNet-50 (4×) but does not help on ResNet-50.
res_05cf55aeImportantly, our random initialization baselines are trained for 40,000 steps, which is sufficiently long to achieve near-maximal accuracy, as demonstrated in Figure 8 of Kornblith et al. (2019).
res_5ca8615fWe found that, although stronger data augmentation and longer training time do not benefit accuracy on ImageNet, these models performed significantly better than a supervised baseline trained for 90 epochs … (excerpt)
res_cda4882aThe supervised ResNet-50 baseline achieves 76.3% top-1 accuracy on ImageNet, vs. 69.3% for the self-supervised counterpart, while the ResNet-50 (4×) baseline achieves 78.3%, vs. 76.5% for the self-supervised model.
res_a0a4ac24The supervised ResNet-50 model outperforms the self-supervised model on all datasets with linear evaluation, and most (10 of 12) datasets with fine-tuning.
res_e8ae8730The self-supervised ResNet gets 69.3% top-1 accuracy, 6.8% worse than the supervised model in absolute terms, whereas the self-supervised ResNet (4×) model gets 76.5%, which is only 1.8% worse than … (excerpt)
res_5721e1b5Our best model trained with batch size 1024 can achieve a linear evaluation accuracy of 94.0%, compared to 95.1% from the supervised baseline using the same architecture and batch size.
res_6375dcbeHowever, the performance withτ = 0.1 improves as batch size increases, which may suggest a small shift of optimal temperature towards 0.1.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2002.05709) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.