MearcAll papers

arXiv 1911.05722 · cs.CV

Momentum Contrast for Unsupervised Visual Representation Learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_9cec3e17
MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d1cd14c
We show that in 7 downstream tasks related to detection or segmentation, MoCo unsupervised pre-training can surpass its ImageNet supervised counterpart, in some cases by nontrivial margins.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a79e8193
We propose a momentum update to address this issue.

Momentum Contrast · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ea9ad6b6
A similar trend has been observed in [61, 56] under the memory bank mechanism, while here we show that this trend is more general and can be seen in all … (excerpt)

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ac88ac4e
But it is 2.6% worse than MoCo.

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_44ffb4bd
Note the memory bank result of 58.0% reflects our improved implementation of [61].2 Ablation: momentum.

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_41059ed4
The table below shows ResNet-50 accuracy with different MoCo momentum values ( m in Eqn.(2)) used in pre-training (K = 4096 here) : momentum m 0 0.9 0.99 0.999 0.9999 … (excerpt)

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ca7015b8
MoCo with R50 performs competitively and achieves 60.6% accuracy, better than all competitors of similar model sizes (∼24M).

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ae888a2f
MoCo benefits from larger models and achieves 68.6% accuracy with R50w4×.

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2dde224d
Notably, we achieve competitive results using astandard ResNet-50 and require no specific architecture designs,e.g., 2Here 58.0% is with InfoNCE and K=65536.

Experiments · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5d1dc942
For R50-dilated- C5 (Table 2a), MoCo pre-trained on IN-1M is comparable to the supervised pre-training counterpart, and MoCo pretrained on IG-1B surpasses it.

PASCAL VOC Object Detection · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_02ac533d
We point out that these results are partially because we establish solid detection baselines for contrastive learning.

PASCAL VOC Object Detection · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6bac1dcc
In green are the gaps of at least +0.5 point. †: this entry is with BN frozen, which improves results; see main text. with ImageNet supervised pre-training: COCO keypoint detection … (excerpt)

More Downstream Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_aa057dc8
MoCo with IG-1B surpasses it in all metrics.

More Downstream Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_13dd269b
In sum, MoCo can outperform its ImageNet supervised pre-training counterpart in 7 detection or segmentation tasks.5 Besides, MoCo is on par on Cityscapes instance segmentation, and lags behind on VOC … (excerpt)

More Downstream Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d57c193d
MoCo’s improvement from IN-1M to IG-1B is consistently noticeable but relatively small, suggesting that the larger-scale data may not be fully exploited.

Discussion and Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ca1d7558
MoCo has 24.1 AP mk with IN-1M and 24.9 APmk with IG-1B, both outperforming the supervised pretraining counterpart under the same tunable BN setting.

Implementation: LVIS instance segmentation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_37f7f73b
Under the best individual settings, MoCo can still outperform the supervised pre-training case (24.9 vs. 24.4, as reported in Table 6 in Sec 4.2).

Implementation: LVIS instance segmentation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ea2a1420
IN-1M MoCoIN-1M MoCoIG-1B accuracy (%) 61.8 66.1 65.6 65.8 MoCo is ∼4% better than training from random initialization, and is closely comparable with its ImageNet supervised counterpart.

Implementation: Semantic segmentation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1239acb5
But unsupervised pre-training in the separate, unlabeled dataset of IG-1B represents a typical scenario: in this case, MoCo improves by 0.8%. pre-train APbb APbb50 APbb75 APmk APmk50 APmk75 random init.

Fine-tuning in ImageNet · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1c2c2662
We observe: (i) fine-tuning with ImageNet-supervised pre-training still has improvements (41.9 AP bb); (ii) training from scratch largely catches up (41.4 AP bb); (iii) the MoCo counterparts improve further (e.g., … (excerpt)

Fine-tuning in ImageNet · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1911.05722) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.