MearcAll papers

arXiv 2111.06377 · cs.CV

Masked Autoencoders Are Scalable Vision Learners

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_a1002c82
Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7a7e54e2
Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_510f597e
Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_15baeb6f
Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aacf1c08
To overcome this difference and encourage learning useful features, we show that a simple strategy works well in computer vision: masking a very high portion of random patches.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_29d2abe0
Driven by this analysis, we present a simple, effective, and scalable form of a masked autoencoder (MAE) for visual representation learning.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb4e4d7a
Under this design, a very high masking ratio (e.g., 75%) can achieve a win-win scenario: it optimizes accuracy while allowing the encoder to process only a small portion ( e.g., 25%) of patches.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8956f914
This can reduce overall pre-training time by 3× or more and likewise reduce memory consumption, enabling us to easily scale our MAE to large models.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f48fa2f6
With MAE pre-training, we can train datahungry models like ViT-Large/-Huge [16] on ImageNet-1K with improved generalization performance.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a8ec5ae8
With a vanilla ViT-Huge model, we achieve 87.8% accuracy when finetuned on ImageNet-1K.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_90606318
This outperforms all previous results that use only ImageNet-1K data.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fb585b9e
Reconstructions of ImageNet validation images using an MAE pre-trained with a masking ratio of 75% but applied on inputs with higher masking ratios.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fdc25107
The ratio of 75% is good for both linear probing and fine-tuning.

Main Properties · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4ca3da1c
Our masking ratios are also much higher than those in related works [6, 16, 2] in computer vision (20% to 50%).

Main Properties · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_29a9300e
All fine-tuning results in Figure 5 are better than training from scratch (82.5%).

Main Properties · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_89addba2
This design can yield up to 8% improvement in linear probing (Table 1a, ‘lin’).

Main Properties · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a9ad689
The decoder depth is less influential for improving fine-tuning (Table 1a, ‘ft’).

Main Properties · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_69a17da1
If the encoder uses mask tokens, it performs worse: its accuracy drops by 14% in linear probing.

Main Properties · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4fd21193
This tokenization improves fine-tuning accuracy by 0.4% vs. unnormalized pixels, but has no advantage vs. normalized pixels.

Main Properties · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f0d368c1
Our MAE with block-wise masking works reasonably well at a ratio of 50%, but degrades at a ratio of 75%.

Main Properties · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_717d8e90
By fine-tuning with a 448 size, we achieve 87.8% accuracy, using only IN1K data .

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4c77849c
We improve over the state-of-the-art by a nontrivial margin in the highly competitive benchmark of IN1K (no external data).

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_410e3a05
Comparing with BEiT [2], our MAE is more accurate while being simpler and faster.

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f7464f12
Moreover, our MAE is considerably faster (3.5× per epoch) than BEiT, for the reason as studied in Table 1c.

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b5da2a6f
2We observed the degradation also in BEiT with ViT-L: it produces 85.2% (tokens) and 83.5% (pixels), reproduced from the official code.

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_be719204
The MAE models in Table 3 are pre-trained for 1600 epochs for better accuracy (Figure 7).

Comparisons with Previous Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5163cd9b
Moreover, if we fine-tune only “half” of the last block ( i.e., its MLP sub-block), we can get 79.1%, much better than linear probing.

Partial Fine-tuning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4a4771c3
Fine-tuning a few blocks ( e.g., 4 or 6) can achieve accuracy close to full fine-tuning.

Partial Fine-tuning · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_271fca60
More significantly, with the larger ViT-L, our MAE pre-training outperforms supervised pre-training by 4.0 points (53.3 vs. 49.3).

Transfer Learning Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cac49307
Table 5 shows that our pretraining significantly improves results over supervised pretraining, e.g., by 3.7 points for ViT-L.

Transfer Learning Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ac1f2fad
Our pixel-based MAE also outperforms the token-based BEiT.

Transfer Learning Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_38fc738d
On Places, our MAE outperforms the previous best results [19, 40], which were obtained via pre-training on billions of images.

Transfer Learning Experiments · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c06af3c1
In §4.3 we have shown that linear probing accuracy and fine-tuning accuracy are largely uncorrelated and they have different focuses about linear separability.

Comparison on Linear Probing Results · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cb1c227c
In Table 13 we evaluate the robustness of our models on different variants of ImageNet validation sets.

Robustness Evaluation on ImageNet · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9c301afe
Our results outperform the previous best results (of specialized systems) by large margins.

Robustness Evaluation on ImageNet · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5ab40f1c
In contrast, supervised training performs much worse (Table 13 bottom; models described in A.2).

Robustness Evaluation on ImageNet · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cc3d1b22
For example, with ViT-H, our MAE pre-training is 35% better on IN-A (68.2% vs 33.1%) than the supervised counterpart.

Robustness Evaluation on ImageNet · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2111.06377) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.