res_a1002c82Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task.
arXiv 2111.06377 · cs.CV
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
res_a1002c82Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task.
res_7a7e54e2Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy.
res_510f597eOur scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data.
clm_15baeb6fTransfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.
clm_aacf1c08To overcome this difference and encourage learning useful features, we show that a simple strategy works well in computer vision: masking a very high portion of random patches.
clm_29d2abe0Driven by this analysis, we present a simple, effective, and scalable form of a masked autoencoder (MAE) for visual representation learning.
res_bb4e4d7aUnder this design, a very high masking ratio (e.g., 75%) can achieve a win-win scenario: it optimizes accuracy while allowing the encoder to process only a small portion ( e.g., 25%) of patches.
res_8956f914This can reduce overall pre-training time by 3× or more and likewise reduce memory consumption, enabling us to easily scale our MAE to large models.
res_f48fa2f6With MAE pre-training, we can train datahungry models like ViT-Large/-Huge [16] on ImageNet-1K with improved generalization performance.
res_a8ec5ae8With a vanilla ViT-Huge model, we achieve 87.8% accuracy when finetuned on ImageNet-1K.
res_90606318This outperforms all previous results that use only ImageNet-1K data.
res_fb585b9eReconstructions of ImageNet validation images using an MAE pre-trained with a masking ratio of 75% but applied on inputs with higher masking ratios.
res_fdc25107The ratio of 75% is good for both linear probing and fine-tuning.
res_4ca3da1cOur masking ratios are also much higher than those in related works [6, 16, 2] in computer vision (20% to 50%).
res_29a9300eAll fine-tuning results in Figure 5 are better than training from scratch (82.5%).
res_89addba2This design can yield up to 8% improvement in linear probing (Table 1a, ‘lin’).
res_9a9ad689The decoder depth is less influential for improving fine-tuning (Table 1a, ‘ft’).
res_69a17da1If the encoder uses mask tokens, it performs worse: its accuracy drops by 14% in linear probing.
res_4fd21193This tokenization improves fine-tuning accuracy by 0.4% vs. unnormalized pixels, but has no advantage vs. normalized pixels.
res_f0d368c1Our MAE with block-wise masking works reasonably well at a ratio of 50%, but degrades at a ratio of 75%.
res_717d8e90By fine-tuning with a 448 size, we achieve 87.8% accuracy, using only IN1K data .
clm_4c77849cWe improve over the state-of-the-art by a nontrivial margin in the highly competitive benchmark of IN1K (no external data).
res_410e3a05Comparing with BEiT [2], our MAE is more accurate while being simpler and faster.
res_f7464f12Moreover, our MAE is considerably faster (3.5× per epoch) than BEiT, for the reason as studied in Table 1c.
res_b5da2a6f2We observed the degradation also in BEiT with ViT-L: it produces 85.2% (tokens) and 83.5% (pixels), reproduced from the official code.
res_be719204The MAE models in Table 3 are pre-trained for 1600 epochs for better accuracy (Figure 7).
res_5163cd9bMoreover, if we fine-tune only “half” of the last block ( i.e., its MLP sub-block), we can get 79.1%, much better than linear probing.
res_4a4771c3Fine-tuning a few blocks ( e.g., 4 or 6) can achieve accuracy close to full fine-tuning.
res_271fca60More significantly, with the larger ViT-L, our MAE pre-training outperforms supervised pre-training by 4.0 points (53.3 vs. 49.3).
res_cac49307Table 5 shows that our pretraining significantly improves results over supervised pretraining, e.g., by 3.7 points for ViT-L.
clm_ac1f2fadOur pixel-based MAE also outperforms the token-based BEiT.
res_38fc738dOn Places, our MAE outperforms the previous best results [19, 40], which were obtained via pre-training on billions of images.
res_c06af3c1In §4.3 we have shown that linear probing accuracy and fine-tuning accuracy are largely uncorrelated and they have different focuses about linear separability.
res_cb1c227cIn Table 13 we evaluate the robustness of our models on different variants of ImageNet validation sets.
clm_9c301afeOur results outperform the previous best results (of specialized systems) by large margins.
res_5ab40f1cIn contrast, supervised training performs much worse (Table 13 bottom; models described in A.2).
res_cc3d1b22For example, with ViT-H, our MAE pre-training is 35% better on IN-A (68.2% vs 33.1%) than the supervised counterpart.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2111.06377) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.