Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.
Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
We demonstrate that a standard ConvNet model can achieve the same level of scalability as hierarchical vision Transformers while being much simpler in design. visual feature learning.
Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions
One primary focus of ViT is on the scaling behavior: with the help of larger model and dataset sizes, Transformers can outperform standard ResNets by a significant margin.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
To do this, we start with a standard ResNet (e.g. ResNet- 50) trained with an improved procedure.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
As a result, we propose a family of pure ConvNets dubbed ConvNeXt.
Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions
In the end, our pure ConvNet model, named ConvNeXt, can outperform the Swin Transformer. follows.
Modernizing a ConvNet: a Roadmap · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
We first train it with similar training techniques used to train vision Transformers and obtain much improved results compared to the original ResNet-50.
Modernizing a ConvNet: a Roadmap · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
In Figure 2, we show the procedure and the results we are able to achieve with each step of the “network modernization”.
Modernizing a ConvNet: a Roadmap · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
This improves the model accuracy from 78.8% to 79.4%.
Macro Design · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions
In this part, we attempt to adopt the idea of ResNeXt [87], which has a better FLOPs/accuracy trade-off than a vanilla ResNet.
ResNeXt-ify · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions
Despite the increased FLOPs for the depthwise convolution layer, this change reduces the whole network FLOPs to 4.6G, due to the significant FLOPs reduction in the downsampling residual blocks’ shortcut 1×1 conv layer.
Inverted Bottleneck · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions
Interestingly, this results in slightly improved performance (80.5% to 80.6%).
Inverted Bottleneck · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions
This intermediate step reduces the FLOPs to 4.1G, resulting in a temporary performance degradation to 79.9%.
Large Kernel Sizes · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions
Additionally, we observe that the benefit of larger kernel sizes reaches a saturation point at 7×7.
Large Kernel Sizes · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
This process improves the result by 0.7% to 81.3%, practically matching the performance of Swin-T.
Micro Design · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
This further boosts the performance to 81.4%, already surpassing Swin-T’s result.
Micro Design · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
For simplicity, we note the linear layers in Transformer MLP blocks also as “1×1 convs” since they are equivalent. that we have even fewer normalization layers per block than Transformers, as empirically we find that adding one additional BN layer at the beginning of the block does not improve the performance.
Micro Design · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
We observe that our ConvNet model does not have any difficulties training with LN; in fact, the performance is slightly better, obtaining an accuracy of 81.5%.
Micro Design · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
In ResNet, the spatial downsampling is achieved by the residual block at the start of each stage, using 3×3 conv with stride 2 (and 1×1 conv with stride 2 at the shortcut connection).
Micro Design · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions
We can improve the accuracy to 82.0%, significantly exceeding Swin-T’s 81.3%.
Micro Design · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
We have finished our first “playthrough” and discovered ConvNeXt, a pure ConvNet, that can outperform the Swin Transformer for ImageNet-1K classification in this compute regime.
Micro Design · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
ConvNeXt also outperforms Swin Transformer of similar complexities across the model imagesize #param.
Results · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions
A highlight from the results is ConvNeXt-B at 384 2: it outperforms Swin-B by 0.6% (85.1% vs. 84.5%), but with 12.5% higher inference throughput (95.7 vs. 85.1 image/s).
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Additionally, we observe an improved result of 85.5% when further scaling to ConvNeXt-L.
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
We present results with models fine-tuned from ImageNet-22K pre-training at Table 1 (lower).
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Our results demonstrate that properly designed ConvNets are not inferior to vision Transformers when pre-trained with large dataset — ConvNeXts still perform on par or better than similarly-sized Swin Transformers, with slightly higher throughput.
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
Additionally, our ConvNeXt-XL model achieves an accuracy of 87.8% — a decent improvement over ConvNeXt-L at 3842, demonstrating that ConvNeXts are scalable architectures.
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
However, with ImageNet-22K pre-training, ConvNeXt is able to outperform EfficientNetV2, further demonstrating the importance of large-scale training.
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
We use the supervised training results from DeiT [73] for ViT-S/B and MAE [26] for ViT-L, as they employ improved training procedures over the original ViTs [20].
Results · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions
When scaled up to bigger models (ConvNeXt-B/L/XL) pre-trained on ImageNet-22K, in many cases ConvNeXt is significantly better(e.g. +1.0 AP) than Swin Transformers in terms of box and mask AP.
Empirical Evaluation on Downstream Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
This is true for both classification and other tasks requiring higher-resolution inputs (see Table 1,3 for comparisons of throughput/FPS).
Empirical Evaluation on Downstream Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions
We propose ConvNeXts, a pure ConvNet model that can compete favorably with state-of-the-art hierarchical vision Transformers across multiple computer vision benchmarks, while retaining the simplicity and efficiency of standard ConvNets.
Conclusions · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
The only exception is ConvNeXt-L pre-trained on ImageNet-1K, where the model accuracy is significantly lower than the EMA accuracy due to overfitting, and we select its best EMA model during pre-training as the starting point for fine-tuning.
ImageNet Fine-tuning · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions
Additional robustness evaluation results for ConvNeXt models are presented in Table 8.
Robustness Evaluation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
We directly test our ImageNet-1K trained/fine-tuned classification models on several robustness benchmark datasets such as ImageNet-A [33], ImageNet-R [30], ImageNet-Sketch [78] and ImageNet- C/¯C [31, 48] datasets.
Robustness Evaluation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
With extra ImageNet-22K data, ConvNeXt- XL demonstrates strong domain generalization capabilities (e.g. achieving 69.3%/68.2%/55.0% accuracy on ImageNet- A/R/Sketch benchmarks, respectively).
Robustness Evaluation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
One interesting difference is that inverting dimensions brings a larger improvement at ResNet-200 regime than at ResNet-50 regime (+0.79% vs. +0.14%).
Modernizing ResNets: detailed results · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions
Using fewer normalization layers also has a bigger gain compared with the ResNet-50 regime (+0.46% vs. +0.14%). model IN-1K acc.
Modernizing ResNets: detailed results · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions
Swin Transformers and ConvNeXts both achieve faster inference throughput than V100 GPUs, but ConvNeXts’ advantage is now significantly greater, sometimes up to 49% faster.
Benchmarking on A100 GPUs · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions
Using TF32 data format and “channel last” memory layout, ConvNeXt enjoys up to ∼49% higher throughput compared with a Swin Transformer with similar FLOPs.
Benchmarking on A100 GPUs · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions