MearcAll papers

arXiv 1611.03530 · cs.LG

Understanding deep learning requires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_a2c94d5d
Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_070bfb28
More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.

O UR CONTRIBUTIONS · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7139efd5
We establish this fact for several different standard architectures trained on the CIFAR10 and ImageNet classification benchmarks.

O UR CONTRIBUTIONS · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b1d5a3f2
We discuss in further detail below how these observations rule out all of VC-dimension, Rademacher complexity, and uniform stability as possible explanations for the generalization performance of state-of-the-art neural networks.

O UR CONTRIBUTIONS · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f59bd4c3
We show that explicit forms of regularization, such as weight decay, dropout, and data augmentation, do not adequately explain the generalization error of neural networks.

O UR CONTRIBUTIONS · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_43278963
Our results show that even empirically training neural networks is not uniformly stable for many passes over the data.

R ELATED WORK · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5892e713
Specifically, the top-1 accuracy without regularization is 59.80%, while random guessing only achieves 0.1% top-1 accuracy on ImageNet.

T HE ROLE OF REGULARIZATION · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3f19a95e
More strikingly, with data-augmentation on but other explicit regularizers off, Inception is able to achieve a top-1 accuracy of 72.95%.

T HE ROLE OF REGULARIZATION · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a9637192
The normalization operator helps stablize the learning dynamics, but the impact on the generalization performance is only 3∼4%.

I MPLICIT REGULARIZATIONS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e992508b
We argue that what is more relevant in practice is the expressive power of neural networks on a finite sample of size n.

F INITE -SAMPLE EXPRESSIVITY · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cd4e2163
On MNIST with no preprocessing, we are able to achieve a test error of 1.2% by simply solving (3).

I MPLICIT REGULARIZATION : AN APPEAL TO LINEAR MODELS · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ba174b91
Simply applying a Gaussian kernel on pixels and using no regularization achieves 46% test error.

I MPLICIT REGULARIZATION : AN APPEAL TO LINEAR MODELS · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_08472d14
We argue that we have yet to discover a precise formal measure under which these enormous models are simple.

C ONCLUSION · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1611.03530) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.