MearcAll papers

arXiv 2011.13456 · cs.LG stat.ML

Score-Based Generative Modeling through Stochastic Differential Equations

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_c264f2d0
We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f3eefc20
Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 … (excerpt)

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_01979539
In addition, we propose two special methods not viable for general SDEs: (i) Predictor-Corrector (PC) samplers that combine numerical SDE solvers with score-based MCMC approaches, such as Langevin MCMC (Parisi, … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c005302d
Although DDPM (Ho et al., 2020) was recently reported to achieve higher sample quality than SMLD (Song & Ermon, 2019; 2020), we show that with better architectures and new sampling … (excerpt)

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_be52b8bd
In addition, we propose a new SDE under our framework that achieves a likelihood value of 2.99 bits/dim on uniformly dequantized CIFAR-10 images, setting a new record on this task.

I NTRODUCTION · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fae63d12
We propose to generalize this idea further to an infinite number of noise scales, such that perturbed data distributions evolve according to an SDE as the noise intensifies.

S CORE -BASED GENERATIVE MODELING WITH SDE S · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_41804a44
To remedy this, we propose reverse diffusion samplers (details in Appendix E), which discretize the reverse-time SDE in the same way as the forward one, and thus can be readily … (excerpt)

G ENERAL -PURPOSE NUMERICAL SDE SOLVERS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6bbce9f2
We observe that our reverse diffusion sampler always outperform ancestral sampling, and corrector-only methods (C2000) perform worse than other competitors (P2000, PC1000) with the same computation (In fact, we need … (excerpt)

P REDICTOR -CORRECTOR SAMPLERS · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6d035fb2
(deep, VE) 2.20 9.89 models trained with the continuous objective Eq. (7) on 256ˆ 256 LSUN images and the VE SDE, where PC samplers clearly surpass predictor-only samplers under comparable … (excerpt)

P REDICTOR -CORRECTOR SAMPLERS · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2755b2a4
As an example, we report negative log-likelihoods (NLLs) measured in bits/dim on the CIFAR-10 dataset in Table 2.

P ROBABILITY FLOW AND CONNECTION TO NEURAL ODE S · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_abcf2adb
Main results: (i) For the same DDPM model in Ho et al. (2020), we obtain better bits/dim than ELBO, since our likelihoods are exact; (ii) Using the same architecture, we … (excerpt)

P ROBABILITY FLOW AND CONNECTION TO NEURAL ODE S · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_18403a6e
Our optimal architecture for the VE SDE, named NCSN++, achieves an FID of 2.45 on CIFAR-10 with PC samplers, while our optimal architecture for the VP SDE, called DDPM++, achieves … (excerpt)

A RCHITECTURE IMPROVEMENTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cb53630a
By switching to the continuous training objective in Eq. (7), and increasing the network depth, we can further improve sample quality for all models.

A RCHITECTURE IMPROVEMENTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_60cc42af
With all improvements together, we also obtain the first set of high-fidelity samples on CelebA-HQ 1024ˆ 1024 from score-based models (see Appendix H.3).

A RCHITECTURE IMPROVEMENTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0c43c241
(deep, sub-VP), similarly doubles the network depth and achieves a log-likelihood of 2.99 bits/dim with the continuous objective in Eq. (7).

A RCHITECTURE IMPROVEMENTS · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3cd69e57
Fig. 4 (right) shows results for inpainting and colorization achieved with unconditional time-dependent score-based models.

C ONTROLLABLE GENERATION · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4fc443ea
Inspired by the VP SDE, we propose a new SDE called the sub-VP SDE, namely dx“´ 12βptqx dt`bβptqp1´e´2şt0βpsqdsqdw.

VE, VP AND SUB -VP SDE S · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_180bd7a7
Our results demonstrate that for the same inputs, Model A and Model B provide encodings that are close in every dimension, despite having different model architectures and training runs.

U NIQUELY IDENTIFIABLE ENCODING · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_057d1a05
All architecture components introduced above can improve the performance of score-based models trained with VE SDEs, as shown in Fig. 10.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bc90abf7
On both CIFAR-10 and CelebA, the additional components that we explored always improve the performance on average for VE SDEs.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bd498ca5
The basic NCSN++ model with 4 residual blocks per resolution achieves an FID of 2.45 on CIFAR-10, whereas the basic DDPM++ model achieves an FID of 2.78.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_60d535b4
Switching from discrete training objectives to continuous ones in Eq. (7) further improves the FID values for all SDEs.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_016ca73a
These changes improve the FID on CIFAR-10 from 2.45 to 2.38 for NCSN++ trained with the VE SDE, resulting in a model called “NCSN++ cont.”.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b9b11e54
In addition, we can further improve the FID from 2.38 to 2.20 by doubling the number of residual blocks per resolution for NCSN++ cont., resulting in the model denoted as … (excerpt)

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_213d98ee
When trained with the VP SDE, it improves the FID of 2.78 from DDPM++ to 2.55.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a6e26c6e
When trained with the sub-VP SDE, it achieves an FID of 2.61.

R ESULTS ON CIFAR-10 · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c862afb6
28 Published as a conference paper at ICLR 2021 0.95M iterations, we can improve both FID and bits/dim for both VP and sub-VP SDEs, leading to a model “DDPM++ cont.

R ESULTS ON CIFAR-10 · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f127d39d
When trained with the sub-VP SDE, it can achieve a likelihood of 2.99 bits/dim.

R ESULTS ON CIFAR-10 · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fdcf60bb
Encouraged by the success of NCSN++ on CIFAR-10, we proceed to test it on 1024ˆ 1024 CelebA- HQ (Karras et al., 2018), a task that was previously only achievable by … (excerpt)

H IGH RESOLUTION IMAGES · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2011.13456) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.