clm_c264f2d0We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities.
arXiv 2011.13456 · cs.LG stat.ML
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_c264f2d0We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities.
res_f3eefc20Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 … (excerpt)
clm_01979539In addition, we propose two special methods not viable for general SDEs: (i) Predictor-Corrector (PC) samplers that combine numerical SDE solvers with score-based MCMC approaches, such as Langevin MCMC (Parisi, … (excerpt)
res_c005302dAlthough DDPM (Ho et al., 2020) was recently reported to achieve higher sample quality than SMLD (Song & Ermon, 2019; 2020), we show that with better architectures and new sampling … (excerpt)
res_be52b8bdIn addition, we propose a new SDE under our framework that achieves a likelihood value of 2.99 bits/dim on uniformly dequantized CIFAR-10 images, setting a new record on this task.
clm_fae63d12We propose to generalize this idea further to an infinite number of noise scales, such that perturbed data distributions evolve according to an SDE as the noise intensifies.
clm_41804a44To remedy this, we propose reverse diffusion samplers (details in Appendix E), which discretize the reverse-time SDE in the same way as the forward one, and thus can be readily … (excerpt)
clm_6bbce9f2We observe that our reverse diffusion sampler always outperform ancestral sampling, and corrector-only methods (C2000) perform worse than other competitors (P2000, PC1000) with the same computation (In fact, we need … (excerpt)
res_6d035fb2(deep, VE) 2.20 9.89 models trained with the continuous objective Eq. (7) on 256ˆ 256 LSUN images and the VE SDE, where PC samplers clearly surpass predictor-only samplers under comparable … (excerpt)
clm_2755b2a4As an example, we report negative log-likelihoods (NLLs) measured in bits/dim on the CIFAR-10 dataset in Table 2.
res_abcf2adbMain results: (i) For the same DDPM model in Ho et al. (2020), we obtain better bits/dim than ELBO, since our likelihoods are exact; (ii) Using the same architecture, we … (excerpt)
res_18403a6eOur optimal architecture for the VE SDE, named NCSN++, achieves an FID of 2.45 on CIFAR-10 with PC samplers, while our optimal architecture for the VP SDE, called DDPM++, achieves … (excerpt)
res_cb53630aBy switching to the continuous training objective in Eq. (7), and increasing the network depth, we can further improve sample quality for all models.
res_60cc42afWith all improvements together, we also obtain the first set of high-fidelity samples on CelebA-HQ 1024ˆ 1024 from score-based models (see Appendix H.3).
res_0c43c241(deep, sub-VP), similarly doubles the network depth and achieves a log-likelihood of 2.99 bits/dim with the continuous objective in Eq. (7).
res_3cd69e57Fig. 4 (right) shows results for inpainting and colorization achieved with unconditional time-dependent score-based models.
clm_4fc443eaInspired by the VP SDE, we propose a new SDE called the sub-VP SDE, namely dx“´ 12βptqx dt`bβptqp1´e´2şt0βpsqdsqdw.
clm_180bd7a7Our results demonstrate that for the same inputs, Model A and Model B provide encodings that are close in every dimension, despite having different model architectures and training runs.
res_057d1a05All architecture components introduced above can improve the performance of score-based models trained with VE SDEs, as shown in Fig. 10.
res_bc90abf7On both CIFAR-10 and CelebA, the additional components that we explored always improve the performance on average for VE SDEs.
res_bd498ca5The basic NCSN++ model with 4 residual blocks per resolution achieves an FID of 2.45 on CIFAR-10, whereas the basic DDPM++ model achieves an FID of 2.78.
res_60d535b4Switching from discrete training objectives to continuous ones in Eq. (7) further improves the FID values for all SDEs.
res_016ca73aThese changes improve the FID on CIFAR-10 from 2.45 to 2.38 for NCSN++ trained with the VE SDE, resulting in a model called “NCSN++ cont.”.
res_b9b11e54In addition, we can further improve the FID from 2.38 to 2.20 by doubling the number of residual blocks per resolution for NCSN++ cont., resulting in the model denoted as … (excerpt)
res_213d98eeWhen trained with the VP SDE, it improves the FID of 2.78 from DDPM++ to 2.55.
res_a6e26c6eWhen trained with the sub-VP SDE, it achieves an FID of 2.61.
res_c862afb628 Published as a conference paper at ICLR 2021 0.95M iterations, we can improve both FID and bits/dim for both VP and sub-VP SDEs, leading to a model “DDPM++ cont.
res_f127d39dWhen trained with the sub-VP SDE, it can achieve a likelihood of 2.99 bits/dim.
res_fdcf60bbEncouraged by the success of NCSN++ on CIFAR-10, we proceed to test it on 1024ˆ 1024 CelebA- HQ (Karras et al., 2018), a task that was previously only achievable by … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2011.13456) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.