clm_a262d696To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.
arXiv 2204.06125 · cs.CV
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
clm_a262d696To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.
clm_5ec0a528We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity.
res_f8e0d6fbTo achieve best results, diffusion models leverage a guidance technique [ 11, 24] which improves sample fidelity (for images, photorealism) at the cost of sample diversity.
res_4643502aWhile we can sample from the conditional distribution of the decoder directly, past work using diffusion models shows using guidance on the conditioning information [11, 24, 35] improves sample quality a lot.
res_692bd26cTo reduce training compute and improve numerical stability, we follow Rombach et al. [42] and train on random crops of images that are one-fourth the target size.
res_bb8505a9To improve sample quality we also enable sampling using classifier-free guidance for both the AR and diffusion prior, by randomly dropping this text conditioning information 10% of the time during training.
res_c4a83a72In particular, we find that the rank of the CLIP representation space is drastically reduced when training CLIP with SAM [15] while slightly improving evaluation metrics.
res_f6bc1b90Instead of using theϵ-prediction formulation from Ho et al. [25], we find it better to train our model to predict the unnoisedzi directly, and use a mean-squared error loss on this prediction: Lprior = Et∼[1,T ],z(t)i ∼qt [ ‖fθ(z(t)i ,t,y )−zi‖2] 3We swept over percentiles 50%, 70%, 85%, 95% and found 50% to be optimal in all experiments.
res_a41c899fIn all cases (Sections 5.2, 5.4, and 5.5), we find that the diffusion prior outperforms the AR prior for comparable model size and reduced training compute.
res_2efb675cReported figures are 95% confidence intervals of the probability that the unCLIP model specified by the row beats GLIDE.
res_63a65d18In Section 5.2, we observed that unCLIP achieves similar photorealism as GLIDE while maintaining more diversity, but that its caption matching capabilities were slightly worse.
res_91d00a10We use guidance scale 1.25 for the decoder for both the AR and diffusion prior, and achieve the best results using the diffusion prior. matching.
res_9c1be9c6We find that, compared to these other zero-shot models, unCLIP achieves a new state-of-the-art FID of 10.39 when sampling with the diffusion prior.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2204.06125) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.