MearcAll papers

arXiv 2204.06125 · cs.CV

Hierarchical Text-Conditional Image Generation with CLIP Latents

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_a262d696
To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5ec0a528
We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f8e0d6fb
To achieve best results, diffusion models leverage a guidance technique [ 11, 24] which improves sample fidelity (for images, photorealism) at the cost of sample diversity.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4643502a
While we can sample from the conditional distribution of the decoder directly, past work using diffusion models shows using guidance on the conditioning information [11, 24, 35] improves sample quality a lot.

Decoder · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_692bd26c
To reduce training compute and improve numerical stability, we follow Rombach et al. [42] and train on random crops of images that are one-fourth the target size.

Decoder · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb8505a9
To improve sample quality we also enable sampling using classifier-free guidance for both the AR and diffusion prior, by randomly dropping this text conditioning information 10% of the time during training.

Prior · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c4a83a72
In particular, we find that the rank of the CLIP representation space is drastically reduced when training CLIP with SAM [15] while slightly improving evaluation metrics.

Prior · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f6bc1b90
Instead of using theϵ-prediction formulation from Ho et al. [25], we find it better to train our model to predict the unnoisedzi directly, and use a mean-squared error loss on this prediction: Lprior = Et∼[1,T ],z(t)i ∼qt [ ‖fθ(z(t)i ,t,y )−zi‖2] 3We swept over percentiles 50%, 70%, 85%, 95% and found 50% to be optimal in all experiments.

Prior · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a41c899f
In all cases (Sections 5.2, 5.4, and 5.5), we find that the diffusion prior outperforms the AR prior for comparable model size and reduced training compute.

Importance of the Prior · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2efb675c
Reported figures are 95% confidence intervals of the probability that the unCLIP model specified by the row beats GLIDE.

Human Evaluations · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_63a65d18
In Section 5.2, we observed that unCLIP achieves similar photorealism as GLIDE while maintaining more diversity, but that its caption matching capabilities were slightly worse.

Improved Diversity-Fidelity Trade-off with Guidance · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_91d00a10
We use guidance scale 1.25 for the decoder for both the AR and diffusion prior, and achieve the best results using the diffusion prior. matching.

Improved Diversity-Fidelity Trade-off with Guidance · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c1be9c6
We find that, compared to these other zero-shot models, unCLIP achieves a new state-of-the-art FID of 10.39 when sampling with the diffusion prior.

Comparison on MS-COCO · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2204.06125) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.