MearcAll papers

arXiv 2212.08073 · cs.CL cs.AI

Constitutional AI: Harmlessness from AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, and 45 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_8394dbc8
Scaling Supervision We use the term ‘Scaling Supervision’ for techniques that leverage AI to help humans to more efficiently supervise AI, making it possible to train systems to behave in desirable ways (e.g. to be helpful, honest, and 1That is, helpfulness tends to increase harmfulness, since models are willing to obey pernicious requests, and conversely models trained to be harmless tend to be more evasive and generally less helpful.

Motivations · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1d77034f
2 Pretrained Base Constitutional RL (Pareto Improvement)With Chain of Thought Standard RLHFConstitutional SL Helpful-Only Helpful + Harmless Figure 2 We show harmlessness versus helpfulness Elo scores (higher is better, only differences are meaningful) computed from crowdworkers’ model comparisons for all 52B RL runs.

Motivations · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_072027de
Error bars are visible in Figure 3 but are suppressed here for clarity. harmless [Askell et al., 2021]) with a smaller quantity of higher quality human supervision.

Motivations · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8606c2e6
Our training process has two stages (see Figure 1), where the first supervised phase gets the model "ondistribution" and the second RL stage refines and significantly improves performance: (Supervised Stage) Critique → Revision → Supervised Learning In the first stage of the process, we first generate responses to harmfulness prompts using a helpful-only AI assistant.

The Constitutional AI Approach · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6b1e9dc5
We demonstrate constitutional methods to utilize a helpful RLHF model to train helpfuland harmless models (as discussed and defined in [Askell et al., 2021, Bai et al., 2022]) without using any human feedback labels for harmlessness: • We find that as language model capabilities improve, AI identification of harms improves significantly.

Contributions · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_80ecb10d
Furthermore, chain-of-thought reasoning improves this ability, and leads to evaluations that are becoming competitive with preference models trained on human feedback labels (see Figure 4). • We show that model-generated critiques and revisions can be applied repeatedly to progressively reduce harmfulness (see Figure 5).

Contributions · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_de61c607
Generating critiques improves harmlessness compared to simply generating revisions directly (Figure 7).

Contributions · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ccb09734
We use this method to specifically address the evasiveness of our prior human feedback based model [Bai et al., 2022]. • Using self-supervised preference labels for RL further improves model behavior as evaluated by crowdworkers (see Figures 2 and 3), equaling or exceeding the performance when using human feedback to evaluate harmlessness.

Contributions · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_40c3af8c
We find that models can now achieve well over 90% binary accuracy in their ability to predict the better response (see Figure 11 in the appendix), so for this paper we have written 217 more challenging comparisons, primarily focusing on more subtle tests of harmlessness, including examples where an evasive response is disfavored over a harmless and helpful message.

Evaluating the Potential for AI Supervision of HHH · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_33ac8a0a
In one case we formulate it as a preference model evaluation, and evaluate PMs that trained on several hundred thousand 6 human preference labels by the accuracy with which they assign a higher score to the better response.

Evaluating the Potential for AI Supervision of HHH · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_63508ded
We find that harmlessness and HH scores improve monotonically with respect to number of revisions, where revision 0 refers to the initial response, but pure helpfulness scores decrease.

Main Results · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2debc6ca
Preference Labels (Soft vs. Hard vs. Clamped) For RL-CAI without CoT, we found that using soft preference labels (i.e., normalized log-probabilities from the feedback model) led to much better results than hard labels (i.e., 0’s and 1’s).

Main Results · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6daf9cfd
Instead we found that clamping the probabilities at 20-80 percent slightly improved results, while clamping at 40-60 improved results further.

Main Results · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_42331fc2
We established two methods: (1) Constitutional AI which ‘bootstraps’ a helpful RLHF’s instruction-following abilities to critique and revise its own responses so as to remove harmful content, and (2) RL with model-generated labels for harmlessness, which further improves harmlessness.

Discussion · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2212.08073) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.