Constitutional AI and RLAIF

Constitutional AI uses written principles to guide model-generated critiques, revisions, and comparisons. It asks whether some feedback needed for post-training can come from a model rather than a human annotator.

The original recipe has two distinct stages: supervised training on revised responses, then reinforcement learning using AI-labeled harmlessness preferences. Human helpfulness feedback remains part of that recipe.

Prerequisites

This post assumes you've read:

• The RLHF Pipeline for the reward model and PPO machinery we're swapping labels into.
• DPO and Its Variants so you know preference pairs can feed DPO as well as a full reward-model-plus-RL loop.

What problem CAI solves

Both SFT and preference training can shape helpfulness and safety behavior. Human comparisons are useful, but they take time and require clear rubrics and review.

A model can apply the same written rubric repeatedly, producing feedback at inference speed. That creates a different set of costs and errors: the judge may miss subtle issues, favor its own style, or apply a principle inconsistently.

The CAI pipeline

Constitutional AI, as described in Bai et al. 2022, has two halves: a supervised half (SL-CAI) that trains a model to produce revised responses directly, and an RL half (RLAIF) that trains on AI-labeled preferences. Both use written principles to guide model-generated feedback.

Stage 1: Starting assistant. Begin with a helpful model. Its existing safety behavior depends on prior training; being instruction-tuned does not imply it answers every harmful request.

Stage 2: Generate. Feed the model a bank of red-team prompts (prompts designed to elicit harmful or evasive behavior). Let it generate responses. Inspect the outputs to find cases that exercise the principles.

Stage 3: Critique. Sample a principle from the constitution, paste the prompt and the response into a critique template, and ask the model to articulate in its own words what's wrong with the response relative to the principle.

Stage 4: Revise. In a new prompt, show the model its original response and its critique, and ask it to rewrite the response to address the critique. The revised response is what we keep.

Stage 5: SL-CAI. Take all those (prompt, revised response) pairs and do a normal supervised fine-tune. The resulting model produces revised-style answers by default, without needing the critique/revise loop at inference time.

Stage 6: A/B sampling. Now sample two responses for the same prompt from the SL-CAI model. Their quality and framing can differ.

Stage 7: Preference model. Ask a model to compare sampled responses against a principle. Train a preference model using AI harmlessness labels together with human helpfulness preferences, as in the original paper.

Stage 8: RLAIF. Optimize the policy against that learned reward using RL. The paper avoided human harmlessness labels, not all human preferences. DPO on model-labeled pairs is a possible alternative procedure; it does not optimize directly against a separately trained reward model.

The paper evaluates supervised revisions and subsequent RL separately.

Using AI-labeled preferences in the loss

The original CAI recipe uses a reward-model loss and policy objective from RLHF. A preference pair $(x, y_w, y_l)$ with winner $y_w$ and loser $y_l$ trains a Bradley–Terry reward model by:

And then the policy is trained against $r_\phi$ with a KL regularizer back to the SFT reference model, discouraging drift from the reference:

The pairwise preference loss can stay the same when AI labels replace human labels. Changing the optimizer or replacing reward-model RL with DPO changes the training procedure too. Constitutional supervision describes how desired behavior is specified and labeled, not one mandatory optimizer.

Changing the label source does not remove reward hacking or reward-model generalization problems. It adds questions about the feedback model: which errors does it recognize, where do its judgments disagree with independent checks, and how consistently does it apply the principles?

A worked example

Here is a constructed example, not a transcript from a model or a quotation from the paper.

Prompt: “Should I take a new job for higher pay?”

Draft: “Yes. Higher pay is what matters most.”

Principle: “Respect the user’s autonomy and acknowledge relevant uncertainty.”

Critique: “The draft assumes pay is the user’s only priority and recommends a decision without learning the tradeoffs.”

Revision: “Higher pay may help, but the work, commute, team, and growth prospects matter too. Which of those are most important to you?”

The revision is a candidate supervised target. It is not automatically a verified improvement or a preference label. In the original RL stage, a model compares separately sampled response pairs.

A written principle makes a preference explicit, but reasonable principles can conflict. Test the resulting behavior on concrete cases and inspect whether the judge applied the intended priorities.

The critique/revise prompt, minimally

The example below generates a critique and revision, then sends the candidate through an acceptance check. The published Anthropic prompts also use sampled principles and few-shot examples.

This produces candidate SFT records after a separate acceptance check. Preference training needs comparative labels, not an assumption that every revision beats its draft.

The constitution, critique template, few-shot examples, and prompt distribution all help specify the supervision. The published prompts provide concrete examples; their effectiveness should be evaluated on the intended tasks.

Why principles, not just "safer"

A constitution is a written specification of intended behavior. It can be versioned, compared across revisions, and used to investigate why a judge preferred one response.

Human-feedback pipelines can also use explicit rubrics. The practical question is whether the chosen labeler, human or model, applies the rubric consistently. Neither a written specification nor a generated explanation proves that the trained policy follows it.

Limitations and the bootstrapping problem

Generator and judge can share blind spots. Sharma et al. (2024) studied a related confound: gains attributed to an AI-feedback RL stage often depended on using a stronger critic than the teacher that supplied SFT data. In their experiments, SFT from the stronger teacher could outperform the compared RLAIF pipelines. Comparing feedback methods therefore requires controlling teacher quality.

A useful evaluation includes issues the judge is expected to recognize and cases where recognition is uncertain. External evidence, executable checks, and independent human judgments can provide information missing from the feedback model.

Style preferences are another possible confound. A judge might reward verbose hedging or excessive refusal while appearing to reward safety. Separate harmful compliance, false refusal, factual correctness, and style in evaluation rather than treating one overall preference score as all four.

Misconceptions

"The constitution is the only thing that matters." Prompts, examples, principle sampling, and the training and evaluation distributions also affect the result. Compare changes on concrete cases rather than assuming a better-written principle guarantees better behavior.

"CAI eliminates human labor entirely." The original method replaced human harmlessness labels while retaining human helpfulness preferences. Humans also define principles, select data, and evaluate results.

"RLAIF requires a constitution." RLAIF is a broader category. You can do RLAIF with a much simpler preference-elicitation prompt ("Which of these two responses better follows the instructions?") and no written constitution at all. The Google RLAIF paper takes exactly this approach and shows it matches RLHF on summarization and helpful-dialogue benchmarks. CAI is one principled instance of RLAIF; not every RLAIF setup is CAI.

What's next

The next post covers tool-use fine-tuning: training on requests, tool calls, and results so the model can choose a tool and supply its arguments.

Additional reading (and watching)

• Bai, Y., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. The RLHF baseline that CAI was designed to replace the labeling step of; sets up the Bradley–Terry preference loss and documents inter-annotator agreement numbers.

• Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. The original CAI paper. The pipeline, the constitutional principles, the SL-CAI and RL-CAI halves, and ablations on which parts of the constitution matter.

• Lee, H., et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. Google's head-to-head comparison of AI feedback vs human feedback on summarization, helpful dialogue, and harmlessness. Reports task-specific comparisons of the two feedback sources.

• Ganguli, D., et al. (2022). Red Teaming Language Models to Reduce Harms. arXiv:2209.07858. Anthropic's red-team dataset and methodology. Gives a concrete picture of what the labeler workflow looked like before CAI moved it onto a model.

• Kundu, S., et al. (2023). Specific versus General Principles for Constitutional AI. arXiv:2310.13798. Ablations on how specific or general the constitution text needs to be.

• Anthropic. (2023). Claude's Constitution. The published principles used to guide critiques and revisions.

• OpenAI. (2024). Model Spec. OpenAI's public model spec, structured as policies, defaults, and guidelines. A companion read to Anthropic's constitution for comparing philosophies.

• Sharma, A., et al. (2024). A Critical Evaluation of AI Feedback for Aligning Large Language Models. arXiv:2402.12366. Examines teacher/critic strength as a confound in comparisons of SFT and RLAIF.

• Grattafiori, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Describes Meta’s SFT, rejection sampling, DPO, and safety-data procedures.