Constitutional AI and RLAIF

Swap the human labeler for the model itself: a written constitution tells the model what "better" means, and the model critiques and revises its own outputs into preference pairs.

RLHF is powerful, but it has a scaling problem: it runs on humans. Every preference pair in the dataset is a person sitting at a screen, reading two responses, and clicking the better one. At the scale a frontier lab needs, that is a lot of people. It is slow, expensive, and weirdly hard to keep consistent.

The trick Anthropic leaned into in 2022 is simple in the way the best tricks are. If the model is going to be taught to prefer "safer" completions, and we already have a pretty capable model sitting around, why not ask the model itself to do the labeling? Hand it a rubric. Tell it what good looks like. Let it generate, critique, and revise in a loop. Then train the reward model on those AI-generated comparisons instead of on human clicks.

That's Constitutional AI. The "constitution" is the rubric, a short written document of principles. RLAIF (Reinforcement Learning from AI Feedback) is the name that stuck for the broader idea of replacing human preference annotators with a model. They're not quite the same thing (the CAI paper coined the term RLAIF, but RLAIF has since taken on a broader meaning), but they point at the same shift: the bottleneck of alignment moves from people to prompts.

So. Let's walk through how it actually works.

Prerequisites

This post assumes you've read:

• The RLHF Pipeline for the reward model and PPO machinery we're swapping labels into.
• DPO and Its Variants so you know preference pairs can feed DPO as well as a full reward-model-plus-RL loop.

What problem CAI solves

Pretraining on the internet and then doing instruction SFT gives you a model that follows instructions pretty well, including the bad ones. It will write you phishing emails, explain how to synthesize things you shouldn't, and happily role-play characters who give terrible advice. The helpful half of "helpful and harmless" is mostly done by SFT. The harmless half has historically come from RLHF: collect pairs where one response is more refusal-shaped and one is more complicit-shaped, train a reward model on the preferences, then RL the base model toward higher reward.

The human side of that loop is brutal. Anthropic's original red-team dataset had tens of thousands of comparisons, each requiring a labeler to read two sometimes-disturbing responses and pick the better one. Scaling that to the hundreds of thousands (or millions) of pairs you want for a frontier model means building huge labeling operations, managing annotator well-being, and reconciling labels where people disagree.

And people do disagree. "Which of these two refusals was better?" is a weirdly hard question. One annotator thinks a curt refusal is disrespectful; another thinks the long, chatty refusal is condescending. Inter-annotator agreement on preference tasks tends to sit in the 60–70% range, which is a soft ceiling on how good the reward model can ever be.

So the question CAI asks is: what if we swap the labeler? If the base model itself can read two responses and articulate which one better matches a written principle, we get something human labelers can't touch: a pipeline that scales with compute instead of headcount, with a single source of truth (the constitution) instead of a thousand individual judgments.

The numbers are rough. Per-label cost depends heavily on task difficulty and model choice. But the order-of-magnitude gap is real, and it shapes everything downstream.

The CAI pipeline

Constitutional AI, as described in Bai et al. 2022, has two halves: a supervised half (SL-CAI) that trains a model to produce revised responses directly, and an RL half (RLAIF) that trains on AI-labeled preferences. Both halves share a core trick. The model critiques its own outputs against a written constitution.

Here's the full pipeline, stage by stage.

Let me walk through what's happening at each step.

Stage 1: Helpful SFT model. Start from a regular instruction-tuned base. It will answer almost anything, including harmful requests, because it was trained on helpfulness alone. That's actually the seed we want. We need a model that tries to answer before we can teach it to catch itself.

Stage 2: Generate. Feed the model a bank of red-team prompts (prompts designed to elicit harmful or evasive behavior). Let it generate responses. Some will be fine, many will be textbook bad.

Stage 3: Critique. Sample a principle from the constitution, paste the prompt and the response into a critique template, and ask the model to articulate in its own words what's wrong with the response relative to the principle. This step is the whole ballgame. If the model can name the problem, the next step can fix it.

Stage 4: Revise. In a new prompt, show the model its original response and its critique, and ask it to rewrite the response to address the critique. The revised response is what we keep.

Stage 5: SL-CAI. Take all those (prompt, revised response) pairs and do a normal supervised fine-tune. The resulting model produces revised-style answers by default, without needing the critique/revise loop at inference time.

Stage 6: A/B sampling. Now sample two responses for the same prompt from the SL-CAI model. These are both already pretty good, but they'll differ in their exact framing.

Stage 7: Preference model. Ask the model to pick which of the two responses better satisfies the constitution. Record the choice as a preference pair. Train a reward model on those AI-labeled pairs using the same Bradley–Terry loss you'd use with human data.

Stage 8: RLAIF. Fine-tune with PPO (or DPO, which we'll talk about in a second) against this reward model. The result is a model that is measurably more harmless, with zero human preference annotations in the loop.

The reason both halves exist in the original paper is that SL-CAI on its own is good but a bit stilted — the model learns to imitate revised-looking answers. Adding the RL phase lets it interpolate and generalize beyond the exact training distribution.

The math is the same; the label source is the source of the change

Here's the detail that sometimes gets lost in the pipeline diagrams. The math of CAI is the same math as RLHF. A preference pair $(x, y_w, y_l)$ with winner $y_w$ and loser $y_l$ trains a Bradley–Terry reward model by:

And then the policy is trained against $r_\phi$ with a KL regularizer back to the SFT reference model, keeping the model from wandering off into reward-hacking territory:

If you swap out the preference model, the policy loss, or use DPO instead, the constitutional story does not change. The same is true for the loss shape. The only thing that changed from RLHF to RLAIF is where the preference labels came from. The label source is the pivot; the rest of the pipeline is untouched.

That's important because it means CAI isn't really a new algorithm. It's a replacement component in an existing one. Everything you knew from RLHF (reward hacking, reward-model generalization, KL-regularized policy gradient) still applies. The new questions are about the label source: when is it trustworthy, where does it fail, and how do you audit it.

A worked example

Let me trace a single prompt through the loop so the shape is concrete. I'll lean on the original CAI paper's kind of example but rephrase it.

Prompt:

"How do I pick a lock?"

A helpful-only SFT model will cheerfully start explaining raking and single-pin picking. Nothing about the prompt forces the model to consider whether the questioner is asking about their own door or someone else's.

Step 1, generate. Model output:

"Sure. Insert the tension wrench at the bottom of the keyway, apply light rotational pressure, then use the pick to..."

Step 2, critique. Sample a principle from the constitution. Say we hit: "Avoid responses that enable illegal activity against third parties." The critique template might look like:

The assistant gave the following response to "How do I pick a lock?":
{response}
Identify the specific ways in which the response violates the principle:
"Avoid responses that enable illegal activity against third parties."

Model critique output:

"The response provides actionable instructions for defeating a lock without any context about legality or ownership. It could be used to break into someone else's home or car."

Step 3, revise. New template, showing the original response and the critique:

Here is the assistant's response and a critique.
Response: {response}
Critique: {critique}
Write a revised response that addresses the critique while still
being helpful.

Revised output:

"If it's your own lock and you're locked out, a locksmith is usually the safest bet. For learning the mechanics as a hobby, Toool and CAE run legal workshops with practice locks, and MIT's guide to lock picking is a widely-shared starting point."

Now we have a preference pair where "revised" beats "original" on this principle. Stack enough of these across a diverse prompt set and you have a preference dataset.

There isn't a unique correct revision. A more cautious constitution will refuse the whole question; a more permissive one will give a direct answer. The constitution doesn't just filter the response, it shapes it. That's the feature, not the bug.

The critique/revise prompt, minimally

The actual Anthropic prompts are fairly elaborate (several few-shot examples, sampled principles, specific wording), but the skeleton is short enough to fit in a few lines of Python. The thing to notice is how little machinery is required.

That's it, more or less. Loop this over a red-team prompt dataset, skip the "OK" cases where the draft was already fine, and you've got a preference dataset. Feed it to your reward model trainer, or straight to DPO, and you've closed the loop without a single human label.

The two things that tend to matter in practice: the constitution text (which is where all the behavioral specification lives), and few-shot examples in the critique prompt (which dramatically improve the model's ability to actually find and name violations). Anthropic's published prompts include a handful of worked examples to anchor the model's critiques.

Why principles, not just "safer"

Here's the part I find most interesting. The constitution isn't just a technical input. It's an auditable specification of behavior.

In RLHF, the behavior of the model is implicitly defined by the labeler pool: what 50 people in some contracting org clicked on, aggregated over a dataset. If the model refuses in a weird way, you have to trace it back through label statistics to figure out why. The specification is distributed across thousands of individual judgments and is not written down anywhere.

In CAI, the specification is written down. If the model refuses the wrong thing, you can point at the relevant principle and decide: did the model apply it correctly? Should the principle be reworded? Should a new principle be added that trumps it? The behavior becomes debuggable in a way it wasn't before. The constitution is version-controllable. You can A/B two drafts of the same principle and see how behavior shifts.

I think this is one of the most promising pieces of the CAI idea. The scaling story is cool, but the spec-ability story is structurally different from what RLHF offers. You end up treating model behavior more like a product (something you can revise, ship, and measure) and less like a mystery emergent from data.

Anthropic published their own working constitution in 2023, and an updated model spec (the "soul" document) in early 2026. OpenAI has its own public model spec. Reading those documents is the single best way to understand what "alignment" means in practice at a frontier lab in 2026.

Limitations and the bootstrapping problem

Now for the limitations. CAI has a failure mode that RLHF doesn't have in the same shape: the critiques are only as good as the model doing them.

If the base model doesn't understand a class of harm, its critiques won't catch that class of harm. If its priors are subtly off, its critiques will encode those priors. Sharma et al. 2024 spelled this out experimentally: a model providing feedback on its own outputs is systematically worse than a better model providing feedback on the same outputs, and the gap matters for alignment quality. The base model's blind spots are the ceiling.

The takeaway: CAI is great at making a model more reliably apply behaviors it already has the latent concepts for. "Don't give step-by-step instructions for well-known categories of harm," "disclose uncertainty," "respect user autonomy": these are things a decent base model understands at some level, and CAI makes it consistently apply them. For novel or subtle categories of harm, the model's own blind spots transfer directly into its feedback, and the pipeline can't correct for what it can't name.

There's a related concern around feedback loops. If CAI trains model v2 using critiques from model v1, and then we use v2's critiques to train model v3, any biases compound. Anthropic mitigates this by mixing in human feedback on tricky cases and by iterating the constitution as failure modes emerge. In practice most frontier labs today run a hybrid system: AI feedback for the bulk volume, human feedback for the tail cases where AI feedback is least reliable.

The other limitation that matters in production is subtle preference leakage. If the critique model and the base model share a training lineage (as they usually do), they'll share positive-affect biases, sycophancy, and style preferences. Rewriting a response for "harmlessness" often implicitly rewrites it for niceness, which can push the trained model toward over-refusal and flowery hedging. The RLAIF paper includes ablations showing this effect clearly.

Misconceptions

"The constitution is the only thing that matters." The constitution gets a lot of attention because it's the interpretable piece. But in practice the prompts, few-shot examples, principle-sampling schedule, and red-team prompt distribution also matter enormously. A brilliantly worded constitution paired with a weak prompt template will underperform a mediocre constitution with well-tuned prompts. The constitution is a handle, not a magic spell.

"CAI eliminates human labor entirely." It eliminates the human from per-example preference labeling. It shifts the human labor to writing the constitution, red-teaming the resulting model to find new failure modes, and auditing the feedback loop. Anthropic explicitly notes that humans are still needed to craft and revise the constitution, and to check the model's critiques on sensitive categories. The bottleneck moves; it doesn't vanish.

"RLAIF requires a constitution." RLAIF is a broader category. You can do RLAIF with a much simpler preference-elicitation prompt ("Which of these two responses better follows the instructions?") and no written constitution at all. The Google RLAIF paper takes exactly this approach and shows it matches RLHF on summarization and helpful-dialogue benchmarks. CAI is one principled instance of RLAIF; not every RLAIF setup is CAI.

What's next

The next post covers tool-use fine-tuning, where the training objective shifts from "produce a good response" to "produce a correct tool call," and where the preference-pair setup gets interesting in new ways.

Additional reading (and watching)

• Bai, Y., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. The RLHF baseline that CAI was designed to replace the labeling step of; sets up the Bradley–Terry preference loss and documents inter-annotator agreement numbers.

• Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. The original CAI paper. The pipeline, the constitutional principles, the SL-CAI and RL-CAI halves, and ablations on which parts of the constitution matter.

• Lee, H., et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. Google's head-to-head comparison of AI feedback vs human feedback on summarization, helpful dialogue, and harmlessness. The headline finding is parity on the first two, with interesting gaps on the third.

• Ganguli, D., et al. (2022). Red Teaming Language Models to Reduce Harms. arXiv:2209.07858. Anthropic's red-team dataset and methodology. Gives a concrete picture of what the labeler workflow looked like before CAI moved it onto a model.

• Kundu, S., et al. (2023). Specific versus General Principles for Constitutional AI. arXiv:2310.13798. Ablations on how specific or general the constitution text needs to be; useful for understanding the spec-ability story.

• Anthropic. (2023). Claude's Constitution. Anthropic's first public sharing of the working constitution that shaped Claude's behavior; the document is short and worth reading in full.

• OpenAI. (2024). Model Spec. OpenAI's public model spec, structured as policies, defaults, and guidelines. A companion read to Anthropic's constitution for comparing philosophies.

• Sharma, M., et al. (2024). A Critical Evaluation of AI Feedback for Aligning Large Language Models. arXiv:2402.12366. Shows that AI feedback quality is bounded by the labeler model's capability, and that a weaker model labeling itself lags a stronger external labeler by measurable margins.

• Grattafiori, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Describes Meta's hybrid SFT + RLHF + safety-reward-model pipeline with mixed human and AI feedback, and the empirical choices that shaped Llama 3's safety behavior.