Decoder-Only vs. Encoder-Decoder: Architecture Trade-offs

Three ways to wire a transformer. Modern LLMs almost all use the decoder-only path.

The problem

You have all the transformer building blocks now: residual streams, self-attention with Q/K/V, multi-head projections, causal masking, positional encoding, FFN blocks, layer norms. The question is how to assemble them. There are three valid wiring patterns, and the field spent about five years trying all three before largely settling on one.

I want to walk through why, because the reasoning tells you a lot about what actually matters when you scale a language model. The answer comes down to which wiring is simplest to scale, simplest to serve, and most naturally aligned with the training objective that turned out to matter most: next-token prediction.

Prerequisites

You should be comfortable with self-attention and Q/K/V (Post 4.1), causal masking (Post 4.3), and the layer norm / residual stream assembly from Post 4.6. If you remember Bahdanau attention from the seq2seq post, the cross-attention section will feel familiar, but it is not required.

Three families

Every major transformer model falls into one of three architectural families. They all use the same building blocks we've covered: residual streams, self-attention heads, feed-forward networks, layer norms, positional encodings. The difference is how they wire these blocks together and, critically, what kind of attention mask they apply.

The attention mask is the fork in the road. A bidirectional mask lets every token see every other token. A causal mask restricts each token to only see positions before it. That one choice cascades into completely different training objectives, completely different capabilities, and completely different use cases.

Encoder-only: BERT and friends

The encoder-only architecture is the simplest to understand. You take a sequence of tokens, run them through a stack of transformer layers, and get back a sequence of hidden states. Nothing more.

The key property: bidirectional attention. Every token can attend to every other token in the sequence, with no masking and no restrictions. When the word "bank" shows up at position 5, it gets to look at position 12 where "river" sits. It also gets to look at position 2 where "investment" sits. It sees everything, in both directions, simultaneously.

This is great for understanding. If you want to classify a sentence, extract named entities, or produce an embedding for retrieval, bidirectional context is exactly what you want. BERT demonstrated this in 2018 and basically took over NLP for the next two years.

BERT's training objective is called masked language modeling (MLM). You randomly replace about 15% of input tokens with a special [MASK] token, and the model has to predict what the masked tokens were. Because the model sees context in both directions (left and right of the mask), it builds strong representations of what words mean in context. There's also a next-sentence prediction task, though later work (RoBERTa) showed you could drop that without losing much.

But there's a catch. Because every token sees every other token, there's no causal structure. The model has no notion of "what comes next" because it already sees what comes next. You can't use it for autoregressive text generation. You can't ask it to continue a sentence token by token, because during training it never learned to predict the next token given only the preceding context. It learned to predict masked tokens given surrounding context. Those are different skills.

So encoder-only models are great at understanding language, they just can't produce it.

The BERT era (roughly 2019-2021) was a fascinating time in NLP. The standard workflow was: take a pre-trained BERT model, slap a classification head on top, fine-tune on your task-specific dataset, deploy. It worked shockingly well. BERT crushed benchmarks in question answering, sentiment analysis, named entity recognition, you name it. If your task was "input some text, output a label or a span," BERT was the answer.

Encoder-decoder: T5, BART, Whisper

The encoder-decoder architecture is the original transformer from Vaswani et al. (2017). It has two separate stacks.

The encoder processes the input with bidirectional attention, exactly like BERT. Every input token can see every other input token. This gives you a rich, fully contextualized representation of the input.

The decoder generates output tokens one at a time, using causal attention so each output token can only see previous output tokens. This is the autoregressive part. But the decoder has an extra module that the encoder doesn't: cross-attention.

Cross-attention: the bridge

Cross-attention is how the decoder "looks at" the encoder's output. The mechanism is almost identical to self-attention, with one critical difference in where Q, K, and V come from.

In self-attention, all three projections come from the same sequence:

In cross-attention, Q comes from the decoder, but K and V come from the encoder:

So when a decoder token asks "what am I looking for?" (its query), it gets to match against what every encoder token is advertising (their keys), and then pull information from those encoder tokens (their values). The decoder can query any position in the input at any time.

This is a natural fit for sequence-to-sequence tasks. In translation, the encoder digests the French sentence bidirectionally, and the decoder generates English tokens one at a time, cross-attending back to the French whenever it needs to.

Raffel et al. showed with T5 that you could frame almost any NLP task as text-to-text with this architecture. Want sentiment analysis? Input: "sentiment: I loved this movie." Output: "positive." Want translation? Input: "translate English to French: Hello." Output: "Bonjour." Want summarization? Same pattern. The text-to-text framing was elegant and surprisingly general.

Radford et al. later used the encoder-decoder pattern for Whisper, where the encoder processes audio spectrograms and the decoder generates text transcriptions.

But it comes with complexity. You have two separate stacks of parameters, two types of attention in the decoder (self and cross), and two separate KV caches during inference. That's a lot more moving parts than a single stack.

Each decoder layer in an encoder-decoder model has three sub-layers instead of two: causal self-attention, cross-attention, and the feed-forward network. Compare that to a decoder-only layer, which just has self-attention and the feed-forward network. That extra cross-attention sub-layer adds parameters, adds memory, and adds engineering complexity. It also means each decoder layer needs access to the encoder's final hidden states, which have to be stored and passed around during the entire generation process.

If you've followed the seq2seq posts from Arc 2, this will feel familiar. The encoder-decoder transformer is really the same architecture as the RNN-based seq2seq model from Bahdanau et al., just with transformers replacing the recurrent layers and self-attention replacing the sequential hidden states. The cross-attention mechanism is the direct descendant of Bahdanau attention, with the recurrence stripped out.

Decoder-only: GPT and the modern LLM

The decoder-only architecture throws out the encoder entirely. You have a single stack of transformer layers with causal masking. The "input" (your prompt) is just the prefix of the sequence. There is no separate encoding step. There is no cross-attention.

When you type a question into ChatGPT, the model doesn't "encode" your question and then "decode" a response as two separate phases. It treats your question and its response as one continuous sequence. Your question tokens are the prefix, and the model generates response tokens by extending that prefix one token at a time, with each new token attending only to everything before it.

The training objective is pure next-token prediction, the same one we covered way back in Post 2.1. Given a sequence of tokens, predict the next one. That's the entire training signal. No masking, no text-to-text formatting, no separate encoder and decoder losses. Just predict the next token, billions of times, over trillions of tokens.

This might sound like a limitation. Doesn't the decoder lose something by not having a separate bidirectional encoder for the input?

A little, in principle. But during the prefill phase (when the model first processes your prompt), all the prompt tokens get processed in parallel. Each prompt token can attend to every other prompt token that precedes it. And because they're processed together in one forward pass, the KV cache for earlier prompt tokens effectively gets "seen" by later ones. The prompt portion is doing something very close to bidirectional encoding already, just through the lens of the causal mask.

In an encoder-decoder model, the encoder has a separate stack of parameters dedicated to understanding the input. In a decoder-only model, the same parameters that process the prompt also generate the output. There's no specialized "understanding" module; the same causal stack does both roles. With enough parameters and training data, that works just fine, and maybe even better, because the understanding and generation share representations instead of being siloed.

The three families reduce to three attention-mask shapes. Everything else (parameter allocation, KV-cache layout, training objective) cascades from the mask.

Radford et al. bet on this idea all the way back in 2018 with the original GPT paper. At the time, it looked like the weaker bet. BERT was crushing benchmarks, and GPT-1 with 117M parameters was fine but not spectacular on the standard NLP benchmarks of the day. GPT's bet was on scale, and scale eventually vindicated it: GPT-2 was clearly better, and GPT-3 was a paradigm shift. By the time you get to hundreds of billions of parameters, the "understanding" capabilities of a decoder-only model are hard to distinguish from a dedicated encoder, and you get generation for free.

Why decoder-only won

By 2024, essentially every frontier LLM was decoder-only. GPT-4, Claude, Gemini, LLaMA, Mistral. The encoder-decoder architecture didn't disappear entirely, but it got pushed into specific niches. A few factors drove this:

Simplicity. One set of weights, one type of attention, one KV cache. When you're trying to scale a model to hundreds of billions of parameters across thousands of GPUs, simplicity matters enormously. Every additional architectural component is another thing to debug, another thing to shard across devices, another thing that can go wrong during distributed training. The encoder-decoder architecture has roughly twice the surface area for bugs: two stacks, two attention types, two caching strategies. The decoder-only model has one of each.

Scaling. Decoder-only models empirically scaled more gracefully. When researchers doubled the parameters or the data, performance improved predictably along smooth power-law curves. The scaling laws that Kaplan et al. and Hoffmann et al. mapped out were studied primarily on decoder-only architectures, and the results were clean enough to guide trillion-token training runs worth hundreds of millions of dollars. When that much money is riding on your architecture choice, you want predictability. Decoder-only delivered it.

In-context learning. This was the big one. GPT-3 showed that a sufficiently large decoder-only model could perform tasks it was never explicitly trained for, just by putting examples in the prompt. You didn't need to fine-tune a model for sentiment analysis. You just wrote "Positive or negative?" after a movie review and the model figured it out. Want translation? Put a few English-French pairs in the prompt and the model gets the pattern. This replaced the entire fine-tuning pipeline that encoder-decoder models relied on, and it only worked because the decoder-only architecture processes the prompt and the completion as one unified sequence. The task specification is the prefix.

KV cache efficiency. During autoregressive generation, you cache the K and V tensors so you don't recompute them for every new token (we'll cover this in detail in the inference posts). With a decoder-only model, there's one stack to cache. With an encoder-decoder model, you need to cache the encoder's output and the decoder's growing KV cache. That's two separate memory allocations to manage, two different cache lifetimes, and more complexity in the serving infrastructure. The decoder-only setup is simpler and more memory-efficient, especially for long generations where the KV cache can grow to gigabytes.

Wang et al. (2022) ran a systematic comparison and found that decoder-only models performed best for zero-shot generalization, while encoder-decoder models had an edge after multitask fine-tuning. But the industry was already moving toward prompting over fine-tuning, so zero-shot performance is what mattered.

The "prompt as encoder" realization. There's one more subtle factor. People sometimes describe decoder-only models as having "no encoder," but that's not quite right. The prompt tokens, once processed through all the layers during prefill, are functionally encoded. Their KV cache entries persist and get attended to by every generated token. The prompt processing is the encoding. It just happens inside the same set of weights, using the same attention mechanism, as the generation. You could almost say the decoder-only model has a built-in encoder. It's just not a separate module.

Where encoder-decoder survives

Encoder-decoder didn't completely vanish. It survives where the input and output have fundamentally different structures.

Speech recognition. Whisper uses an encoder-decoder architecture because audio spectrograms and text are completely different modalities. The encoder processes continuous audio features, the decoder emits discrete text tokens. Cross-attention is the natural bridge between these two very different representation spaces.

Some translation systems. When you know the task is always "input in language A, output in language B," having a dedicated encoder for the source language still makes sense, especially at moderate scale where in-context learning isn't reliable.

Structured I/O tasks. When the input and output have very different lengths or structures (think: a long document goes in, a short SQL query comes out), the encoder-decoder separation maps cleanly onto the problem. The encoder can process the entire document with bidirectional attention, building up a rich representation, and the decoder can focus on generating the short structured output while cross-attending to the most relevant parts.

Vision-language models. Some multimodal architectures use an image encoder paired with a text decoder. The image encoder processes patches of the input image, and the text decoder cross-attends to those visual features while generating captions or answering questions. This is essentially the same encoder-decoder pattern, just applied across modalities.

But for the general-purpose "throw any task at it" use case, decoder-only won.

Worked example

To make the architectural difference concrete, let's trace how the same task (summarize a paragraph) flows through each architecture.

Say the input is: "The transformer architecture was introduced in 2017 by Vaswani et al. It replaced recurrence with self-attention and achieved state-of-the-art results on machine translation."

Encoder-decoder (T5): The input is formatted as "summarize: The transformer architecture was..." and fed to the encoder. The encoder processes this with bidirectional attention across all its layers. Each word has full context of the entire paragraph. The encoder produces a sequence of hidden states, one per input token. Then the decoder starts generating summary tokens, one at a time. Each decoder layer does three things: (1) causal self-attention over previously generated summary tokens, (2) cross-attention where the decoder queries the encoder's output to find the most relevant parts of the input paragraph, (3) a feed-forward block. The decoder keeps generating until it emits an end-of-sequence token.

Decoder-only (GPT): The entire prompt is one sequence: the paragraph, followed by something like "TL;DR:". No special formatting, no separate encoder step. During prefill, all tokens are processed in parallel through the causal stack. The model builds up a KV cache for the whole prompt. Then it generates summary tokens autoregressively. Each new token attends to the entire prefix (the original paragraph plus any summary tokens generated so far) through a single causal self-attention mechanism. No cross-attention needed. The input and output live in the same sequence, separated only by the "TL;DR:" marker that the model has learned to recognize as a generation boundary.

The decoder-only approach is less elegant in some ways. The model has to "learn" to treat the prompt differently from its own generation, even though they flow through the same attention mechanism. It works anyway, and it's simpler.

Notice something interesting about the attention patterns. In the encoder-decoder case, there are three distinct attention types happening: bidirectional self-attention in the encoder, causal self-attention in the decoder, and cross-attention bridging them. In the decoder-only case, there's just one: causal self-attention over the concatenated input-and-output sequence. That simplicity ends up being load-bearing.

It means one attention kernel, one KV cache format, one set of engineering decisions to optimize. When you're trying to serve millions of requests per day, that kind of uniformity matters. The inference engine doesn't need to handle different attention patterns for different parts of the model; it's the same operation everywhere.

The parameter budget question

One thing I find helpful is thinking about where the parameters actually go in each architecture.

In an encoder-only model like BERT-base, you have about 110M parameters, all in one stack. Every parameter is dedicated to understanding the input.

In an encoder-decoder model like T5-base, you have about 220M parameters split roughly evenly between the encoder and decoder. The encoder parameters focus on understanding, the decoder parameters focus on generation, and the cross-attention parameters serve as the bridge. But if you had 220M parameters to spend and chose a decoder-only architecture instead, all 220M parameters would be in a single, deeper stack. The understanding and generation happen in the same parameters. Whether that's better or worse depends on the task, but it's a fundamentally different allocation strategy.

At small scale, this distinction matters. T5-base and GPT-2 (which had about 117M parameters) performed quite differently on different tasks. But at large scale, with hundreds of billions of parameters, the decoder-only models had so much capacity that the parameter allocation question became less important than the simplicity and scaling advantages.

Implementation: cross-attention in PyTorch

The cross-attention module is almost identical to self-attention. The only difference is the source of K and V.

import torch
import torch.nn as nn
import math
 
class SelfAttention(nn.Module):
    """Standard self-attention: Q, K, V all come from the same input."""
    def __init__(self, d_model: int, d_k: int):
        super().__init__()
        self.W_Q = nn.Linear(d_model, d_k, bias=False)
        self.W_K = nn.Linear(d_model, d_k, bias=False)
        self.W_V = nn.Linear(d_model, d_k, bias=False)
        self.scale = math.sqrt(d_k)
 
    def forward(self, x: torch.Tensor, mask=None):
        # x: (batch, seq_len, d_model)
        Q = self.W_Q(x)   # (batch, seq_len, d_k)
        K = self.W_K(x)   # (batch, seq_len, d_k)
        V = self.W_V(x)   # (batch, seq_len, d_k)
 
        scores = Q @ K.transpose(-2, -1) / self.scale
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        weights = torch.softmax(scores, dim=-1)
        return weights @ V
 
class CrossAttention(nn.Module):
    """Cross-attention: Q from decoder, K and V from encoder."""
    def __init__(self, d_model: int, d_k: int):
        super().__init__()
        self.W_Q = nn.Linear(d_model, d_k, bias=False)
        self.W_K = nn.Linear(d_model, d_k, bias=False)
        self.W_V = nn.Linear(d_model, d_k, bias=False)
        self.scale = math.sqrt(d_k)
 
    def forward(self, x_dec: torch.Tensor, x_enc: torch.Tensor):
        # x_dec: (batch, dec_len, d_model)  — decoder hidden states
        # x_enc: (batch, enc_len, d_model)  — encoder output
 
        Q = self.W_Q(x_dec)  # (batch, dec_len, d_k)
        K = self.W_K(x_enc)  # (batch, enc_len, d_k)
        V = self.W_V(x_enc)  # (batch, enc_len, d_k)
 
        # Scores: each decoder position queries all encoder positions
        # Shape: (batch, dec_len, enc_len)
        scores = Q @ K.transpose(-2, -1) / self.scale
        weights = torch.softmax(scores, dim=-1)
 
        # Output: (batch, dec_len, d_k)
        return weights @ V

Look at the difference. In SelfAttention, the Q, K, and V projections all take x as input. In CrossAttention, Q takes x_dec (decoder states) while K and V take x_enc (encoder output). That's the entire structural change. The attention score matrix goes from being square (seq_len by seq_len) to rectangular (dec_len by enc_len). Every decoder position can query every encoder position, with no masking needed.

One detail here: the encoder's K and V projections are computed once from the encoder's output and then reused for every decoder layer and every generated token. This is why encoder-decoder models need to cache the encoder's output separately. During generation, the encoder isn't re-run. Its output is frozen, and the decoder's cross-attention layers just keep querying it.

Misconceptions

"Decoder-only models can't do bidirectional understanding." This comes from the BERT era, when "understanding" got associated with bidirectional attention and "generation" with causal attention. Decoder-only models understand plenty well. In-context learning, chain-of-thought reasoning, and instruction following all require deep understanding of the input. The model reads your entire prompt before generating a single token, and the quality of its generation is bounded by the quality of its understanding. There's also a recurring research line on prefix-LM training that explicitly drops the causal mask over the prompt portion, giving you literal bidirectional attention there while keeping causal attention over the output, and it doesn't change the high-level picture much. The intuition that "causal can't see context" was wrong.

"Encoder-decoder is dead." Not even close. T5 and Flan-T5 are still excellent for structured input-to-output tasks like summarization, translation, table-to-text, and code-to-doc generation, and they often beat similarly sized decoder-only models on those benchmarks. Whisper is encoder-decoder and is the production standard for speech recognition. Vision-language models like Flamingo and several modern VLMs use an image encoder bridged by cross-attention to a text decoder. What's true is that frontier general-purpose chat models are decoder-only. What's not true is that nobody ships encoder-decoder anymore.

"You need cross-attention for the decoder to access the input." In an encoder-decoder model, yes. Decoder-only models skip this entirely. The prompt tokens are already part of the same sequence as the generated tokens. The decoder attends to the prompt through the same causal self-attention it uses for everything else. There's no separate mechanism needed to "access the input" because the input is right there in the KV cache.

What's next

You now have a working mental model of all three transformer families and why the field converged on decoder-only for general-purpose LLMs. In the next post, A Close Reading of 'Attention Is All You Need', I go back to the original 2017 paper and walk through it section by section with the full vocabulary we've built up across Arc 4. Reading the source paper hits differently when you already know what every component does.

If you want to review pieces of Arc 4: causal masking for why decoder-only models don't cheat, seq2seq and Bahdanau attention for the RNN-era ancestor of cross-attention, and next-token prediction for the training objective that made decoder-only models inevitable.

Additional reading (and watching)

• Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The paper that made bidirectional pre-training the default in NLP and seeded the entire encoder-only family that still powers retrieval.
• Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR. Introduces T5 and the text-to-text framing that treats every NLP task as a seq2seq problem.
• Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. The original GPT paper. At the time it looked like the weaker architectural bet; scale eventually vindicated it.
• Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. GPT-3. The paper that established in-context learning as a first-class capability and made decoder-only the default.
• Wang, T., et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?. ICML 2022. Head-to-head comparison of decoder-only vs encoder-decoder under matched compute.
• Radford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. Whisper. The canonical modern encoder-decoder, bridging two modalities through cross-attention.
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The original transformer paper introduced the encoder-decoder architecture that started the whole family tree.
• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. Established smooth power-law scaling relationships for decoder-only transformers, giving labs confidence to invest in ever-larger models.