Decoder-Only vs. Encoder-Decoder: Architecture Trade-offs

Three ways to wire a transformer. The attention masks and cross-attention links distinguish the layouts.

The problem

The transformer building blocks include residual streams, self-attention, causal masks, positional encodings, FFNs, and normalization. Three common arrangements use them differently: encoder-only, encoder-decoder, and decoder-only. Their attention patterns determine which context each position can use and what can be cached during generation.

We will compare those arrangements on the same translation task, then look at the training and serving tradeoffs behind decoder-only language models.

Prerequisites

You should be comfortable with self-attention and Q/K/V (Post 4.1), causal masking (Post 4.3), and the layer norm / residual stream assembly from Post 4.6. If you remember Bahdanau attention from the seq2seq post, the cross-attention section will feel familiar, but it is not required.

Three families

Three useful transformer families organize this comparison. They all use the same building blocks we've covered: residual streams, self-attention heads, feed-forward networks, layer norms, positional encodings. The difference is how they wire these blocks together and, critically, what kind of attention mask they apply.

A bidirectional mask allows attention in both directions; a causal mask restricts each position to itself and earlier positions. Together with the training objective and arrangement of stacks, this determines how the model can use its input.

Encoder-only: BERT and friends

An encoder-only model runs a sequence of tokens through a stack of transformer layers and returns a sequence of hidden states.

The key property: bidirectional attention. Every token can attend to every other token in the sequence, without a causal mask (padding and task-specific masks may still apply). When the word "bank" shows up at position 5, it gets to look at position 12 where "river" sits. It also gets to look at position 2 where "investment" sits. It sees everything, in both directions, simultaneously.

BERT's training objective is called masked language modeling (MLM). You select 15% of input tokens as prediction targets, replacing 80% of those with [MASK], 10% with a random token, and leaving 10% unchanged, and the model has to predict what the masked tokens were. Because the model sees context in both directions (left and right of the mask), it builds strong representations of what words mean in context. There's also a next-sentence prediction task, though later work (RoBERTa) showed you could drop that without losing much.

Bidirectional attention would leak future targets in ordinary parallel next-token training. BERT avoids that objective by predicting selected corrupted positions.

BERT made pretraining followed by task-specific fine-tuning a widely used workflow: add a classification or span-prediction head, train on the task dataset, and deploy. Its bidirectional representations worked well for tasks such as sentiment classification, named entity recognition, and extractive question answering.

Encoder-decoder: T5, BART, Whisper

The encoder-decoder architecture is the original transformer from Vaswani et al. (2017). It has two separate stacks.

The encoder processes the input with bidirectional attention, exactly like BERT. Every input token can see every other input token. This gives you a rich, fully contextualized representation of the input.

The decoder generates output tokens one at a time, using causal attention so each output token can only see previous output tokens. This is the autoregressive part. But the decoder has an extra module that the encoder doesn't: cross-attention.

Cross-attention: the bridge

Cross-attention is how the decoder "looks at" the encoder's output. The mechanism is almost identical to self-attention, with one critical difference in where Q, K, and V come from.

In self-attention, all three projections come from the same sequence:

In cross-attention, Q comes from the decoder, but K and V come from the encoder:

So when a decoder token asks "what am I looking for?" (its query), it gets to match against what every encoder token is advertising (their keys), and then pull information from those encoder tokens (their values). The decoder can query any position in the input at any time.

This is a natural fit for sequence-to-sequence tasks. In translation, the encoder digests the French sentence bidirectionally, and the decoder generates English tokens one at a time, cross-attending back to the French whenever it needs to.

Raffel et al. showed with T5 that you could frame almost any NLP task as text-to-text with this architecture. Want sentiment analysis? Input: "sentiment: I loved this movie." Output: "positive." Want translation? Input: "translate English to French: Hello." Output: "Bonjour." Want summarization? Same pattern. The same input/output format served all of these tasks.

Radford et al. later used the encoder-decoder pattern for Whisper, where the encoder processes audio spectrograms and the decoder generates text transcriptions.

You have two separate stacks of parameters, two types of attention in the decoder (self and cross), and two separate KV caches during inference.

Each decoder layer in an encoder-decoder model has three sub-layers instead of two: causal self-attention, cross-attention, and the feed-forward network. Compare that to a decoder-only layer, which just has self-attention and the feed-forward network. That extra cross-attention sub-layer adds parameters, adds memory, and adds engineering complexity. It also means each decoder layer needs access to the encoder's final hidden states, which have to be stored and passed around during the entire generation process.

The encoder-decoder transformer retains the source/target division of Bahdanau’s RNN-based seq2seq model. Transformer blocks replace the recurrent layers, and cross-attention supplies the decoder with source context.

Decoder-only: GPT and the modern LLM

The decoder-only architecture throws out the encoder entirely. You have a single stack of transformer layers with causal masking. The "input" (your prompt) is just the prefix of the sequence. There is no separate encoding step. There is no cross-attention.

In a decoder-only language model, prompt processing and response generation use the same transformer stack. It treats your question and its response as one continuous sequence. Your question tokens are the prefix, and the model generates response tokens by extending that prefix one token at a time, with each processed token attending to itself and the preceding prefix.

A common decoder-only pretraining objective is next-token prediction, as in Post 2.1. A causal mask prevents target leakage. Later supervised or preference training can use different losses and token masks.

Earlier prompt positions never receive information from later ones. Prefill processes prompt positions in parallel, but still applies the causal mask at every layer. The final prompt position can gather information from the whole prompt; parallel execution does not make the earlier representations bidirectional.

An encoder-decoder model allocates separate parameters to source processing and target generation. A decoder-only model shares one stack across prompt and continuation.

The masks show which positions can communicate. Stack separation, cross-attention, parameter allocation, and the training objective are additional design choices.

GPT (2018) and BERT (2018) explored different pretraining objectives and attention patterns. Later GPT models demonstrated broad prompt-completion capabilities at larger scales. Those results helped establish the decoder-only approach, but they do not isolate architecture from differences in data, objectives, training compute, and evaluation.

Why decoder-only became common

Decoder-only designs became common for general-purpose language generation, including the published Llama and Mistral architectures. The encoder-decoder architecture didn't disappear entirely, but it got pushed into specific niches. A few factors drove this:

Simplicity. A causal decoder uses one stack and self-attention throughout. Encoder-decoder models add a separate source stack and decoder cross-attention. That creates more choices for distributing layers, storing source states, and serving requests, though it can also suit the task better.

Scaling. Large causal language-model experiments established useful empirical loss relationships as data, parameters, and compute increased. These measurements apply to their tested training setups; they are not evidence that another architecture cannot scale.

In-context learning. GPT-3 demonstrated task conditioning through examples in a prompt, without updating the model weights for each task. A continuous prompt-completion sequence makes that interface natural. In-context learning also exists in other architectures and complements fine-tuning rather than replacing it in every application.

KV caching. Both architectures reuse causal decoder K/V activations. An encoder-decoder also commonly caches source K/V projections for every decoder cross-attention layer. The latter stay fixed during generation; neither design removes the need to retain usable representations of its conditioning input.

Wang et al. (2022) found causal decoder models strongest for zero-shot generalization directly after unsupervised pretraining in their comparison. Non-causal encoder-decoder models performed best after multitask fine-tuning. The choice therefore depends partly on the training objective and adaptation procedure, not only on the wiring diagram.

Where encoder-decoder fits

Encoder-decoder remains useful when separate input and output processing suits the task.

Speech recognition. Whisper uses an encoder-decoder architecture because audio spectrograms and text are completely different modalities. The encoder processes continuous audio features, the decoder emits discrete text tokens. Cross-attention is the natural bridge between these two very different representation spaces.

Some translation systems. When you know the task is always "input in language A, output in language B," having a dedicated encoder for the source language still makes sense, especially at moderate scale where in-context learning isn't reliable.

Structured I/O tasks. When the input and output have very different lengths or structures (think: a long document goes in, a short SQL query comes out), the encoder-decoder separation maps cleanly onto the problem. The encoder can process the entire document with bidirectional attention, building up a rich representation, and the decoder can focus on generating the short structured output while cross-attending to the most relevant parts.

Vision-language models. Some multimodal architectures use an image encoder paired with a text decoder. The image encoder processes patches of the input image, and the text decoder cross-attends to those visual features while generating captions or answering questions. This is essentially the same encoder-decoder pattern, just applied across modalities.

Worked example

To make the architectural difference concrete, let's trace how the same task (summarize a paragraph) flows through each architecture.

Say the input is: "The transformer architecture was introduced in 2017 by Vaswani et al. It replaced recurrence with self-attention and achieved state-of-the-art results on machine translation."

Encoder-decoder (T5): The input is formatted as "summarize: The transformer architecture was..." and fed to the encoder. The encoder processes this with bidirectional attention across all its layers. Each word has full context of the entire paragraph. The encoder produces a sequence of hidden states, one per input token. Then the decoder starts generating summary tokens, one at a time. Each decoder layer does three things: (1) causal self-attention over previously generated summary tokens, (2) cross-attention where the decoder queries the encoder's output to find the most relevant parts of the input paragraph, (3) a feed-forward block. The decoder keeps generating until it emits an end-of-sequence token.

Decoder-only (GPT): The entire prompt is one sequence: the paragraph, followed by something like "TL;DR:". That marker is part of the prompt format; there is no separate encoder stack. During prefill, all tokens are processed in parallel through the causal stack. The model builds up a KV cache for the whole prompt. Then it generates summary tokens autoregressively. Each new token attends to the entire prefix (the original paragraph plus any summary tokens generated so far) through a single causal self-attention mechanism. No cross-attention needed. The input and output live in the same sequence, separated only by the "TL;DR:" marker that the model has learned to recognize as a generation boundary.

In the decoder-only approach, the prompt and generated text pass through the same attention mechanism. Their different roles must be represented within that shared sequence.

The parameter budget question

BERT-base has roughly 110M parameters, including embeddings and its encoder stack.

T5-base has roughly 220M parameters across embeddings, encoder, and decoder. Equal stack depth does not imply equal parameter shares: decoder cross-attention adds projections. The toy budget above counts only layer projections under stated width assumptions.

Comparing T5 and GPT parameter counts alone cannot isolate architecture quality: objectives, data, and training budgets also differ. The useful comparison here is how a stated budget is divided, not which model must perform better.

Implementation: cross-attention in PyTorch

The cross-attention module is almost identical to self-attention. The only difference is the source of K and V.

import torch
import torch.nn as nn
import math
 
class SelfAttention(nn.Module):
    """Standard self-attention: Q, K, V all come from the same input."""
    def __init__(self, d_model: int, d_k: int):
        super().__init__()
        self.W_Q = nn.Linear(d_model, d_k, bias=False)
        self.W_K = nn.Linear(d_model, d_k, bias=False)
        self.W_V = nn.Linear(d_model, d_k, bias=False)
        self.scale = math.sqrt(d_k)
 
    def forward(self, x: torch.Tensor, mask=None):
        # x: (batch, seq_len, d_model)
        Q = self.W_Q(x)   # (batch, seq_len, d_k)
        K = self.W_K(x)   # (batch, seq_len, d_k)
        V = self.W_V(x)   # (batch, seq_len, d_k)
 
        scores = Q @ K.transpose(-2, -1) / self.scale
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        weights = torch.softmax(scores, dim=-1)
        return weights @ V
 
class CrossAttention(nn.Module):
    """Cross-attention: Q from decoder, K and V from encoder."""
    def __init__(self, d_model: int, d_k: int):
        super().__init__()
        self.W_Q = nn.Linear(d_model, d_k, bias=False)
        self.W_K = nn.Linear(d_model, d_k, bias=False)
        self.W_V = nn.Linear(d_model, d_k, bias=False)
        self.scale = math.sqrt(d_k)
 
    def forward(self, x_dec: torch.Tensor, x_enc: torch.Tensor):
        # x_dec: (batch, dec_len, d_model)  — decoder hidden states
        # x_enc: (batch, enc_len, d_model)  — encoder output
 
        Q = self.W_Q(x_dec)  # (batch, dec_len, d_k)
        K = self.W_K(x_enc)  # (batch, enc_len, d_k)
        V = self.W_V(x_enc)  # (batch, enc_len, d_k)
 
        # Scores: each decoder position queries all encoder positions
        # Shape: (batch, dec_len, enc_len)
        scores = Q @ K.transpose(-2, -1) / self.scale
        weights = torch.softmax(scores, dim=-1)
 
        # Output: (batch, dec_len, d_k)
        return weights @ V

In SelfAttention, the Q, K, and V projections all take x as input. In CrossAttention, Q takes x_dec (decoder states) while K and V take x_enc (encoder output). The attention score matrix goes from being square (seq_len by seq_len) to rectangular (dec_len by enc_len). Every decoder position can query every encoder position, with no masking needed.

Each decoder layer projects the encoder output with its own cross-attention K/V weights. Those activations can be cached for every generated token. Once all required K/V projections are retained, the original encoder states need not also remain resident solely for cross-attention.

Misconceptions

"Decoder-only models can't do bidirectional understanding." A causal decoder’s final prompt position can use the entire prompt, so it can perform tasks that require context from across that prompt. Earlier positions still cannot attend to later ones. Prefix-LM training uses a different mask to give prompt positions bidirectional access while keeping the continuation causal.

"Encoder-decoder is obsolete." T5, BART, and Whisper provide published examples of the architecture on text and speech tasks. Task suitability depends on the model and training recipe, not merely whether it has one stack or two.

"You need cross-attention for the decoder to access the input." In an encoder-decoder model, yes. Decoder-only models skip this entirely. The prompt tokens are already part of the same sequence as the generated tokens. The decoder attends to the prompt through the same causal self-attention it uses for everything else. There's no separate mechanism needed to "access the input" because the input is right there in the KV cache.

What's next

In the next post, A Close Reading of 'Attention Is All You Need', I go back to the original 2017 paper and walk through it section by section with the full vocabulary we've built up across Arc 4.

If you want to review pieces of Arc 4: causal masking for why decoder-only models don't cheat, seq2seq and Bahdanau attention for the RNN-era ancestor of cross-attention, and next-token prediction for the training objective used by causal language models.

Additional reading (and watching)

• Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. The paper that made bidirectional pre-training the default in NLP and seeded the entire encoder-only family that still powers retrieval.
• Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR. Introduces T5 and the text-to-text framing that treats every NLP task as a seq2seq problem.
• Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. The original GPT paper.
• Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. GPT-3. Reports few-shot task performance from examples placed in the prompt.
• Wang, T., et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?. ICML 2022. Head-to-head comparison of decoder-only vs encoder-decoder under matched compute.
• Radford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. Whisper. The canonical modern encoder-decoder, bridging two modalities through cross-attention.
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The original transformer paper introduced the encoder-decoder architecture that started the whole family tree.
• Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. Established smooth power-law scaling relationships for decoder-only transformers, giving labs confidence to invest in ever-larger models.