A Close Reading of ‘Attention Is All You Need’

Why read a paper from 2017?

“Attention Is All You Need” describes a machine-translation system. Reading its design choices alongside a later model separates the original transformer’s mechanisms from choices made for that task.

Models such as Llama and Mistral build on this paper, while changing much of its surrounding architecture. The original uses encoder-decoder stacks, ReLU FFNs, sinusoidal positions, and post-norm residual blocks.

Which choices have persisted, and which changed as transformers moved beyond translation?

Prerequisites

This one leans on the rest of Arc 4 harder than usual: self-attention from scratch (4.1, which also sets up the residual stream), multi-head attention (4.2), causal masking (4.3), positional encodings and RoPE (4.4), FFN as memory (4.5), layer norms (4.6), and decoder-only vs. encoder-decoder (4.7). Each one is linked again where the paper first introduces it, for reference.

Section 1: Introduction

The paper opens with a clear argument: recurrent models are fundamentally sequential, and that sequentiality prevents parallelization.

The world it's arguing against is the one from Post 2.7: RNN seq2seq with Bahdanau attention bolted on. Attention had already fixed the information bottleneck there. What it hadn't fixed was the sequential dependency, because each hidden state still waits on the previous one. The transformer's move is to delete the recurrence rather than improve it.

The introduction emphasizes parallelism and reports translation results after twelve hours of training on eight P100 GPUs. Its immediate argument is better translation at a lower training cost.

Section 3: Model Architecture

Figure 1 shows the stacked encoder and decoder blocks labeled “Multi-Head Attention,” “Add & Norm,” and “Feed Forward.”

Every box on it has a post behind it by now. "Multi-Head Attention" is 4.1 and 4.2. "Add & Norm" is the residual stream from 4.1 plus the normalization from 4.6. "Feed Forward" is 4.5. "Positional Encoding," sitting at the bottom, is 4.4.

The “Norm” in “Add & Norm” sits after the addition: Post-Norm. Many later decoder-only models use Pre-Norm, so reproducing their blocks requires moving the normalization operation.

The paper also ties weights between its two embedding layers and the pre-softmax linear transformation. Weight tying appears in later models too, though it is not universal.

The paper uses two stacks: an encoder reads the source sentence, and a decoder generates the target through causal self-attention and cross-attention to the encoder output. A decoder-only model such as GPT-2 or Llama omits the source encoder and its cross-attention. GPT (Radford et al., 2018) applied that arrangement to language-model pretraining.

Section 3.2.1: Scaled Dot-Product Attention

Here's the core equation:

We built this operation in Post 4.1, so what I want to look at here is the presentation: what the paper argues for, and what it just hands you.

The paper motivates the square-root scaling with a variance argument: independent zero-mean, unit-variance query and key coordinates give a dot product whose variance grows with the head dimension. Large score differences can saturate softmax. This is an initialization argument, not a guarantee about every trained attention head.

The paper defines separate Q, K, and V projections without an ablation of that decomposition. Its experiments therefore do not isolate the contribution of having three projections.

The paper uses softmax to turn scores into weights. Dense all-pairs attention gives the quadratic arithmetic cost; it does not require materializing the full score matrix. FlashAttention later showed how to compute the same result with tiles.

The paper compares dot-product and additive attention and favors the former for its efficient matrix-multiplication implementation. It also reports that unscaled dot products fare worse at larger head dimensions, motivating the scale factor.

Section 3.2.2: Multi-Head Attention

The base model uses eight heads with $d_k=d_v=64$ and $d_\text{model}=512$. Reducing each head width keeps projection parameters and the leading matmul work comparable to a single full-width head. Separate attention maps still have memory and kernel costs.

The motivation is to attend to different positions through different representation subspaces. Post 4.2 unpacks how separate softmax distributions make that possible.

Table 3 compares 1, 4, 8, 16, and 32 heads. One head scores 24.9 BLEU, eight scores 25.8, sixteen scores 25.8, and thirty-two scores 25.4 in that ablation. Eight heads is an experimental design choice, not a general optimum.

The paper uses equal numbers of query and KV heads. Later MQA and GQA experiments tested sharing K/V projections while retaining separate queries (Post 4.2 covers both).

Section 3.3: Position-wise Feed-Forward Networks

The paper's FFN formulation is clean:

The paper specifies two linear layers with ReLU, $d_{ff} = 2048$ and $d_\text{model} = 512$, applied independently at every position. It does not separately motivate each of these choices. Post 4.5 derives the block’s key-value interpretation.

At the base model’s dimensions, the FFN holds two thirds of an encoder layer’s main projection weights, or half in a decoder layer with cross-attention.

“Position-wise” specifies that this MLP runs on each token independently, with no mixing across the sequence. Geva et al. (2021) later read that same constraint as a key-value memory, where, in this row-vector notation, $W_1$'s columns are patterns to match and $W_2$'s rows are things to write into the residual stream.

Gated variants add another projection and change the activation, while retaining position-wise expand-and-project computation.

# Original (Vaswani et al., 2017)
import torch
import torch.nn.functional as F
# FFN(x) = max(0, xW1 + b1)W2 + b2
def ffn_original(x, W1, b1, W2, b2):
    hidden = torch.relu(x @ W1 + b1)   # ReLU activation
    return hidden @ W2 + b2             # project back down
 
# Modern (e.g., Llama, Mistral)
# SwiGLU(x) = (Swish(xW_gate) ⊙ xW_up) W_down
def ffn_swiglu(x, W_gate, W_up, W_down):
    gate = F.silu(x @ W_gate)           # Swish/SiLU activation
    up = x @ W_up                       # linear projection
    return (gate * up) @ W_down         # gated, then project back
 
# Key differences:
# - ReLU → SiLU (Swish): a smooth, signed gate
# - One linear layer → gated: gate and up projections, element-wise multiply
# - This Llama-style example omits biases
# - Expansion factor: 4x → ~2.67x (8/3) to compensate for the extra gate matrix

Section 3.5: Positional Encoding

The paper proposes sinusoidal positional encodings:

The paper specifies 10000 as the frequency base without deriving that value. The original RoPE formulation uses the same base.

Table 3 reports nearly identical translation results for learned and sinusoidal positions. The authors choose sinusoids partly because the formula extends to unseen sequence lengths. That gives a defined encoding, not a guarantee that a trained model will generalize to those lengths.

The paper notes that a fixed positional offset is a linear transformation of the sinusoidal encoding. That makes relative information available, but does not establish how the trained model uses it. RoPE instead puts a relative-position identity directly into Q/K dot products.

The paper adds position vectors once, at the input. Later schemes make different choices: GPT-2 retains learned absolute input embeddings, while RoPE rotates Q and K inside attention.

Section 4: Why Self-Attention

This is the section where the paper makes its theoretical case. Table 1 compares self-attention against recurrent layers and convolutional layers across three criteria: computational complexity per layer, sequential operations required, and maximum path length between any two positions.

Self-attention has $O(n^2 \cdot d)$ computation per layer, $O(1)$ sequential operations, and $O(1)$ maximum path length. A recurrent layer has $O(n \cdot d^2)$ computation per layer, $O(n)$ sequential operations, and $O(n)$ maximum path length.

The constant path length matters: distant positions can exchange information in one attention operation rather than through a chain of recurrent states. It does not imply equal gradient strength at every distance; that also depends on attention weights and the other operations.

The table also compares convolutional layers. Ordinary convolutions need depth proportional to distance at fixed kernel width; dilated convolutions can connect distant positions in logarithmic depth.

For the translation lengths considered, the paper notes that $n$ is typically smaller than $d$. Under that condition, attention’s $O(n^2)$ term compares favorably with the recurrent layer’s cost. It also discusses a restricted local-attention variant with $O(n \cdot r \cdot d)$ work, though the reported model uses full attention.

At much longer sequence lengths, dense attention’s $O(n^2d)$ arithmetic becomes a substantial cost. Tiled exact attention reduces memory traffic; sparse patterns and other attention rules change the work. Constant interaction depth does not mathematically require all-pairs quadratic attention.

Section 6: Results

Section 6 reports 28.4 BLEU on WMT 2014 English-to-German, exceeding the prior ensembles in its comparison. The NeurIPS proceedings version reports 41.0 BLEU for English-to-French as a new single-model result, not a win over every ensemble. The big model trained for 3.5 days on eight P100 GPUs. These results belong to the paper’s translation setup; they are not a general speed ratio against recurrent networks.

Table 3 is useful because it tests alternatives within one setup. For example, eight heads and sixteen heads both score 25.8 BLEU, while removing dropout scores 24.6.

What survived

Scaled dot-product attention. The paper’s operation remains a common baseline. FlashAttention changes how it is executed while preserving the result; other architectures may change score normalization, head sharing, or the attention rule itself.

Multi-head attention. The concept of splitting the representation into parallel attention heads, each operating on a subspace, has persisted. The number of heads changed, and the K/V sharing scheme changed (GQA/MQA), but the multi-head principle is still there.

Residual connections. The "Add" in "Add & Norm" is still exactly the same: $x + \text{sublayer}(x)$. The residual stream as the backbone of information flow has only become more central to how we understand transformers, especially through the circuits framework from Elhage et al.

The position-wise FFN block. Two linear projections with a nonlinearity, applied independently at each position. The nonlinearity changed. The expansion ratio changed. A gating mechanism was added. But the structural role of the FFN block within each transformer layer is the same.

The overall layer structure. Attention, then FFN, repeated $N$ times. The paper stacked six encoder layers and six decoder layers. Modern models stack 32, 80, or more. But the alternating attention-FFN structure, with residual connections around each sublayer, is unchanged.

The Q/K/V formulation. Separate learned projections give matching and information transfer distinct roles. Variants can share or compress projections, but this remains a useful starting point for understanding attention.

The training recipe structure. Adam optimizer, warmup schedule, residual dropout. The specific numbers have changed, but the general recipe (Adam-family optimizer, warmup, some form of regularization) has been persistent. AdamW replaced Adam, warmup lengths changed, dropout was dropped for large models, but the skeleton of the training setup from this paper is still recognizable in modern training runs.

What got replaced

The following changes distinguish the original architecture from later models such as Llama. They concern separate choices and need not occur together.

Post-norm and pre-norm. The paper applies $\text{LayerNorm}(x+\text{sublayer}(x))$. Pre-Norm instead applies $x+\text{sublayer}(\text{Norm}(x))$, leaving an identity term in the derivative of each residual block. Xiong et al. (2020) analyzed gradient scales at initialization and found that Pre-Norm could train without warmup in their tested settings.

RMSNorm changes a separate choice: it omits mean subtraction and usually learns only a gain. Models such as Llama combine Pre-Norm placement with RMSNorm. The normalization post separates these two decisions.

Sinusoidal encodings became RoPE. Llama-style models use RoPE inside attention; learned absolute embeddings remain part of other published architectures.

Encoder-decoder became decoder-only. For generative language models, at least. The encoder-decoder structure is still used in some settings (speech recognition, translation, and some multimodal models), but the dominant LLM architecture is decoder-only. The original paper chose separate source and target stacks for translation; decoder-only models can perform translation too. A single causal stack is a direct fit for language modeling on a continuous text sequence. We covered the full set of tradeoffs in Post 4.7.

ReLU gained gated alternatives. SwiGLU performed well in the matched-budget experiments of Shazeer (2020) and appears in models such as Llama and Mistral.

The hyperparameters scaled. $d_\text{model}$ went from 512 to 4096 or more. Heads went from 8 to 32-128. Layers went from 6 to 32-80+. The expansion factor in the FFN changed to accommodate SwiGLU. The specific numbers in the paper were tuned for WMT English-to-German translation, not for general language modeling.

GQA/MQA reduce the KV cache. Sharing key/value heads across query heads reduces stored activations during generation. GQA offers a tunable compromise between standard multi-head attention and one shared KV head. Cache costs depend on sequence length, batch size, layer count, head dimensions, and precision; encoder-decoder models also benefit from caching.

Later developments and omissions

The paper evaluates sequence transduction, chiefly machine translation. Several later language-model practices are outside that scope.

Unconditional language-model pretraining. The original decoder already predicts the next target token given the source and earlier target tokens. GPT later emphasized causal language-model pretraining on text without a separate source encoder. Next-token prediction itself long predates both papers.

Scaling laws. Kaplan et al. (2020) showed that transformer loss scales as a power law with model size, dataset size, and compute. Those scaling experiments are not part of the 2017 paper.

Few-shot learning and foundation models. GPT-2 showed that large language models could perform tasks they weren't explicitly trained for. GPT-3 turned this into a practical paradigm with in-context learning. The original paper did not evaluate this prompt-based task setup.

KV caching. When you generate tokens autoregressively, the K and V projections for all previous tokens don't change. So you can cache them and only compute K and V for the new token at each step. The paper does not describe this cache. We'll cover KV caching in detail when we get to Arc 5.

The annotated versions

Two resources from 2018 explain the paper in more detail. Alexander Rush's "The Annotated Transformer" reimplements it line by line in PyTorch, which is where to go for everything the paper glosses over: how masking actually gets applied, how the LR schedule is coded. Jay Alammar's "The Illustrated Transformer" explains the architecture visually.

Both describe the paper's encoder-decoder architecture. Read The Annotated Transformer and then go look at Llama and the encoder half is simply gone, cross-attention with it. What's left is the same self-attention and FFN blocks, with the norms moved and the activation swapped.

A note on training details

Section 5 describes the learning-rate schedule. The learning rate ramps up linearly during a warmup period, then decays with the inverse square root of the step:

The paper uses 4,000 warmup steps followed by inverse-square-root decay. The schedule is often called “Noam.” The plot compares its shape with a chosen warmup-plus-cosine schedule at equal peaks. Warmup requirements depend on initialization, normalization, optimizer, and learning rate.

They also use label smoothing at 0.1. The paper reports worse perplexity but better accuracy and BLEU, a useful example of optimizing a metric that matches the application rather than treating every score as interchangeable.

So what's the legacy?

The paper demonstrated an attention-based sequence model that processed known positions in parallel, replacing the recurrent dependency between them. Its translation experiments tested both output quality and training cost.

Elhage et al. (2021) later used the Q/K score computation and weighted value sum from Section 3.2 to analyze transformer circuits. The same operations remain recognizable even when surrounding normalization, positional schemes, and head-sharing arrangements change.

The title argues that an attention-based architecture can replace recurrent and convolutional sequence processing in this setting. The model still needs feed-forward blocks, normalization, embeddings, and a training objective.

Misconceptions

"The transformer paper invented attention." Bahdanau attention predates it, and the paper’s related-work section cites earlier self-attention. Its contribution was the Transformer architecture and evidence that attention-based sequence processing could replace recurrence and convolution in its translation setup. FFNs, embeddings, and normalization remain part of the computation.

"Modern LLMs use the architecture from this paper." A model such as Llama retains multi-head attention, residual additions, and position-wise FFNs, while changing the stack arrangement, normalization, positions, and FFN activation. Reproducing a named model requires those choices, not just the label “transformer.”

"The result was only an accuracy improvement." The paper reports both translation quality and training cost. Its parallel processing of source and target positions reduced the sequential training work.

What's next

Post 4.9 covers Mixture-of-Experts layers, which replace the dense FFN with sparse expert routing.

Arc 5 covers decoding strategies, KV caching, and speculative decoding.

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The paper this post is a close reading of. Pitched as a faster translation model; introduced the encoder-decoder Transformer and influenced later language models.
• Rush, A. (2018). The Annotated Transformer. Harvard NLP. Line-by-line PyTorch reimplementation of the paper — the best single resource for the implementation details the paper glosses over.
• Alammar, J. (2018). The Illustrated Transformer. Visualization-first walkthrough of Q/K/V and multi-head attention. Still the clearest visual introduction to the architecture.
• Raschka, S. (2023). About LayerNorm Variants in the Original Transformer Paper. Clean explanation of why post-norm was replaced by pre-norm, and how the original Post-Norm diagram differs from Pre-Norm models.
• Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. The original GPT paper — the architectural move from encoder-decoder to decoder-only that the Vaswani paper didn't take.
• Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. The attention mechanism the transformer was built to generalize — RNN seq2seq with soft alignment over encoder states.
• Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic. Rewrites the paper's attention operation as a mathematical framework for reasoning about what specific heads and layers compute.
• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. Same scaled dot-product attention from Section 3.2; faster execution by fusing softmax and matmul into one IO-aware kernel.
• Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. The SwiGLU paper; introduced the gated FFN variant that replaced ReLU+two-linear in modern LLMs.