A Close Reading of ‘Attention Is All You Need’

What the 2017 paper specified versus what today's transformer actually looks like.

Why read a paper from 2017?

It's easy to work with LLMs in 2026 without ever having read "Attention Is All You Need". The architecture diagram has been redrawn a hundred times in tutorials and blog posts, and most of us have absorbed the ideas secondhand. That's fine for getting work done. But there's a difference between knowing what a transformer is and understanding which parts of the original design were deliberate architectural choices versus which parts were just the thing that happened to work for machine translation in 2017.

Every model you interact with today descends from this paper. GPT-4, Claude, Gemini, Llama, Mistral. All of them. And yet the paper itself describes none of those things. It describes a machine translation system. It uses an encoder-decoder architecture that most modern LLMs have abandoned. Its specific hyperparameters, activation functions, and normalization scheme have all been replaced.

So the interesting question isn't "what does the paper say?" It's "what was essential and what was contingent?" Which ideas turned out to be load-bearing, and which were scaffolding that got removed once the building was up?

There's a second reason to read it. The paper is short. Fifteen pages including references, which is unusually compact for something this consequential, and it's well written. Consistent notation, informative ablations. It's a good paper, not just an important one.

Since we've spent this whole arc building the mechanisms from scratch, I'm not going to re-explain them here. This is a close reading. I want to look at what the paper argues for, what it asserts without argument, what it never thought to ask, and what reads differently with 2026 eyes.

Prerequisites

This one leans on the rest of Arc 4 harder than usual: self-attention from scratch (4.1, which also sets up the residual stream), multi-head attention (4.2), causal masking (4.3), positional encodings and RoPE (4.4), FFN as memory (4.5), layer norms (4.6), and decoder-only vs. encoder-decoder (4.7). Each one is linked again where the paper first introduces it, so you can jump back if a piece has gone fuzzy.

Section 1: Introduction

The paper opens with a clear argument: recurrent models are fundamentally sequential, and that sequentiality prevents parallelization.

The world it's arguing against is the one from Post 2.7: RNN seq2seq with Bahdanau attention bolted on. Attention had already fixed the information bottleneck there. What it hadn't fixed was the sequential dependency, because each hidden state still waits on the previous one. The transformer's move is to delete the recurrence rather than improve it.

What I notice reading the introduction now is how modest the pitch is. The authors sell the transformer as a speed improvement: "The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs." Twelve hours on eight GPUs. That's the argument. Better BLEU, much cheaper.

The claim was true and the framing was too small. Parallelism wasn't just a wall-clock win on 2017 hardware. It was the property that let the architecture be spread across thousands of GPUs, which is what scaling from 100M to 100B parameters requires, and which recurrence is structurally hostile to. The authors were arguing about training time. What they'd actually built was the only architecture that could ride the next decade of compute.

Section 3: Model Architecture

This is where the famous Figure 1 lives. If you've spent any time in the LLM world, you've seen it: the stacked encoder and decoder blocks with their boxes labeled "Multi-Head Attention," "Add & Norm," and "Feed Forward."

Every box on it has a post behind it by now. "Multi-Head Attention" is 4.1 and 4.2. "Add & Norm" is the residual stream from 4.1 plus the normalization from 4.6. "Feed Forward" is 4.5. "Positional Encoding," sitting at the bottom, is 4.4. So rather than walk the boxes, I want to look at what the figure quietly commits to.

The "Norm" in "Add & Norm" sits after the addition. Post-norm. Virtually every model since uses pre-norm, and if you're reading Figure 1 to understand how a modern model works, that ordering is the single most misleading thing on the page. I'll come back to it.

There's also a detail in the prose that's easy to miss: they tie the weight matrix between the two embedding layers and the pre-softmax linear transformation. Weight tying, mentioned once, no fanfare. It survived, mostly unchanged, into models a thousand times the size.

And then the shape of the thing. Two stacks: an encoder that reads the source sentence, a decoder that generates the target one token at a time through a cross-attention layer pointed at the encoder's output. This is not what GPT, Claude, or Llama do. Those are decoder-only (Post 4.7), no encoder, no cross-attention, just the right half of the diagram with the causal mask from 4.3. The paper is translating German to English. "Ich bin ein Berliner" goes in the left stack, "I am a Berliner" comes out of the right one. Radford et al. (2018) showed that deleting the left stack and training the rest on next-token prediction gets you GPT. That's one of the most consequential edits anyone ever made to this architecture, and it isn't in this paper.

Figure 1 shows the boxes but not the ordering, which is the thing that's actually hard to hold in your head. The encoder runs once, in parallel, over the whole source. The decoder then runs autoregressively against the frozen encoder output. Here's that sequence on the paper's own example.

Section 3.2: Scaled Dot-Product Attention

Here's the core equation:

We built this operation in Post 4.1, so what I want to look at here is the presentation: what the paper argues for, and what it just hands you.

It argues for almost nothing. The $\sqrt{d_k}$ scaling gets a single sentence in a footnote-shaped aside: "We suspect that for large values of $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients." That is the entire defense of a term that has survived unchanged for nearly a decade. And I like that they wrote "we suspect." It's a back-of-the-envelope variance argument, offered as a hunch rather than a theorem, and it was right.

Then there's what never gets asked. The paper never motivates why there are three projections. Q, K, and V show up fully formed, inherited from the Bahdanau lineage, and the decomposition is treated as given. Nobody defends it, and it turned out to be one of the most durable choices in the whole design.

Softmax gets the same treatment. No alternatives are discussed, because in 2017 there weren't any worth discussing. But softmax is what forces the full $n \times n$ score matrix into existence, so it's the origin of the quadratic cost that has occupied the field ever since. It enters as an assumption and leaves as a constraint.

The one choice the paper does defend is dot-product over additive attention. A footnote concedes that additive attention "theoretically has more expressive power," then picks dot-product anyway because it's one matmul and GPUs are good at matmuls. Speed over expressiveness. Nobody has used additive attention in a transformer since.

Section 3.3: Multi-Head Attention

Eight heads, $d_k = d_v = 64$, $d_\text{model} = 512$. The heads partition the model dimension instead of adding to it, and the paper justifies that in a parenthetical it never returns to: "Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality." Multi-head attention is free. That's the argument for the design, and the paper spends one parenthetical on it.

The argument for why you'd want it is one sentence: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this." We spent a whole post (4.2) unpacking what "averaging inhibits this" means. The paper leaves it compressed and moves on.

The actual evidence is in Table 3, and it's easy to skip. They tried 1, 4, 8, 16, and 32 heads. Single-head is notably worse. Sixteen is a hair better than eight, thirty-two is a hair worse. So the famous "8 heads" isn't a principle, it's the top of a very flat curve at $d_\text{model} = 512$. Modern models run 32 to 128 heads because $d_\text{model}$ grew alongside, keeping each head's subspace roomy enough to be worth having.

The interesting silence is on the K/V side. Nothing in the paper says query heads and key/value heads have to come in matched pairs, and nothing questions it either. It took until MQA and GQA for anyone to check, and the answer was that the K and V projections are much more redundant across heads than the Q projections are (Post 4.2 covers both).

Section 3.4: Position-wise Feed-Forward Networks

The paper's FFN formulation is clean:

The paper gives this a paragraph. Two linear layers with a ReLU between them, $d_{ff} = 2048$ against $d_\text{model} = 512$, run identically at every position, and no argument for any of it. The 4x expansion is stated, not defended. ReLU is stated, not defended. We spent Post 4.5 on this block; the paper spends a hundred words and moves on.

Which is strange in hindsight, because this is where most of the parameters live. At the base model's dimensions, the FFN holds two thirds of each layer's weights, and it gets a fraction of the space attention does. The title tells you where the authors thought the interesting part was. The parameter count tells a different story.

The word doing the real work is "position-wise," and the paper uses it purely as a shape constraint: this MLP runs on each token independently, with no mixing across the sequence. Geva et al. (2021) later read that same constraint as a key-value memory, where $W_1$'s rows are patterns to match and $W_2$'s columns are things to write into the residual stream. None of that is here. The authors chose the structure for reasons unrelated to the reason it turned out to matter.

Everything inside the block got replaced. The shape constraint survived intact.

# Original (Vaswani et al., 2017)
# FFN(x) = max(0, xW1 + b1)W2 + b2
def ffn_original(x, W1, b1, W2, b2):
    hidden = torch.relu(x @ W1 + b1)   # ReLU activation
    return hidden @ W2 + b2             # project back down
 
# Modern (e.g., Llama, Mistral)
# SwiGLU(x) = (Swish(xW_gate) ⊙ xW_up) W_down
def ffn_swiglu(x, W_gate, W_up, W_down):
    gate = F.silu(x @ W_gate)           # Swish/SiLU activation
    up = x @ W_up                       # linear projection
    return (gate * up) @ W_down         # gated, then project back
 
# Key differences:
# - ReLU → SiLU (Swish): smoother, no dead neurons
# - One linear layer → gated: gate and up projections, element-wise multiply
# - Biases removed in modern models (fewer params, no measurable quality loss)
# - Expansion factor: 4x → ~2.67x (8/3) to compensate for the extra gate matrix

Section 3.5: Positional Encoding

The paper proposes sinusoidal positional encodings:

That 10000 has no derivation anywhere in the paper. It's a magic number in 2017, it's still a magic number in 2026, and RoPE inherited it.

What strikes me about this section is how tentative it is. The authors tried learned absolute embeddings too, report "nearly identical results" (Table 3, row E), and then choose sinusoids anyway on a hypothesis: that it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training." May. They're guessing, and they say so plainly. The guess didn't hold. Sinusoidal encodings extrapolate badly, and a model trained at length 512 comes apart at 1024.

The reasoning underneath the guess was right, though, and that's the part of the section I like. They point out that for any fixed offset $k$, $PE_{pos+k}$ is a linear function of $PE_{pos}$, so relative position is available to the model. Available, not enforced. The model has to learn to use it, and it mostly doesn't. RoPE (Post 4.4) takes that same relative-position intuition and makes it structural rather than learnable, by rotating Q and K inside the attention operation instead of adding a vector at the bottom of the stack. Same goal, written down in this paragraph in 2017, actually reached four years later.

Notice where the encoding enters: added to the embeddings, once, at the very bottom of the model. Every scheme that replaced it moved position deeper into the computation.

Section 4: Why Self-Attention

This is the section where the paper makes its theoretical case. Table 1 compares self-attention against recurrent layers and convolutional layers across three criteria: computational complexity per layer, sequential operations required, and maximum path length between any two positions.

Self-attention has $O(n^2 \cdot d)$ computation per layer, $O(1)$ sequential operations, and $O(1)$ maximum path length. A recurrent layer has $O(n \cdot d^2)$ computation per layer, $O(n)$ sequential operations, and $O(n)$ maximum path length.

The middle column is the one the authors lead with, but the third column is the one that mattered. $O(1)$ path length means token 0 and token 500 interact in a single hop, so the gradient signal between them is as strong as between token 499 and token 500. Every dependency costs the same, regardless of distance. That's the property RNNs could never buy, gating or no gating.

The convolution row is the fossil in this table. In 2017 ConvS2S was a serious contender and the paper takes it seriously, arguing that convolutions need $O(\log_k n)$ layers to connect positions $n$ apart. Convolutions for NLP have since evaporated, so the argument won and stopped mattering at the same time.

The part that reads strangest in 2026 is how relaxed they are about $O(n^2)$. They point out that for typical translation sentences, $n$ is smaller than $d$, so the quadratic term isn't even the dominant cost. They mention a restricted local-attention variant that would bring it to $O(n \cdot r)$, and then don't use it, because why would they. At 512 tokens, it's a non-issue.

They were right about their setting and the setting moved. At 128K or 1M tokens, that quadratic term is the whole engineering problem, and it spawned FlashAttention, sparse patterns, linear approximations, and most of the long-context literature. The feature that makes the transformer work, every token reaching every other token, is the same feature that makes it expensive. The $O(1)$ path length and the $O(n^2)$ cost are the same fact written two ways.

Section 6: Results

The numbers are about translation, so most of them don't concern us. 28.4 BLEU on WMT 2014 English-to-German, 41.0 on English-to-French, both state of the art, both beating ensembles. The line that did the damage is the training cost: 3.5 days on 8 P100s, against weeks for the models it beat. Better and cheaper at once is rare, and it's usually what lets an architecture take over rather than merely get cited.

Read Table 3 with an eye on the spread rather than the winners. A BLEU point here, a BLEU point there. Almost nothing they varied (heads, key dimension, dropout, positional scheme) moves the result much, which is a strange thing to notice in your own ablation table and not remark on. The architecture was forgiving, and I suspect that's a real part of why it spread so fast. You didn't have to get it exactly right to get it working.

What survived

Let me try to separate the load-bearing parts from the contingent ones.

Scaled dot-product attention. The exact formula from Section 3.2, including the $\sqrt{d_k}$ scaling, is still used in every transformer I'm aware of. FlashAttention changed how it's computed at the hardware level, but the mathematical operation is unchanged.

Multi-head attention. The concept of splitting the representation into parallel attention heads, each operating on a subspace, has persisted. The number of heads changed, and the K/V sharing scheme changed (GQA/MQA), but the multi-head principle is still there.

Residual connections. The "Add" in "Add & Norm" is still exactly the same: $x + \text{sublayer}(x)$. The residual stream as the backbone of information flow has only become more central to how we understand transformers, especially through the circuits framework from Elhage et al.

The position-wise FFN block. Two linear projections with a nonlinearity, applied independently at each position. The nonlinearity changed. The expansion ratio changed. A gating mechanism was added. But the structural role of the FFN block within each transformer layer is the same.

The overall layer structure. Attention, then FFN, repeated $N$ times. The paper stacked 6 layers. Modern models stack 32, 80, or more. But the alternating attention-FFN structure, with residual connections around each sublayer, is unchanged.

The Q/K/V formulation. The idea that the input is linearly projected into three separate roles (query, key, value) has been surprisingly durable. This is one of those things that seems like it could have gone a different way, but didn't. You could imagine architectures where Q and K are the same (symmetric attention), or where there's no separate V projection (just use the input directly as the value). People have tried these simplifications. They're worse. The three-role decomposition seems to be genuinely important for the model's expressiveness.

The training recipe structure. Adam optimizer, warmup schedule, residual dropout. The specific numbers have changed, but the general recipe (Adam-family optimizer, warmup, some form of regularization) has been persistent. AdamW replaced Adam, warmup lengths changed, dropout was dropped for large models, but the skeleton of the training setup from this paper is still recognizable in modern training runs.

What got replaced

I count six major changes between the 2017 architecture and what a typical large language model uses in 2026. Most of these happened between 2019 and 2023. It's a surprisingly short list given how many years have passed.

Post-norm became pre-norm (and then RMSNorm). The paper normalizes after the residual addition: $\text{LayerNorm}(x + \text{sublayer}(x))$. This seems like a small detail, but it turned out to matter a lot. Raschka (2023) has a nice write-up on the problem. With post-norm, the gradient path from the loss to early layers has to pass through every normalization layer. That creates vanishing gradient issues in deep networks. Pre-norm ($x + \text{sublayer}(\text{Norm}(x))$) keeps the residual path completely clean, an unobstructed highway from the top of the network to the bottom. That makes training much more stable, especially for the deep models (32+ layers) that came after the original 6-layer design.

And then RMSNorm replaced LayerNorm because the mean-centering step in LayerNorm turned out to be unnecessary. Just normalizing by the root-mean-square works fine and is faster. We covered this whole trajectory in Post 4.6. The interesting thing is that the paper's authors clearly knew about residual connections from the ResNet literature, but they placed the norm in the position that made the residual path less clean. I suspect this was inherited from convention rather than deliberately chosen.

Sinusoidal encodings became RoPE. As discussed above. Absolute position, whether sinusoidal or learned, has been almost entirely displaced by relative position schemes.

Encoder-decoder became decoder-only. For generative language models, at least. The encoder-decoder structure is still used in some settings (speech recognition, translation, and some multimodal models), but the dominant LLM architecture is decoder-only. The original paper needed both stacks because translation is a two-language problem: you need to encode the source and decode into the target. When the task became "model language itself," the two-stack design was unnecessary complexity. We covered the full set of tradeoffs in Post 4.7.

ReLU became SwiGLU. A strictly better activation function for this setting. ReLU zeros out all negative inputs, which means a large fraction of neurons can become permanently inactive ("dead neurons"). SwiGLU's smooth gating avoids this entirely.

The hyperparameters scaled. $d_\text{model}$ went from 512 to 4096 or more. Heads went from 8 to 32-128. Layers went from 6 to 32-80+. The expansion factor in the FFN changed to accommodate SwiGLU. None of this is surprising, but the specific numbers in the paper were tuned for WMT English-to-German translation, not for general language modeling.

Standard MHA became GQA/MQA. When the KV cache became a bottleneck at inference time (which the paper didn't discuss at all), the community found that sharing K/V heads across query head groups saved enormous memory with minimal quality loss. This is a good example of a modification driven by a concern the original paper never had to think about. In an encoder-decoder translation model, you generate maybe 50 tokens of output. KV cache size is irrelevant. When you're generating 8,000-token essays or 100,000-token code files, the KV cache can easily consume more memory than the model weights themselves.

What was never in the paper

This is maybe the most interesting part. The paper is about sequence transduction, specifically machine translation. Several things that people associate with "the transformer" aren't in here at all.

Next-token prediction. The training objective in this paper is sequence-to-sequence: given a source sentence, produce a target sentence. The autoregressive language modeling objective (predict the next token given all previous tokens) came from the GPT line of work a year later. This matters because the training objective shapes everything about how the model is used. A translation model takes an input and produces an output. A language model generates from a prompt. The decoder-only transformer with a causal language modeling objective is what enabled chat, code generation, reasoning, and all the things people now associate with "AI." That framing wasn't in this paper.

Scaling laws. Kaplan et al. (2020) showed that transformer loss scales as a power law with model size, dataset size, and compute. This is arguably the most important empirical finding about transformers, and it explains why the field went from training models on 8 GPUs to training them on thousands of GPUs in just a few years. It's not in the 2017 paper. The authors had no particular reason to believe that making the model 100x bigger would yield smoothly predictable improvements.

Few-shot learning and foundation models. GPT-2 showed that large language models could perform tasks they weren't explicitly trained for. GPT-3 turned this into a practical paradigm with in-context learning. None of this was anticipated in the original paper, which evaluated on BLEU scores for translation. The idea that you could give a model a handful of examples in the prompt and it would generalize was a qualitative surprise, not a predictable consequence of the architecture.

KV caching. When you generate tokens autoregressively, the K and V projections for all previous tokens don't change. So you can cache them and only compute K and V for the new token at each step. This is fundamental to efficient inference, and it follows naturally from the architecture. But the paper doesn't mention it, probably because their focus was on training speed, not inference efficiency. We'll cover KV caching in detail when we get to Arc 5.

The word "language model." I find this striking. The paper's abstract says "sequence transduction models." It talks about "translation" and "parsing." The paper is not trying to build a language model. It's trying to build a better translation model. The idea that this architecture would become the universal backbone for language modeling, code generation, reasoning, multimodal understanding, and agent systems was not on the table.

Dropout as the primary regularizer. The paper uses dropout at a rate of 0.1 on attention weights and sublayer outputs. Modern large models typically don't use dropout at all, relying instead on the regularizing effect of enormous datasets and other techniques. When your training set is the internet, overfitting is a different kind of problem.

What I find interesting about this list is that every replacement was driven by a specific practical problem that surfaced at scale. Pre-norm fixed training instability in deep networks. RoPE fixed length generalization. Decoder-only simplified the architecture for language modeling. SwiGLU improved quality per parameter. GQA reduced memory cost at inference. None of these were changes to the fundamental computational structure. They were refinements to the supporting machinery around the core attention operation.

The annotated versions

Two resources from 2018 are still the best companions to the paper. Alexander Rush's "The Annotated Transformer" reimplements it line by line in PyTorch, which is where to go for everything the paper glosses over: how masking actually gets applied, how the LR schedule is coded. Jay Alammar's "The Illustrated Transformer" is the visualization-first version, and his blog is one of the things that got me into this area in the first place.

Both describe the paper's encoder-decoder, not the decoder-only stack you'll meet in practice. Read The Annotated Transformer and then go look at Llama and the encoder half is simply gone, cross-attention with it. What's left is the same self-attention and FFN blocks, with the norms moved and the activation swapped.

A note on training details

Section 5 has one thing in it worth stopping on. The learning rate ramps up linearly during a warmup period, then decays with the inverse square root of the step:

The paper presents this as a hyperparameter choice, one formula among the others in Section 5. It escaped the paper and got informally named the "Noam scheduler" after the second author. Modern runs use cosine instead, but the warmup ramp survived, because a transformer at full learning rate from step zero tends to diverge.

They also use label smoothing at 0.1, and they're candid about the tradeoff: it "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score." Knowingly making your headline metric worse because the metric you actually care about improves is a nice bit of honesty in a results-driven paper. Modern LLMs have mostly dropped it.

So what's the legacy?

I think the paper's lasting contribution is the computational primitive itself: self-attention as a general-purpose mechanism for sequence processing. The idea that you can replace sequential recurrence with a single parallel operation that lets every position interact with every other position. That was the breakthrough, and almost everything else in the paper has been replaced.

Everything built on top of that primitive has been modified, optimized, replaced, and scaled past anything the original authors were working with, as the last two sections went through. But the core operation (computing attention weights from queries and keys, using those weights to aggregate values, doing this in parallel across multiple heads) hasn't changed. Elhage et al. (2021) formalized this into a mathematical framework for understanding transformer circuits, and even their analysis is built directly on the operations described in Section 3.2 of this paper.

I also want to note something about the paper's authorship. Eight authors from Google Brain and Google Research. Several of them went on to found AI companies (Vaswani co-founded Adept and then Essential AI; Shazeer co-founded Character.AI). The paper has been cited over 100,000 times. It's one of the most influential papers in the history of computer science by any quantitative measure. It was posted to arXiv in June 2017 and accepted at NeurIPS that same year.

Reading the paper in 2026 is a useful exercise in separating the permanent from the temporary. The permanent part is surprisingly small: one equation, one pattern (multi-head), one structural principle (alternating attention and FFN with residual connections). Everything else was reasonable for 2017 and has since been improved. That's not a criticism of the paper. That's how good research works. You get the core idea right, and the details get refined by the thousands of researchers who come after you.

There's something funny in the ratio, too. The paper introduces nearly all of this in fifteen pages. We took nine posts to unpack it. Compression like that is what a good paper looks like from the inside, and it's also why reading one cold is so hard.

One last thought. The paper's title is "Attention Is All You Need." In 2017, that was a bold claim against a field dominated by recurrence and convolution. In 2026, it reads almost like prophecy. Self-attention really is all you need, in the sense that every major language model uses it as the primary mechanism for inter-token communication. The supporting cast changed. The star of the show didn't.

Misconceptions

"The transformer paper invented attention." It didn't. Bahdanau et al. (2015) introduced attention for neural machine translation two years earlier. The paper's contribution was showing that attention alone, without recurrence, could be the entire computation. The title "Attention Is All You Need" is a claim about architecture, not a claim about novelty. Self-attention as a sequence-processing primitive (rather than an add-on to an RNN) was the new idea.

"Modern LLMs use the architecture from this paper." They use the computational primitives from it. The architecture around those primitives has been rebuilt, as the two lists above go through. Saying "GPT-4 uses the transformer architecture" is about as precise as saying a 2026 car uses the internal combustion engine from 1886. True, and not enough to build either one.

"Transformers replaced RNNs because they were more accurate." This one is half right. In the paper, the transformer beats the best RNN-based systems on BLEU, so accuracy is part of the story. But if it had only been a 1 BLEU bump on translation, the architecture would have been a footnote. What killed RNNs was parallelism. The transformer trained in 3.5 days where its predecessors took weeks, and that gap widens further as you scale up. Once the field figured out it could throw 10x more compute at the same wall-clock training budget, the RNN was structurally locked out of the race.

What's next

One post left in Arc 4. Post 4.9 covers Mixture-of-Experts layers, which replace the dense FFN with sparse expert routing. MoE is the big architectural change since the paper that doesn't fit neatly on the "what got replaced" list, because it's not a one-for-one swap. It's an orthogonal axis along which the FFN block grew.

Then Arc 5 moves into the inference-time world the original paper didn't discuss at all: decoding strategies, KV caching, speculative decoding, and all the things that happen between "model produces logits" and "user sees text."

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. The paper this post is a close reading of. Pitched as a faster translation model; became the architecture behind every frontier LLM.
• Rush, A. (2018). The Annotated Transformer. Harvard NLP. Line-by-line PyTorch reimplementation of the paper — the best single resource for the implementation details the paper glosses over.
• Alammar, J. (2018). The Illustrated Transformer. Visualization-first walkthrough of Q/K/V and multi-head attention. Still the clearest visual introduction to the architecture.
• Raschka, S. (2023). About LayerNorm Variants in the Original Transformer Paper. Clean explanation of why post-norm was replaced by pre-norm, and why the original diagram is quietly misleading.
• Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. The original GPT paper — the architectural move from encoder-decoder to decoder-only that the Vaswani paper didn't take.
• Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. The attention mechanism the transformer was built to generalize — RNN seq2seq with soft alignment over encoder states.
• Elhage, N., et al. (2021). A Mathematical Framework for Transformer Circuits. Anthropic. Rewrites the paper's attention operation as a mathematical framework for reasoning about what specific heads and layers compute.
• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. Same scaled dot-product attention from Section 3.2; faster execution by fusing softmax and matmul into one IO-aware kernel.
• Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. The SwiGLU paper; introduced the gated FFN variant that replaced ReLU+two-linear in modern LLMs.