The Series
Holding the LLM Stack in Your Head
A dependency-ordered walk through the modern LLM stack, from the linear algebra under a single attention head, through training and inference, out to tool use and agent protocols. Ten arcs, eighty-odd posts.
The ten arcs
- 01Vectors, Matrices, and the Spaces They Live In
Vectors as lists of activations, matrix multiplication as a linear map, and how these operations fit into neural networks.
- 02Norms, Dot Products, and Similarity
How vector length, dot products, cosine similarity, and projections support attention scores and embedding comparisons.
- 03Distributions, Softmax, and the Chain Rule of Words
How softmax produces a categorical distribution, how Bayes' rule updates probabilities, and how the chain rule assigns probabilities to sequences.
- 04Cross-Entropy, KL Divergence, and What Loss Functions Measure
Why cross-entropy is the standard LM loss, what it actually measures about two distributions, and how it connects to perplexity.
- 05Gradients and How Machines Learn
What a gradient is, why it points uphill, how backpropagation computes one efficiently via the chain rule, and what SGD does with it.
- 06Optimizers: Momentum, Adam, and Learning Rate Schedules
How momentum and Adam use gradient history, and how warmup and learning-rate schedules affect training.
- 07GPUs, Floating Point, and Why Precision Matters
IEEE 754, the difference between fp32/fp16/bfloat16, why mixed-precision training works, and the basics of GPU parallelism.
- 08A Short Prehistory of Statistical NLP
How rule-based, statistical, and early neural methods represented language and learned from text.
Don't know where to start?
If you want to understand attention
Follow matrix operations into attention scores and multiple attention heads.
If you care about why inference is slow
Start from the one-new-row insight, then follow the KV cache and the systems built around it.
If you're building with RAG
Follow embeddings, document chunks, and reranking into a retrieval pipeline.
If you're building agents
The loop, the protocol, the transcript formats, and what the model actually sees.
The series is a living document. If you spot something wrong, contact info is here.