The Series

Holding the LLM Stack in Your Head

A dependency-ordered walk through the modern LLM stack, from the linear algebra under a single attention head, through training and inference, out to agent protocols shipping in 2026. Twelve arcs, about a hundred posts. The goal isn't rigor, it's intuition that survives contact with real systems.

Why I wrote this →See it as a mapWho I am

The twelve arcs

  1. 01Vectors, Matrices, and the Spaces They Live In

    Vectors as lists of activations, matrix multiplication as a linear map, and why every neural network operation bottoms out in matmuls.

  2. 02Norms, Dot Products, and Similarity

    How cosine similarity, L2 distance, and projections work, and why they show up everywhere from attention scores to embedding retrieval.

  3. 03Distributions, Softmax, and the Chain Rule of Words

    Softmax, categorical distributions, Bayes' rule, and the chain rule of probability — the four tools that make language modeling a well-defined math problem.

  4. 04Cross-Entropy, KL Divergence, and What Loss Functions Measure

    Why cross-entropy is the standard LM loss, what it actually measures about two distributions, and how it connects to perplexity.

  5. 05Gradients and How Machines Learn

    What a gradient is, why it points uphill, how backpropagation computes one efficiently via the chain rule, and what SGD does with it.

  6. 06Optimizers: Momentum, Adam, and Learning Rate Schedules

    Why vanilla SGD is too slow, how Adam adapts per-parameter, and how warmup and cosine decay shape training dynamics.

  7. 07GPUs, Floating Point, and Why Precision Matters

    IEEE 754, the difference between fp32/fp16/bfloat16, why mixed-precision training works, and the basics of GPU parallelism.

  8. 08A Short Prehistory of Statistical NLP

    The arc from rule-based systems through statistical MT and log-linear models to neural approaches, giving you historical context for everything that follows.

Don't know where to start?

If you want to understand attention

The minimum path to really seeing how a transformer layer works, not just reciting the shapes.

  1. 1Vectors, Matrices, and the Spaces They Live In
  2. 2Self-Attention: Q, K, V from First Principles
  3. 3Multi-Head Attention and Representation Subspaces

If you care about why inference is slow

Start from the one-new-row insight, then follow the KV cache and the systems built around it.

  1. 1Prefill vs. Decode: The Two Phases of Inference
  2. 2Why One New Token Means One New Row
  3. 3The KV Cache from First Principles
  4. 4PagedAttention and the vLLM Insight

If you're building with RAG

Just enough retrieval theory to make the engineering decisions actually make sense.

  1. 1Embeddings from Scratch: From Word2Vec to E5
  2. 2Chunking Strategies
  3. 3Rerankers and Cross-Encoders
  4. 4RAG Architectures End to End

If you're building agents

The loop, the protocol, the transcript formats, and what the model actually sees.

  1. 1A Short History of Agents: From ReAct to 2026
  2. 2Function Calling as Structured Generation
  3. 3The Agent Loop: Model, Runtime, Tool, Resume
  4. 4MCP: A Cross-System Standard for Tool Integration

The series is a full first draft. I'll be grinding through polish, corrections, and the odd rewrite. If you spot something wrong, contact info is here.