Holding the LLM Stack in Your Head

This is a post about how and why I wrote Holding the LLM Stack in Your Head, a dependency-ordered series of about a hundred posts walking through the full LLM stack from linear algebra to agent protocols. If you just want to read the series, start here.

Earlier this year I was preparing for a new role at Google, and I had a moment where I realized I'd been building on top of layers I don't actually understand. I was afraid someone would ask me something specific that required me to know the details of the stack (what prefill really does, why a KV cache is its own object, what's going on inside paged attention) and I'd have to go find a doc, then a paper, then realize that I don't really know what I'm doing.

That bothered me. I wanted every layer of the stack to just be there in my head when I thought about it, without the lookup detour. I knew that I couldn't become an encyclopedia, but I wanted to have a mental model of the stack that let me get stuff done without having to look things up all the time.

I'm also into the idea of learning things fast. Something like this same process is how I ended up with a bunch of AWS certifications and a habit of diving into things way too fast. But this felt bigger. The LLM stack is a huge dependency graph, and the only way I've found to really internalize a dependency graph is to walk it in order, building each piece on top of the last, and forcing myself to explain each one before I move on.

So I decided to just write. Starting with the most basic building blocks and working my way up. The writing was the learning. Quickly the writing became boring, so I started collaborating with Claude to add visualizations to keep myself engaged, and now there are over 600 visualizations in the series.

The gap in the educational content

As I was working through the material, I noticed that there was a gap in the educational content. Most of what's already out there is good. Individually, a lot of it is excellent. Karpathy's Zero to Hero is an incredible resource for learning how you might build a transformer from scratch. Jay Alammar's illustrated essays are the best visual explanations of attention. Chip Huyen's AI Engineering book is a thorough builder's view of the application layer. I learned from all of them and I'll link to them throughout the series.

But none of them, individually or together, gave me the throughline I wanted. The chain from full-sequence attention during training to one-row decode during inference to paged-attention memory managers to the KV cache lifecycle to how that cache interacts with continuous batching and prefix caching to fine-tuning and tool use and agents to governance and scaling laws. That chain is nowhere taught end to end.

The same pattern shows up everywhere. Papers explain experiments and mathematical objects but leave out the messy details of runtime. API docs explain surfaces but assume the mental model you're still building. Runtime docs explain narrow serving details with no theory above them. Blogs explain one piece of the puzzle but not the whole thing.

The existing material is locally excellent but doesn't connect end to end. I couldn't find a single throughline from linear algebra to agent protocols where every layer earns the next one and intuition is the actual deliverable. So I started writing the one I wanted.

What the series looks like

Here's the map:

Twelve arcs, about a hundred posts. Each arc builds on the ones before it:

The first four arcs lay foundations. Math prerequisites, language modeling before transformers (because understanding why recurrence hit a wall is how you understand why attention exists), tokenization and the input pipeline, and then transformers from first principles (residual stream, attention, masking, positional encodings, the FFN block, MoE).

Arcs 5 and 6 are the inference story. Training view vs inference view, prefill vs decode, the KV cache derived from first principles, sampling, speculative decoding, continuous batching, then the whole serving systems layer: kernel fusion, PagedAttention, quantization, parallelism strategies, engine comparison.

Arc 7 is training and post-training. Pretraining data, SFT, RLHF, DPO, constitutional AI, tool-use fine-tuning, LoRA. Arc 8 is evaluation, because evaluation methodology is more broken than it looks, and how broken it is matters a lot.

Arcs 9 and 10 are retrieval and agents. Embeddings from scratch, vector search, chunking, rerankers, RAG architectures, context engineering, then the agent loop, function calling, MCP, multi-agent systems.

Arc 11 covers the provider ecosystems in practice. Arc 12 is governance, economics, scaling laws, the education gap, and an honest look at what comes next.

Every post follows the same eight-part spine: the problem this layer solves, prerequisites, minimal math, runtime behavior, a worked example, an implementation sketch, common misconceptions, and what changed by 2026. The uniform structure is on purpose, largely because it helped me stay on track and not get lost in the weeds.

How this actually got written

I should be upfront about the process, because it's half the point.

Every post started from my outline, my framing, and my working intuition. And I leaned heavily on AI through the whole thing. I maxed out a Claude Pro subscription and a Google AI Ultra subscription every day for three months and used them for source curation, visual iteration, prose critique, learning concepts I didn't understand well enough, finding and reading primary sources, bouncing explanations off the model until the framing felt right, iterating on the 600+ interactive visualizations, debugging code, pressure-testing my understanding of things I thought I knew but actually had gaps in. Pretending I did this alone would be dishonest. The AI assistance was a core part of the workflow.

I was heavily inspired by Andrej Karpathy's recent post about the idea file, so I layered that into my process. The idea.md in this repo lays out my philosophy and approach to writing and teaching, my goals for the series, the voice I tried to strike, a study of the reader's mind and what I wanted to give them, and the ground rules I set for myself, as well as specific guidance about the kind of content I wanted to create, like the visuals and the code.

Then came three months of work, every day, late into the night. I'd read a draft, flag what was dull or hand-wavy, flag where a concept was getting used before it had been earned, and hand it back. Visuals went through the same loop. Prose got tuned paragraph by paragraph until the voice was right and the model in my head actually matched the model on the page.

The key idea is that the critique loop is what put the stack in my head. Tuning a paged-attention explanation thirty times until it teaches the right thing is, in practice, how you learn paged attention. The blog is sort of the record of that work. And I hope it serves as a guide for others about how to approach learning and teaching complex systems with AI assistance.

The visuals

The other thing is the visual side, because it took a lot more engineering time than I expected.

The idea itself was simple though. Prose is the default, but it is boring, and it doesn't stick as well. So we make a static diagram when a concept is fundamentally spatial, an animation when it's fundamentally temporal, and an interactive piece when you have to drag a parameter to actually feel what it does, and we are extremely liberal with the visuals, the more the better.

The process produced over 600 custom React components. About ninety of those are themed hero loops, one per post, setting the visual vibe of the topic before you start reading. A couple hundred more are interactive or animated pieces that do real teaching work inside the prose. The rest are supporting primitives. None of it came from a diagram library; each piece got built because a specific post wanted a specific intuition. Doing this as one person only became tractable in the last year or so, which is part of why I wanted to do it now.

Here's one from the very first post in the series. A 2x2 matrix, editable, with presets for rotation, shear, projection, and scaling. The columns of the matrix are exactly where the basis vectors land, and in my experience that sentence only really lands once you've played with it for a minute.

How to read it

You don't have to start at Arc 1. Start wherever your stack is.

If you want the inference story, the fastest path is prefill vs decode → the one new row → the KV cache from first principles → PagedAttention. Four posts and the inference layer will feel different after.

If you're building with RAG, start at embeddings, go through chunking and rerankers, then land at end-to-end RAG.

If you're building agents, the core sequence is history of agents → function calling → the agent loop → MCP.

If you want to understand the transformer from the ground up, start at vectors and spaces, walk into self-attention from scratch, and let Arc 4 pull you through multi-head attention, masking, positional encodings, and the FFN block.

The series index has curated paths for each of these, plus the full map of all twelve arcs.

One more thing: dense material like this works best with breaks. Your brain needs time to consolidate what you just read before you stack the next layer on top. If you hit a wall, step away. I built a little wave-survival game called Squaretato for exactly this purpose (there's a link in the nav bar). Ten minutes of dodging geometric swarms, then come back and the next post will land better.

This started as a personal project to get the stack into my own head. The map I wish I'd had going in is the one I ended up writing. I think it's worth sharing.