Choose a preset or edit the numbers directly. The columns of the matrix are exactly where the basis vectors land.
The Toy Box
Forty interactive pieces, out of their boxes
These forty examples come from the series. Adjust a value, step through a calculation, or inspect a diagram. Each one links back to the post that explains its assumptions and use.
Foundations
The math and the hardware everything else stands on.
Each output cell is one dot product: a row of A against a column of B. Step through the products and their sum.
Compare SGD, momentum, and Adam on the same surface. Their paths depend on the chosen learning rates; this example does not establish a best optimizer.
Drag T. Low temperature collapses the distribution onto the winner; high flattens it toward uniform. The ranking never changes.
Try 65504, 65519, and 65520 to inspect rounding at the fp16 overflow boundary. Compare the exponent range and precision with bfloat16.
Language before transformers
What prediction looked like before attention, and why it worked.
Inspect hand-chosen character distributions as context grows. The entropy and observed-character loss are calculated from these toy probabilities, not measured from English.
Select an analogy and inspect the vector arithmetic. These hand-placed two-dimensional points illustrate offsets; they are not trained embeddings.
Step through supplied gate values and calculate the updated cell and hidden states. The forget gate scales old values; the input gate controls what is added.
Tokens
How text becomes the token IDs a model receives.
One family emoji: 1 grapheme, 7 code points, 25 bytes. Choose an example and watch the three layers come apart.
Follow the most frequent adjacent pair through successive merges in a small training corpus.
Apply learned merge ranks to a new input. The base vocabulary determines which initial symbols can be represented.
Use dynamic programming to find the highest-scoring segmentation among the vocabulary pieces available at each position.
Inside the transformer
The machine itself, one block at a time.
Four tokens, five steps. Select a token and trace its query through the scores, the softmax, and the weighted sum.
Toggle bidirectional against causal. Each token sees one more position than the last, and the upper triangle is exactly zero.
Three dimension pairs, three speeds. The fast pairs encode fine position, the slow pairs encode coarse. That's RoPE.
Select a neuron: its row of W₁ defines a pattern, and its activation scales the corresponding W₂ column added to the output.
Suppress features and inspect the computed logits in this constructed example. It does not identify where a real model stores a fact.
Each token routes to a subset of experts, reducing active feed-forward computation. Router and communication costs still matter.
Inference, one token at a time
What actually happens when a model generates.
Advance decoding and inspect the new query row over the growing key sequence. Cached keys and values are reused; old attention rows are not the KV cache.
Change the model, KV-head count, and sequence length to calculate cache storage. Head sharing reduces this cost, but weights and other buffers also need memory.
Temperature, top-k, top-p, and min-p on live bars. Set a combination, then select Sample and draw a token.
Inspect a draft-and-verify round. Exact speculative sampling preserves the target distribution when implemented correctly; acceptance rates and verification costs determine the speedup.
Step through allowed choices in a small JSON grammar. Constraining syntax does not guarantee a correct answer, and generation can still stop before completion.
Serving systems
The engineering that turns a model into a service.
Compare a fixed batch with a scheduler that admits waiting requests as slots become free. This toy schedule assumes work is ready to run.
Three requests scattered through a shared block pool, each holding a tiny block table. This is PagedAttention.
Two requests share a system prompt until one writes. Only the written block gets cloned; everything else stays shared.
Inspect score tiles and the running softmax statistics. The full score matrix need not be materialized in HBM.
Estimate weight and KV-cache storage under explicit assumptions. Leave room for workspaces, activations, and runtime overhead before choosing concurrency.
Change the batch size under the displayed cost assumptions to compare throughput with per-request latency. The useful operating point depends on the workload and latency target.
Training & post-training
How the weights get made, and made agreeable.
Calculate assistant-only target loss. Gradients can still pass through computations that used the masked prompt tokens.
Change the probability ratio and compare both advantage signs in the clipped surrogate. Clipping limits part of the objective; it does not guarantee stable training or a hard trust region.
Explore a constructed routing-feedback example. Its selected rules illustrate collapse; they do not predict that real training must reach the same allocation.
Retrieval & context
Getting the right information in front of the model.
Follow a constructed question through the retrieval pipeline and inspect the data passed between stages.
HyDE embeds a generated hypothetical document to retrieve source passages. Inspect the example; the generated text is not evidence and may lead retrieval astray.
Old turns compact into a running digest while the last few stay verbatim. The model's view stays small as the conversation grows.
Agents & protocols
The loop, and the standards forming around it.
Follow generation, runtime dispatch, tool results, and resumption. This example keeps the model’s supplied state in the transcript.
Compare three independent calls under the displayed duration assumptions. Parallel execution can reduce the wait when dependencies and service limits permit it.
Compare N×M bespoke connections with N+M protocol adapters in this schematic count. Standardization does not remove deployment, compatibility, or authorization work.
Step through a schematic initialization, tool discovery, and call. Transport and authorization requirements also matter for a working connection.
Follow the code challenge and verifier in a PKCE exchange. The verifier binds redemption to the client that started it; other OAuth checks are still required.