LLMs from highest to lowest (the teaching spine)

This is the map you should be able to walk in either direction. Highest first so you know why the lowest matters. You study lowest first so you are not lying at the top.

L5 — The bet

The bitter lesson (Sutton): general methods that scale with compute beat hand-built knowledge. The scaling hypothesis is the empirical version: more data, more parameters, more (useful) FLOPs → more capability, smoothly enough to plan a company around.

Anthropic's public bet is: that curve is real and the systems on it must be steerable. Constitutional AI, RSP, interpretability, and classifiers are not a side quest. They are how they think you get to AGI without a catastrophe.

Kernel work sits on the bet like this: useful FLOPs per dollar decide how far a given cluster moves you along the curve. A 1.3× decode win is a capability paper that never has to be written. That is not the same as "GPUs are AGI." Algorithms, data, and alignment still decide what you scale.

You should be able to say what you believe about this in 90 seconds, including one thing you are unsure about.

L4 — What an LLM is

A language model is a probability distribution $p(x_1,\ldots,x_n)$ over token sequences, usually factored autoregressively:

\[ p(x) = \prod_t p(x_t \mid x_{<t}) \]

Training maximizes likelihood of data, i.e. minimizes cross-entropy. Perplexity is $\exp(\text{mean nats})$. Bits-per-byte is the tokenization-fair version.

Tokens are not words. BPE / Unigram / SentencePiece cut UTF-8 into a fixed vocabulary so the softmax is tractable. The tokenizer is part of the model. A kernel person who ignores it will mis-cost sequences (a "4k context" is 4k tokens, whose byte length varies by language).

Everything after 2017 that matters for this job is a decoder-only transformer predicting the next token. Encoder-decoder (the original Vaswani NMT setup) lost for pretrained general models. The reason is mundane and important: one stack, one objective, one KV cache story, infinite continuation.

L3 — The residual stream

Think of the model as a residual stream of shape [batch, seq, d_model]. Each layer does:

x = x + Attn(Norm(x))
x = x + MLP(Norm(x))

Pre-LN (norm on the incoming stream) is the 2024–2026 default because Post-LN blows up at depth. RMSNorm beat LayerNorm for cost: no mean subtract, no bias, one RMS and a scale vector. Same idea, fewer bytes.

Attention mixes across sequence. MLP mixes across channels, tokenwise. Almost all FLOPs are in the two matmuls of the MLP (especially with SwiGLU, which is three projections: up, gate, down) and the QKV/out projections. Attention's FLOPs are $O(n^2 d)$ but its bytes are the problem that created FlashAttention.

Multi-head: split d_model into h heads of d_head. GQA/MQA: fewer key/value heads than query heads, because decode is memory-bound on the KV cache and you can share K/V across query groups with little quality loss. MLA (DeepSeek) compresses KV further. Each of these is an architecture change that is really a kernel/memory change.

Positions: RoPE rotates Q/K pairs so inner products depend on relative offset. ALiBi adds a distance bias. Absolute learned embeddings are mostly gone at frontier. NoPE is a research direction, not your default.

MoE: replace the MLP with a routed set of MLPs. FLOPs per token stay similar, parameters explode, communication becomes all-to-all, kernels become grouped GEMM. You will meet this.

L2 — Attention, the thing you must derive

\[ \mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V \]

Shapes, one head: Q,K,V are [B, S, d]. QK^T is [B, S, S]. That matrix is the villain. Standard implementation writes it to HBM. For S=8k that is huge, and you do it per head per layer.

1/sqrt(d) keeps the dots from saturating the softmax as d grows (assume unit-variance coords, the dot has variance d).

Softmax is $\frac{e^{z_i}}{\sum_j e^{z_j}}$. Numerically you must subtract max(z) or you overflow in fp16. Online softmax (running max m and sum l) lets you stream the score matrix in tiles and still get the exact result. That is the FlashAttention trick, plus never writing S×S to HBM, plus fusing the matmul+softmax+matmul into one kernel with SRAM tiles.

Causal mask: future positions get -inf before softmax so a token cannot see the future. Training uses it on the full sequence. Decode does not need a mask if you only score the new query against cached K.

Backward: you need dQ, dK, dV. FlashAttention's backward recomputes tiles instead of storing the full S×S. That is why it is a memory win in training, not only in inference.

If you cannot do this on a whiteboard with shapes, FLOPs, and bytes, you are not in L2 yet.

L1 — Training vs inference are different programs

Training: all positions in parallel, causal mask, store activations for backward, optimizer states (AdamW ≈ 8 bytes/param extra besides weights), mixed precision, data parallel + tensor parallel + pipeline + maybe sequence/context parallel. The hard systems problems are: memory of the optimizer + grads + activations, comm overlap, loss spikes, and not NaN-ing a $10M job.

Inference, prefill: like a forward training pass on the prompt. Compute-heavy. Batch by tokens, not requests.

Inference, decode: one new token per request per step. Tiny GEMMs, huge KV reads. Memory-bound. Continuous batching exists because a request in prefill and a request in decode should not be in the same naive batch.

KV cache bytes (fp16):

\[ 2 \times n_{layers} \times n_{kv\_heads} \times s \times d_{head} \times 2 \times B \]

(the leading 2 is K and V). This is why GQA, paging, prefix cache, and quantization of KV are the serving game.

Speculative decoding: a cheap draft model proposes k tokens, the big model verifies in one parallel forward. Kernel implication: verification is a prefill-shaped chunk, not k decode steps.

Quantization: weights in fp8/int8/int4 reduce bytes and can use tensor-core paths. The PE job is the kernel + the correctness eval. Anthropic's inference-systems req literally treats an output regression from a quant change as a performance bug.

L0 — The metal

A GPU is not a fast CPU. It is a throughput machine: many SMs, each running warps of 32 threads, hiding latency with occupancy. Peak math (tensor cores) is far above peak HBM. If your kernel is not tiled into SRAM / shared memory / TMEM, you are leaving most of the chip idle.

Roofline: attainable FLOP/s = min(peak FLOP/s, peak bytes/s × arithmetic intensity). Intensity = FLOPs / bytes moved from HBM. Decode GEMMs sit on the memory wall. Big prefill GEMMs can sit on the compute roof. Fusion exists to cut bytes. Tiling exists to cut bytes. FlashAttention exists to cut bytes.

Numerics: tf32/bf16/fp16/fp8 have different mantissa widths. Accumulators are usually wider than the inputs (fp32 acc for fp16/bf16 GEMM, sometimes fp16 acc for fp8 with care). Softmax in low precision is how you get inf. Loss scaling exists because fp16 grads vanish. Determinism is mostly a lie once you have atomics and async.

The Anthropic take-home strips this down to a teaching machine: scratch instead of cache, VLIW instead of out-of-order, SIMD instead of warps. Same sport.

The loop you must be able to run

Write the math (shapes, exactness).
Cost it (FLOPs, bytes, comm).
Roofline: what should the time be?
Implement the simple thing.
Measure. The counter, not the story.
Change one thing (tile, fuse, pack, quantize, overlap).
Prove correctness (bitwise where you can, allclose where you must, eval where it matters).
Write the new number and why.

That loop is the job. The papers are so you know which change to try. The take-home is so they can see you run the loop on a machine you have never seen.

What to do with this file

Do not highlight it. Walk it backwards: pick L0 tomorrow (you already are, in month 1). When you can teach a section out loud with a blank page, mark it. When you cannot, that section becomes Deep 1.

A longer annotated corpus of papers and hour estimates is landing in 01-prior-art/ as the research pass finishes.