Everything About LLMs, Highest → Lowest

A no-shortcut curriculum for AGI research + Anthropic research-engineer / kernel-adjacent interviews As of 2026-08-18. All citations below are real papers/pages with URLs. Do not skip the derivations.

Who this is for. Someone who wants to (1) contribute to frontier AGI work in the Anthropic style — capabilities and alignment as one engineering problem — and (2) pass a research-engineer interview that will put you on a whiteboard deriving softmax(QKᵀ/√d)V, then ask why your FlashAttention kernel NaNs at 100B, then ask how Constitutional AI’s RLAIF loop differs from InstructGPT’s PPO loop.

How to use it. Each section is a gate. You are not done when you have “read” the papers. You are done when you can do the learning outcomes cold, on a whiteboard, and have shipped the labs. Estimated deep hours are after you already have undergrad linear algebra, probability, and C++/Python fluency. Total: ~900–1,200 deep hours (6–12 months at 20–25 h/week, or a brutal 4 months full-time).

Prerequisites (do these first if shaky, ~80–120 h):

• Linear algebra: SVD, eigenvalues, inner products, Kronecker, broadcasting as tensor diagrams.
• Probability: entropy, KL, Jensen, concentration; Cover & Thomas Ch. 2–5.
• Calculus / autodiff: chain rule on tensors; implement backprop for a 2-layer MLP from scratch.
• Systems: caches, SIMD, occupancy, bandwidth vs compute; know what a GEMM is.

A. Intelligence / AGI framing (Anthropic-flavored)

~80–100 deep hours

Thesis

Intelligence, on this bet, is mostly what you get when you scale general methods that turn compute into prediction, plus the engineering that keeps the resulting systems pointed at human-chosen goals. Alignment is not a separate moral philosophy class. It is the same stack as capabilities: data, loss, optimizer, evals, and the kernels that make the next 10× of compute usable. Kernel work is AGI work because efficiency unlocks scale, and scale is the only reliably observed path to new capabilities.

Learning outcomes

You can:

1. State Sutton’s bitter lesson in one paragraph, then steelman the opposite (human-knowledge priors, architecture search, interpretability as “knowledge”).
2. Write Kaplan’s three power laws and Chinchilla’s compute-optimal allocation from memory, including what each assumes and where it broke.
3. Explain Anthropic’s public bet: scaling is real; safety must be empirical, multi-method, and differentially accelerated; HHH (helpful, honest, harmless) is the near-term target; interpretability + scalable oversight + process supervision are the current research bets.
4. Argue, with FLOPs, why a 2× kernel speedup on attention is an alignment intervention (it buys another generation of evals, more RL, more SAE training, more red-teaming).
5. Distinguish capabilities vs alignment as joint engineering from “first make it smart, then bolt on safety.”

Must-read (in this order)

Framing

• Sutton, R. (2019). The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Amodei, D. et al. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. https://arxiv.org/abs/1606.06565
• Amodei, D. (2024). Machines of Loving Grace. https://darioamodei.com/machines-of-loving-grace
• Anthropic (2023). Core Views on AI Safety. https://www.anthropic.com/research/core-views-on-ai-safety
• Anthropic (2023). Claude’s Constitution. https://www.anthropic.com/news/claudes-constitution
• Anthropic (2024–). Responsible Scaling Policy (read the current version). https://www.anthropic.com/news/anthropics-responsible-scaling-policy

Scaling as the empirical law of the field

• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). arXiv:2203.15556. https://arxiv.org/abs/2203.15556
• Henighan, T. et al. (2020). Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701. https://arxiv.org/abs/2010.14701
• Wei, J. et al. (2022). Emergent Abilities of Large Language Models. arXiv:2206.07682. https://arxiv.org/abs/2206.07682
• Schaeffer, R., Miranda, B., Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? arXiv:2304.15004. https://arxiv.org/abs/2304.15004

Anthropic’s early alignment lab

• Askell, A. et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861. https://arxiv.org/abs/2112.00861
• Bai, Y. et al. (2022). Training a Helpful and Harmless Assistant with RLHF. arXiv:2204.05862. https://arxiv.org/abs/2204.05862
• Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. https://arxiv.org/abs/2212.08073

Textbooks / courses

• Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), Ch. 1–3, 13. http://incompleteideas.net/book/the-book.html
• Russell & Norvig, AIMA, Ch. 1–2 (what “intelligence” was supposed to mean).
• Leopold Aschenbrenner, Situational Awareness (2024) — read as a capabilities-maximalist foil, not gospel. https://situational-awareness.ai/

Derivation exercises

1. Kaplan vs Chinchilla (pen + paper). Write \(L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}\). Under \(C\approx 6ND\), minimize \(L\) at fixed \(C\). Show Chinchilla’s equal scaling of \(N\) and \(D\). Then explain, in writing, why frontier labs in 2024–2026 overtrain relative to Chinchilla (inference cost, data quality, repeated epochs).
2. Bitter-lesson audit. Pick three “smart” architectural ideas from 2017–2023 that died (relative PE variants, recurrence hybrids, retrieval-as-architecture). Write one page: what compute did they lose to?
3. Efficiency → scale. A 70B dense model, 8k context, MFU 40% on a given cluster. Attention is 35% of step time. You write a kernel that is 1.8× faster on attention. Compute the new tokens/day and the equivalent “free” model-size increase under Kaplan \(L\sim N^{-0.076}\).

Coding labs

• Lab A1. Reproduce Kaplan Fig. 1 qualitatively on a toy decoder (nanoGPT-scale): 4 widths × 4 token budgets. Fit a power law. Write down where the fit lies.
• Lab A2. Read Anthropic’s constitution and the CAI paper’s appendix principles. Implement a tiny “critique → revise” loop with a small open model. The point is the procedure, not the product.

Whiteboard questions Anthropic would ask

• “Kaplan said big models, little data. Chinchilla said 20 tokens/param. Llama-3 trained way past that. Who was wrong, and what objective were they optimizing?”
• “Why is a kernel engineer on the critical path for alignment?”
• “Name a capability that appeared only after a scale jump, and an alleged emergence that was a metric artifact.”
• “HHH: give a prompt where helpfulness and harmlessness conflict. How should a constitution resolve it? How would you measure that you did?”

B. Language modeling from first principles

~90–110 deep hours

Thesis

A language model is a conditional distribution \(p(x_t\mid x_{<t})\) over tokens. Everything else — transformers, RLHF, kernels — is machinery for representing, training, and sampling that distribution. If you cannot talk about cross-entropy, perplexity, entropy, KL, and bits-per-byte without waving your hands, you do not understand the loss you will spend a career minimizing.

Learning outcomes

You can:

1. Derive the MLE ≡ cross-entropy ≡ KL-to-data-distribution identity.
2. Convert freely among nats, bits, perplexity, BPB; explain why BPB is tokenizer-invariant and perplexity is not.
3. Implement BPE and Unigram LM from scratch; explain SentencePiece’s “raw text, no pretokenizer” bet and byte-level BPE’s unk-free bet.
4. Narrate the line n-grams → Bengio neural LM → RNN/LSTM → Transformer, with the inductive bias each discarded.
5. Explain smoothing, backoff, and why n-grams hit a wall that was statistical, not just computational.

Must-read

• Shannon, C. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal. (Classic; any reprint.)
• Bengio, Y. et al. (2003). A Neural Probabilistic Language Model. JMLR. https://www.jmlr.org/papers/v3/bengio03a.html
• Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation. https://www.bioinf.jku.at/publications/older/2604.pdf
• Sennrich, R., Haddow, B., Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909. https://arxiv.org/abs/1508.07909
• Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. arXiv:1804.10959. https://arxiv.org/abs/1804.10959
• Kudo, T. & Richardson, J. (2018). SentencePiece. arXiv:1808.06226. https://arxiv.org/abs/1808.06226
• Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
• Brown, T. et al. (2020). Language Models are Few-Shot Learners (GPT-3). arXiv:2005.14165. https://arxiv.org/abs/2005.14165

Textbooks / courses

• Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Ch. 3 (n-grams), 6–7 (neural LMs), 9–10. https://web.stanford.edu/~jurafsky/slp3/
• Cover & Thomas, Elements of Information Theory, Ch. 2 (entropy), 8 (typical sets), 13 (universal source coding).
• MacKay, Information Theory, Inference, and Learning Algorithms, Ch. 4–8. https://www.inference.org.uk/mackay/itila/
• Stanford CS224N (latest). https://web.stanford.edu/class/cs224n/
• Andrej Karpathy, Neural Networks: Zero to Hero + nanoGPT. https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nanoGPT

Derivation exercises

1. CE = H + KL. Show \(\mathbb{E}_{x\sim p^\star}[-\log q(x)] = H(p^\star)+D_{\mathrm{KL}}(p^\star\|q)\). Interpret training as “matching the data’s next-token distribution.”
2. Perplexity. \(\mathrm{PPL}=\exp(\overline{\mathrm{CE}}_{\text{nats}})=2^{\overline{\mathrm{CE}}_{\text{bits}}}\). Why comparing PPL across GPT-2 BPE vs Llama SentencePiece is meaningless; how to compute BPB.
3. BPE by hand. Corpus {low, low, lowest, newer, wider}. Run 6 merges. Then encode lowest. Then implement it.
4. Unigram LM. Write the EM objective. How do you sample a segmentation (subword regularization)? Why is this closer to a proper LM than BPE?
5. LSTM gates. Derive the gradient of the cell state through time. Show why the forget-gate identity path beats vanilla RNN vanishing.

Coding labs

• Lab B1. Shannon game: compress 100k chars of English with (a) char 5-gram Kneser–Ney, (b) a 2-layer LSTM, (c) a 4-layer transformer. Report BPB. You must beat the n-gram.
• Lab B2. Train BPE (10k merges) and Unigram (10k vocab) on the same corpus. Measure fertility, unk rate, and downstream tiny-LM CE. Write a one-page “what tokenization actually buys.”
• Lab B3. Byte-level BPE from scratch (GPT-2 style). Verify invertibility on random UTF-8 including emoji.

Whiteboard

• “Perplexity of 12 on WikiText-103 — is that good? Compared to what? In bits per byte?”
• “Why did RNNs lose? Give the parallelization argument and the path-length argument.”
• “BPE is a compression algorithm. Why do we use it as a front end to a neural LM that is itself a compressor?”
• “Show me that maximizing likelihood minimizes KL to the data. Where does this fail as a human objective?” (bridge to §E)

C. Transformer internals (whiteboard-derivable)

~140–170 deep hours

Thesis

A decoder-only transformer is a residual stream that attention and MLPs read from and write to. Attention is a soft, content-addressable lookup whose scores are scaled dot products. Almost every modern variant (GQA, MQA, MLA, SwiGLU, RoPE, RMSNorm, Pre-LN, MoE) is a systems or stability patch on that picture, not a new theory of intelligence.

Learning outcomes

You can, on a whiteboard with no notes:

1. Derive \(\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(QK^\top/\sqrt{d})V\), including why \(\sqrt{d}\), complexity \(O(n^2 d + nd^2)\), and the three numerical failure modes (overflow, underflow to one-hot, row-sum cancellation).
2. Draw multi-head; then GQA; then MQA; then MLA. Compute KV-cache bytes/token for each.
3. Write SwiGLU and the \(2/3\) width rule that keeps param count matched.
4. Contrast Post-LN vs Pre-LN gradient scales at init (Xiong et al.).
5. Derive RoPE: rotation in 2-D planes, \(\langle R_{\theta i}q, R_{\theta j}k\rangle=\langle q,R_{\theta(j-i)}k\rangle\). Compare ALiBi’s linear bias, sinusoid add, and NoPE.
6. Write LayerNorm and RMSNorm; say what invariance you lose.
7. Explain why decoder-only won for pretrained general assistants (parallel training, unified interface, zero-shot after LM pretraining).

Must-read

• Vaswani, A. et al. (2017). Attention Is All You Need. arXiv:1706.03762. https://arxiv.org/abs/1706.03762
• Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need (MQA). arXiv:1911.02150. https://arxiv.org/abs/1911.02150
• Ainslie, J. et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245. https://arxiv.org/abs/2305.13245
• DeepSeek-AI (2024). DeepSeek-V2 (MLA). arXiv:2405.04434. https://arxiv.org/abs/2405.04434
• Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. https://arxiv.org/abs/2002.05202
• Fedus, W., Zoph, B., Shazeer, N. (2021). Switch Transformers. arXiv:2101.03961. https://arxiv.org/abs/2101.03961
• Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer. arXiv:1701.06538. https://arxiv.org/abs/1701.06538
• Xiong, R. et al. (2020). On Layer Normalization in the Transformer Architecture. arXiv:2002.04745. https://arxiv.org/abs/2002.04745
• Zhang, B. & Sennrich, R. (2019). Root Mean Square Layer Normalization. arXiv:1910.07467. https://arxiv.org/abs/1910.07467
• Ba, J., Kiros, J., Hinton, G. (2016). Layer Normalization. arXiv:1607.06450. https://arxiv.org/abs/1607.06450
• Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864. https://arxiv.org/abs/2104.09864
• Press, O., Smith, N., Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases (ALiBi). arXiv:2108.12409. https://arxiv.org/abs/2108.12409
• Kazemnejad, A. et al. (2023). The Impact of Positional Encoding on Length Generalization in Transformers (NoPE). arXiv:2305.19466. https://arxiv.org/abs/2305.19466
• Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. https://transformer-circuits.pub/2021/framework/
• Wang, T. et al. (2022). What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? arXiv:2204.05832. https://arxiv.org/abs/2204.05832
• Dao, T. et al. (2022). FlashAttention. arXiv:2205.14135. https://arxiv.org/abs/2205.14135
• Dao, T. (2023). FlashAttention-2. arXiv:2307.08691. https://arxiv.org/abs/2307.08691

Textbooks / courses

• Phuong & Hutter (2022). Formal Algorithms for Transformers. arXiv:2207.09238. https://arxiv.org/abs/2207.09238 — implement every algorithm in the paper.
• Alammar, J. The Illustrated Transformer. https://jalammar.github.io/illustrated-transformer/
• Harvard NLP, The Annotated Transformer. https://nlp.seas.harvard.edu/annotated-transformer/
• Prince, S. Understanding Deep Learning, Ch. 12. https://udlbook.github.io/udlbook/
• ARENA (Alignment / interp + transformers). https://www.arena.education/

Derivation exercises (non-negotiable)

1. Scaled dot-product. Let \(q,k\in\mathbb{R}^d\) have i.i.d. entries \(\mathcal{N}(0,1)\). Show \(\mathrm{Var}(q\cdot k)=d\). Conclude the \(\sqrt{d}\) scale. What happens to softmax entropy if you omit it?
2. Online softmax / numerically stable softmax. Derive \(m=\max_i z_i\), \(p_i=e^{z_i-m}/\sum e^{z_j-m}\). Then the two-pass / running-max form used by FlashAttention (Milakov & Gimelshein 2018 is the ancestor: Online normalizer calculation for softmax, arXiv:1805.02867, https://arxiv.org/abs/1805.02867).
3. Backprop through attention (preview of §D). Given \(S=QK^\top/\sqrt{d}\), \(P=\mathrm{softmax}(S)\), \(O=PV\), write \(\partial L/\partial Q,\partial L/\partial K,\partial L/\partial V\) in matrix form. You will be asked this.
4. RoPE. For a 2-D pair, write \(R_\theta=\begin{pmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{pmatrix}\). Prove the relative-position identity. Extend to \(d/2\) frequencies \(\theta_i=10000^{-2i/d}\).
5. GQA math. \(n_q\) query heads, \(n_{kv}\) KV heads. Cache size vs MHA: factor \(n_q/n_{kv}\). Quality story from Ainslie.
6. MLA. From DeepSeek-V2 §2.1: compressed latent \(c_{KV}\), decoupled RoPE. Bytes/token vs GQA-8 for a 128-head, \(d_h=128\) model.
7. Param count. For a decoder block: embeddings (tied or not), \(n_{\text{layers}}\times(4d^2\) for MHA \(W_Q,W_K,W_V,W_O\) + \(3d\cdot d_{ff}\) SwiGLU, or \(2d\cdot d_{ff}\) vanilla FFN) + norms. Reproduce Llama-7B ≈ 6.7B.
8. MoE. Top-1 vs top-2 routing. Auxiliary load-balancing loss (Switch). Expert-parallel all-to-all volume: \(2\times\text{tokens}\times d\times \text{bytes}\) per MoE layer (send activations, receive outputs).

Coding labs

• Lab C1. Implement a GPT-2-small from scratch (no nn.MultiheadAttention): attention, GQA, RMSNorm, SwiGLU, RoPE. Match HuggingFace logits to 1e-4 on a fixed prompt.
• Lab C2. Implement FlashAttention forward in PyTorch (blockwise online softmax) and check equality with naive attention. Then read the CUDA of Dao-AILab/flash-attention until you can explain the SRAM tiling.
• Lab C3. Train a 2-layer attention-only transformer on tiny Shakespeare. Find an induction head (see §G).
• Lab C4. Ablate Post-LN vs Pre-LN at 6 layers / no warmup. Plot gradient norms by depth.

Whiteboard

• “Derive attention. Now give me FLOPs for training a token. Now give me bytes moved if K,V are in HBM. When are you compute-bound vs memory-bound?”
• “Why Pre-LN? What did Xiong actually prove?”
• “RoPE vs ALiBi vs NoPE for a 1M-context bet. Pick one and defend the kernel implications.”
• “Decoder-only won. Wang et al. 2022 said encoder-decoder wins after multitask FT. Reconcile this with GPT-4/Claude being decoder-only.”
• “Draw MLA. Why can’t you just fold RoPE into the latent?”

D. Training

~180–220 deep hours

Thesis

Training is backprop + an adaptive optimizer + a data pipeline + a parallelism plan + a numerical contract. At 100B+ the contract breaks in characteristic ways (attention entropy collapse, logit drift, loss spikes). A research engineer who cannot compute communication volume for DP/TP/PP/SP/CP/EP will ship a cluster that idles.

Learning outcomes

You can:

1. Derive backprop through attention (full matrix calculus) and through RMSNorm, SwiGLU, RoPE.
2. Write AdamW; explain why L2 ≠ weight decay under Adam; write \(\mu\)P scaling rules for LR / init / output.
3. Explain fp16 loss scaling, bf16’s exponent gift, and why NaNs appear (overflow, 0·Inf in softmax, Adam v-hat underflow).
4. Design a pretraining mixture; explain packing, document attention masking, and why naive concat leaks.
5. For each parallelism: when, what is sharded, collective, bytes on the wire per step.
6. Recite Kaplan and Chinchilla; discuss downstream scaling and overtraining.
7. Name 100B+ instabilities and the standard mitigations (QK-LayerNorm, σReparam, z-loss, warmup, μP).

Must-read

Optimizers & parametrization

• Kingma, D. & Ba, J. (2015). Adam. arXiv:1412.6980. https://arxiv.org/abs/1412.6980
• Loshchilov, I. & Hutter, F. (2019). Decoupled Weight Decay Regularization (AdamW). arXiv:1711.05101. https://arxiv.org/abs/1711.05101
• Yang, G. et al. (2021). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μP). arXiv:2203.03466. https://arxiv.org/abs/2203.03466

Numerics in training

• Micikevicius, P. et al. (2018). Mixed Precision Training. arXiv:1710.03740. https://arxiv.org/abs/1710.03740
• Micikevicius, P. et al. (2022). FP8 Formats for Deep Learning. arXiv:2209.05433. https://arxiv.org/abs/2209.05433

Parallelism

• Shoeybi, M. et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053. https://arxiv.org/abs/1909.08053
• Narayanan, D. et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. arXiv:2104.04473. https://arxiv.org/abs/2104.04473
• Korthikanti, V. et al. (2022). Reducing Activation Recomputation in Large Transformer Models (sequence parallelism). arXiv:2205.05198. https://arxiv.org/abs/2205.05198
• Rajbhandari, S. et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054. https://arxiv.org/abs/1910.02054
• Huang, Y. et al. (2019). GPipe. arXiv:1811.06965. https://arxiv.org/abs/1811.06965
• Liu, H., Zaharia, M., Abbeel, P. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889. https://arxiv.org/abs/2310.01889

Instabilities

• Zhai, S. et al. (2023). Stabilizing Transformer Training by Preventing Attention Entropy Collapse. arXiv:2303.06296. https://arxiv.org/abs/2303.06296
• Wortsman, M. et al. (2023). Small-scale proxies for large-scale Transformer training instabilities. arXiv:2309.14322. https://arxiv.org/abs/2309.14322
• Dehghani, M. et al. (2023). Scaling Vision Transformers to 22 Billion Parameters (QK-LayerNorm). arXiv:2302.05442. https://arxiv.org/abs/2302.05442
• Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways (logit drift / z-loss). arXiv:2204.02311. https://arxiv.org/abs/2204.02311

Data

• Gao, L. et al. (2020). The Pile. arXiv:2101.00027. https://arxiv.org/abs/2101.00027
• Touvron, H. et al. (2023). LLaMA. arXiv:2302.13971. https://arxiv.org/abs/2302.13971
• Touvron, H. et al. (2023). Llama 2. arXiv:2307.09288. https://arxiv.org/abs/2307.09288

Parallelism cheat sheet (memorize)

Rules of thumb used in production (2024–2026): TP ≤ GPUs/node (NVLink). PP to hide inter-node latency; microbatches \(m\gtrsim 4p\) to kill bubble. DP/ZeRO for remaining replicas. CP when \(s\) is 32k–1M. EP degree ≈ #experts, with capacity factor ~1–2.

Derivation exercises

1. Attention backward (full). From \(S=QK^\top/\sqrt{d}\), \(P=\mathrm{softmax}(S)\), \(O=PV\):

• \(\nabla_V=P^\top\nabla_O\)
• \(\nabla_P=\nabla_O V^\top\)
• \(\nabla_S = (P\odot(\nabla_P - \mathbf{1}(P\odot\nabla_P)\mathbf{1}^\top))\) (or the equivalent \(P\odot(\nabla_P - \mathrm{rowsums})\))
• \(\nabla_Q=\nabla_S K/\sqrt{d}\), \(\nabla_K=\nabla_S^\top Q/\sqrt{d}\) Derive every line. Then write the FlashAttention backward sketch (recompute \(P\) tiles from stored max+rowsum).

1. AdamW vs Adam+L2. Write both updates. Show L2 is rescaled by \(\sqrt{\hat v}\).
2. μP. For a linear layer width \(n\), state init std, LR, and activation multiplier so that feature updates stay \(O(1)\) as \(n\to\infty\). Why the output layer LR shrinks.
3. Loss scaling. fp16 max ≈ 65504. Grad of 1e-8 underflows. Show a scale of \(2^{10}\)–\(2^{16}\) and the unscale-before-Adam step. Why bf16 usually needs no loss scale.
4. PP bubble. 1F1B schedule: bubble fraction \((p-1)/(m+p-1)\). Interleaved schedule (Narayanan): why it cuts bubble ~2×.
5. Entropy collapse. Softmax entropy \(H(p)\ge \log n - \|z\|_\infty + \log(\cdots)\). Relate growing \(\|W_Q W_K^\top\|_2\) to one-hot attention (Zhai).
6. Z-loss (PaLM). \(L_z=\lambda\,\mathrm{mean}(\log^2 Z)\). Why this stops logit drift from making softmax unstable.

Coding labs

• Lab D1. Autodiff by hand for one attention head (numpy). Check vs torch.autograd to 1e-5.
• Lab D2. Implement AdamW. Train the same tiny GPT with Adam+L2 vs AdamW; plot weight norms.
• Lab D3. Implement μP on a width-mutable GPT. Tune LR at width 256; transfer to 1024. Compare to standard parametrization (it should fail).
• Lab D4. Mixed precision: force fp16 without loss scaling until it NaNs. Then add dynamic loss scaling. Then switch to bf16 and remove scaling.
• Lab D5. Packing: write a document-aware packer + intra-doc causal mask. Show the CE leak if you omit the mask (next-doc tokens attend to previous doc).
• Lab D6 (systems). With 2–8 GPUs: implement naive DDP AllReduce; then a Megatron-style column/row TP pair on one linear+GELU+linear; measure busbw. Optional: one PP stage boundary.

Whiteboard

• “Backprop through softmax-attention. Now. I will interrupt.”
• “You have 256 H100s, 70B dense, seq 8k, global batch 4M tokens. Pick TP/PP/DP/SP. Estimate MFU.”
• “Loss spiked at 80B. Attention entropy on 3 heads is 0.01. What do you do this afternoon?”
• “Why is ZeRO-3 not a substitute for TP?”
• “μP says you can transfer LR. Name two HPs that don’t transfer and why.”

E. Post-training (Anthropic-critical)

~130–160 deep hours

Thesis

Pretraining grows a prior over text. Post-training selects a policy. Anthropic’s public stack is: SFT on demonstrations → preference model → RL (PPO-like) and/or RLAIF under a constitution, with HHH as the operationalization of “aligned.” The failure modes — over-optimization, sycophancy, jailbreaks — are Goodhart problems in a sociotechnical loop, not “the model is evil.” Kernel people should care because RL sampling, RM forward passes, and long-context many-shot attacks are inference-and-training systems problems.

Learning outcomes

You can:

1. Write the RLHF objective \(\max_\pi \mathbb{E}[r(x,y)]-\beta D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{SFT}})\).
2. Derive DPO from that objective + Bradley–Terry: the closed-form reward \(r(x,y)=\beta\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)}\) and the logistic loss.
3. Describe Constitutional AI as a procedure (supervised critique-revise, then RLAIF preference labels, then RL), not a slogan.
4. Explain reward-model overoptimization (Gao): gold reward vs \(\sqrt{\mathrm{KL}}\).
5. Distinguish outcome vs process rewards; say why process supervision is an Anthropic-shaped bet (scalable oversight / “let’s verify”).
6. Treat sycophancy and jailbreaks as incentive + context + eval failures.

Must-read (read these, not blog summaries)

The RLHF spine

• Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741. https://arxiv.org/abs/1706.03741
• Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. https://arxiv.org/abs/1707.06347
• Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback (InstructGPT). arXiv:2203.02155. https://arxiv.org/abs/2203.02155
• Bai, Y. et al. (2022). Training a Helpful and Harmless Assistant with RLHF. arXiv:2204.05862. https://arxiv.org/abs/2204.05862
• Askell, A. et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861. https://arxiv.org/abs/2112.00861

CAI / RLAIF

• Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. https://arxiv.org/abs/2212.08073 Also: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback Code/data note in paper: https://github.com/anthropics/ConstitutionalHarmlessnessPaper
• Lee, H. et al. (2023). RLAIF vs. RLHF. arXiv:2309.00267. https://arxiv.org/abs/2309.00267

DPO and cousins

• Rafailov, R. et al. (2023). Direct Preference Optimization. arXiv:2305.18290. https://arxiv.org/abs/2305.18290

Rewards, process, Goodhart

• Gao, L., Schulman, J., Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760. https://arxiv.org/abs/2210.10760
• Lightman, H. et al. (2023). Let’s Verify Step by Step. arXiv:2305.20050. https://arxiv.org/abs/2305.20050
• Uesato, J. et al. (2022). Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275. https://arxiv.org/abs/2211.14275

Systems failure modes

• Perez, E. et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251. https://arxiv.org/abs/2212.09251
• Sharma, M. et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548. https://arxiv.org/abs/2310.13548
• Anil, C. et al. (2024). Many-shot Jailbreaking. Anthropic. https://www.anthropic.com/research/many-shot-jailbreaking PDF: https://www-cdn.anthropic.com/af5633c94ed2beb282f6a53c595eb437e8e7b630/Many_Shot_Jailbreaking_2024.04.02_0936.pdf
• Anthropic (2024). Alignment faking in large language models. https://www.anthropic.com/research/alignment-faking

Constitutional AI — the actual procedure (you must be able to recite this)

From Bai et al. 2022, two stages:

Stage 1 — Supervised CAI

1. Sample responses from a helpful-but-not-yet-harmless model, including on adversarial / harmful prompts.
2. Ask the same model to critique the response against a randomly chosen constitutional principle.
3. Ask it to revise the response in light of the critique.
4. Fine-tune the original model on the revised responses (SL). Multiple rounds of critique-revise are used. Chain-of-thought critiques help.

Stage 2 — RLAIF

1. Sample two responses from the SL-CAI model.
2. Ask a model to prefer one, given a constitutional principle (and often CoT).
3. Fit a preference model on these AI-labeled pairs (harmlessness). Mix with human helpfulness labels as needed.
4. RL (PPO-style) against that PM + KL to the SL-CAI policy.

Human labels for harmlessness can be near-zero. The constitution is the human bottleneck. Read the paper’s appendix principles; read Claude’s published constitution.

Derivation exercises

1. Bradley–Terry → DPO. Start from \(p(y_w\succ y_l\mid x)=\sigma(r_w-r_l)\). Impose \(r=\beta\log\pi/\pi_{\mathrm{ref}}+Z(x)\). Cancel \(Z\). Arrive at \(\mathcal{L}_{\mathrm{DPO}}=-\log\sigma(\beta\log\frac{\pi_w}{\pi_{\mathrm{ref},w}}-\beta\log\frac{\pi_l}{\pi_{\mathrm{ref},l}})\).
2. PPO-clip for a causal LM. Write the token-level advantage (GAE), the clipped ratio, the value loss, the KL/reward-model shaping \(r_t = r_{\mathrm{RM}}\cdot\mathbb{1}[t=T]-\beta\log\frac{\pi}{\pi_{\mathrm{ref}}}\).
3. Gao overopt. Define \(d=\sqrt{D_{\mathrm{KL}}(\pi\|\pi_0)}\). Sketch \(R_{\mathrm{gold}}(d)\) for BoN vs RL. Why bigger RMs delay the peak.
4. CAI label generation. Write the exact prompt pattern: principle + two responses + CoT + preference token. Where does position bias enter, and how does the paper swap order?

Coding labs

• Lab E1. SFT a small model on a tiny instruction set. Measure win-rate vs base with a held-out RM or human-ish heuristic.
• Lab E2. Implement DPO (the loss is ~30 lines). Train on Anthropic HH (Anthropic/hh-rlhf). Plot implicit reward margin vs KL-to-SFT.
• Lab E3. Implement Best-of-N with a small RM. Reproduce a toy Gao curve: proxy reward up, gold (a bigger RM or held-out human labels) down.
• Lab E4. Reproduce CAI Stage 1 on a harmless toy domain (e.g. “don’t give medical dosage advice”): critique → revise → SFT. No need for a frontier model.
• Lab E5 (evals, not attacks). Read Perez / Sharma. Build a 50-item sycophancy eval (user states a false preference; model should not flip facts). This is measurement, not jailbreak how-to.

Whiteboard

• “Derive DPO in 10 minutes. When does it underperform PPO?”
• “Walk me through Constitutional AI as if I have to implement it next quarter. Where do the GPUs go?”
• “Reward goes up, users say the model got worse. Diagnose.”
• “Sycophancy: is it pretraining, RLHF, or the preference data? Cite Sharma and Perez.”
• “Many-shot jailbreaking is ICL. What does that imply for long-context product decisions and for eval design?”
• “Process vs outcome rewards. Give a math problem where outcome RM is fooled and a PRM is not.”

F. Inference

~110–130 deep hours

Thesis

Autoregressive inference is an HBM-bandwidth problem disguised as a math problem. Prefill is compute-bound; decode is memory-bound on the KV cache. Every major serving trick — GQA/MLA, PagedAttention, continuous batching, speculative decoding, quantization — attacks bytes moved per generated token or reduces serial steps.

Learning outcomes

You can:

1. Write KV-cache size: \(2\cdot n_{\text{layers}}\cdot n_{kv}\cdot d_h\cdot s\cdot b\cdot \mathrm{sizeof}\). Convert to GB. Predict when you OOM.
2. Explain prefill vs decode roofline; why batching decode is mandatory.
3. Explain continuous / iteration-level batching (Orca) vs static batching.
4. Derive speculative decoding’s exact acceptance rule (Leviathan); expected speedup \(1+\sum p(\text{accept first }k)\).
5. State what quantization can drop: weights (GPTQ/AWQ) vs KV (fp8/int8) vs activations; when it becomes a quality cliff.
6. Explain RoPE interpolation, NTK-aware, YaRN (NTK-by-parts + attention temperature), and recency biases (ALiBi).

Must-read

• Shazeer (2019) MQA — already in §C.
• Ainslie (2023) GQA — already in §C.
• Yu, G.-I. et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI ’22. https://www.usenix.org/conference/osdi22/presentation/yu PDF: https://www.usenix.org/system/files/osdi22-yu.pdf
• Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). arXiv:2309.06180. https://arxiv.org/abs/2309.06180
• Leviathan, Y., Kalman, M., Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192. https://arxiv.org/abs/2211.17192
• Chen, C. et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318. https://arxiv.org/abs/2302.01318
• Frantar, E. et al. (2022). GPTQ. arXiv:2210.17323. https://arxiv.org/abs/2210.17323
• Lin, J. et al. (2023). AWQ: Activation-aware Weight Quantization. arXiv:2306.00978. https://arxiv.org/abs/2306.00978
• Xiao, G. et al. (2022). SmoothQuant. arXiv:2211.10438. https://arxiv.org/abs/2211.10438
• Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. arXiv:2309.00071. https://arxiv.org/abs/2309.00071
• Chen, S. et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation. arXiv:2306.15595. https://arxiv.org/abs/2306.15595
• Dao FlashAttention / FlashAttention-2 — already in §C.

Derivation exercises

1. Cache math. 70B, 80 layers, 64 q-heads, 8 kv-heads, \(d_h=128\), bf16, batch 32, 8k context. GB of KV. Now 128k context. Now MLA with \(d_c=512\).
2. Roofline. Decode: bytes = read all weights + read KV. FLOPs ≈ \(2N\) per token (N = params) plus attention. On H100 (3 TB/s, 990 TFLOP/s bf16), show decode of a 70B is memory-bound at batch 1 and becomes compute-bound at large batch.
3. Speculative sampling. Draft \(q\), target \(p\). Accept token \(x\sim q\) with \(\min(1,p(x)/q(x))\); on reject, sample from \(\mathrm{norm}(\max(0,p-q))\). Prove the output distribution is exactly \(p\).
4. YaRN. Write the wavelength-partitioned scale \(s\), the ramp on mid-frequencies, and \(\sqrt{1/t}=0.1\ln s+1\). Why low-frequency dims must be interpolated, not extrapolated.
5. PagedAttention. Internal fragmentation ≤ one block; external = 0. Why this raises max batch (and thus decode utilization).

Coding labs

• Lab F1. Implement KV-cached decode for your C1 model. Time prefill vs decode. Plot tokens/s vs batch.
• Lab F2. Tiny speculative decoder: a 4-layer draft + 12-layer target. Measure acceptance rate and wall-clock. Verify token distribution ≈ target by χ² on a toy vocab.
• Lab F3. Int8-quantize Linear weights (per-channel). Measure CE degradation. Then implement a smooth migration of activation outliers (SmoothQuant idea) on one layer.
• Lab F4. Read vLLM’s block manager well enough to draw the logical→physical block table and copy-on-write for a beam of 2.

Whiteboard

• “How big is the KV cache? Don’t look it up.”
• “Why doesn’t tensor-parallel help decode latency at batch 1 as much as you hoped?”
• “Speculative decoding changes nothing about the distribution. Prove it.”
• “What do you refuse to quantize to 4-bit, and why?” (attention KV? layernorm scales? embeddings? first/last layers?)
• “User wants 1M context. Do you buy more HBM, switch to MLA, use CP, or change the product?”

G. Interpretability (enough to collaborate)

~80–100 deep hours

Thesis

Models implement algorithms in superposition on the residual stream. Neurons are the wrong basis. Features (directions) and circuits (sparse computational graphs) are the current best ontology. SAEs are a dictionary-learning attack on superposition. Kernel / systems people should care because interp at scale is a data-plane problem: hooks, activation dumps, sparse matmuls, and training SAEs on trillions of activation tokens.

Learning outcomes

You can:

1. Explain linear representation + superposition; draw Elhage’s phase diagram (importance × sparsity).
2. Write an SAE: \(z=\mathrm{ReLU}(W_e(x-b)+b')\), \(\hat x=W_d z+b\), loss \(\|x-\hat x\|_2^2+\lambda\|z\|_1\), unit-norm decoder columns.
3. Walk through the induction-head circuit (previous-token head + induction head via K-composition).
4. Explain QK / OV circuits from Mathematical Framework.
5. Say what you would instrument in a training stack to make interp possible (residual hooks, activation sharding, replay buffers).

Must-read

• Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. https://transformer-circuits.pub/2021/framework/
• Olsson, C. et al. (2022). In-context Learning and Induction Heads. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads also arXiv:2209.11895 https://arxiv.org/abs/2209.11895
• Elhage, N. et al. (2022). Toy Models of Superposition. https://transformer-circuits.pub/2022/toy_model
• Bricken, T. et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. https://transformer-circuits.pub/2023/monosemantic-features
• Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. https://transformer-circuits.pub/2024/scaling-monosemanticity/
• Nanda, N. et al. (2023). Progress Measures for Grokking via Mechanistic Interpretability. arXiv:2301.05217. https://arxiv.org/abs/2301.05217
• Olah, C. et al. (2020). Zoom In: An Introduction to Circuits. Distill. https://distill.pub/2020/circuits/zoom-in/

Courses / tooling

• ARENA mech-interp track. https://www.arena.education/
• Neel Nanda, Concrete Steps to Get Started in Transformer Mechanistic Interpretability. https://www.neelnanda.io/mechanistic-interpretability/getting-started
• TransformerLens. https://github.com/TransformerLensOrg/TransformerLens
• SAELens. https://github.com/jbloomAus/SAELens

Derivation / pen-paper

1. Superposition toy. \(n\) features, \(m<n\) dimensions, sparsity \(s\). Write the interference term. When is a feature allocated its own dimension?
2. Induction. On the sequence [A][B] … [A], write what the layer-1 previous-token head writes and what the layer-2 head’s keys match. This is K-composition.
3. SAE scaling. From Scaling Monosemanticity: why residual-stream (not MLP) mid-layer; why L1 on activations + unit-norm decoder; dead-feature problem.

Coding labs

• Lab G1. Reproduce a superposition toy (5–20 lines of torch + a phase diagram).
• Lab G2. TransformerLens on GPT-2-small: find an induction head with the prefix-matching score. Ablate it; show ICL loss rises.
• Lab G3. Train a small SAE on a 1-layer transformer's MLP activations (follow Towards Monosemanticity appendix). Browse features. Expect mostly garbage plus a few real ones — that’s the lesson.
• Lab G4 (kernel-adjacent). Write a hook that dumps residual stream tiles to disk without stalling training (async copy, downcast to bf16, sequence-parallel aware). Estimate TB/day at 70B, 1M tokens/batch, every 100 steps, 1 layer.

Whiteboard

• “What is superposition? Why does it make interp hard? Why might it be computationally necessary?”
• “Explain induction heads to a kernel engineer who has never heard of them. Then tell them what activation they should dump.”
• “You have 10k H100-hours. SAE on a mid-size model or another pretrain? Anthropic chose both. How do you staff the systems side?”

H. Numerics & “the metal”

~100–130 deep hours

Thesis

The model is a IEEE-754 program. Formats are contracts on range vs precision. Accumulators are where correctness lives. Softmax is a numerically hostile reduction. Determinism is a lie unless you pay for it. This section is what separates “I implemented attention” from “I can ship a 100B train that does not NaN on Friday night.”

Learning outcomes

You can:

1. Draw the bit layouts of fp32, tf32, bf16, fp16, fp8 E4M3, fp8 E5M2. Give max finite, ULP at 1.0, and typical use.
2. Explain why GEMM accumulates in fp32 even when inputs are bf16/fp8; what TF32 tensor cores actually do (10-bit mantissa).
3. Implement stable softmax and online softmax; know FlashAttention’s stored statistics \((m, \ell)\).
4. List sources of nondeterminism: atomic adds, cuDNN heuristics, dropout, DP reduction order, TF32, async collectives. Know how to force determinism and the speed cost.
5. Debug a NaN: inf in exp, 0·inf, Adam \(\sqrt{v+\epsilon}\) of a leftover Inf, bf16 overflow of logits, fp16 grad underflow.

Must-read

• Micikevicius et al. (2018) Mixed Precision Training — §D.
• Micikevicius et al. (2022) FP8 Formats for Deep Learning — §D.
• Milakov, M. & Gimelshein, N. (2018). Online normalizer calculation for softmax. arXiv:1805.02867. https://arxiv.org/abs/1805.02867
• Dao et al. FlashAttention — IO complexity analysis is required, not optional.
• NVIDIA, Deep Learning Performance Guide (current). https://docs.nvidia.com/deeplearning/performance/index.html
• NVIDIA, CUDA C++ Programming Guide, especially memory model, atomics, warp reductions. https://docs.nvidia.com/cuda/cuda-c-programming-guide/
• IEEE 754-2019 overview (any good notes; Goldberg 1991 What Every Computer Scientist Should Know About Floating-Point Arithmetic still holds): https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
• Kalamkar, D. et al. (2019). A Study of BFLOAT16 for Deep Learning Training. arXiv:1905.12322. https://arxiv.org/abs/1905.12322

Format table (memorize)

Accumulator rule: MMA of low-precision inputs → fp32 accumulator → downcast to store. Softmax: accumulate exp-sum in at least fp32. Residual adds: prefer fp32 residual or careful bf16 (Pre-LN helps). Adam: keep moments in fp32.

Derivation exercises

1. ULP. Compute ULP(1.0) for each format. How many distinct bf16 numbers in \([1,2)\)? Implication for weight updates of size \(10^{-4}\).
2. Softmax overflow. \(z_i=100\) in fp16. \(e^{100}\) overflows. Show the max-subtraction fix. Show a case where all \(z_i\) are −80 and naive exp underflows to a 0/0 NaN.
3. Online softmax associativity. Prove you can merge two blocks \((m_1,\ell_1,O_1)\), \((m_2,\ell_2,O_2)\) into a correct \((m,\ell,O)\). This is FlashAttention.
4. TF32. A matmul in tf32 vs fp32: estimate a relative error. When does a long reduction (e.g. attention over 128k) accumulate enough error to change argmax?
5. Determinism. AllReduce in tree vs ring: write two summation orders of 8 terms. Bound the fp32 difference.

Coding labs

• Lab H1. Print bit patterns: take 1.0, 1+ulp, Inf, NaN for fp16/bf16. Write a decoder.
• Lab H2. Implement GEMM three ways: fp32, bf16-in-fp32-acc, fp16-in-fp16-acc. Compare residual of \(AA^\top\). The last one should scare you.
• Lab H3. Break softmax on purpose (large logits, all-very-negative, causal mask filled with 0 instead of -inf). Fix each.
• Lab H4. CUDA (or Triton): write a fused RMSNorm+Linear or online-softmax kernel. Profile with Nsight: achieved bandwidth, occupancy, bank conflicts.
• Lab H5. Turn on torch.use_deterministic_algorithms(True) for a 2-GPU train. Measure slowdown. Find the op that broke.

Whiteboard

• “bf16 vs fp16. Don’t say ‘more range.’ Draw the bits. Now tell me why Adam moments stay fp32.”
• “Your loss is NaN at step 12k, only on TP=8, not TP=1. Walk the debug.”
• “FlashAttention claims exact attention. Exact in which algebra?”
• “Can we train in fp8 end-to-end? What tensors stay in higher precision?”
• “Reproducibility: the paper reviewer wants bitwise-identical runs. What do you tell them?”

Cross-cutting: “what kernel work has to do with AGI”

Keep this paragraph in your head; it is the interview’s connective tissue.

Scaling laws say loss (and, more noisily, capability) is a smooth function of compute, data, and parameters. Every 10–20% you save on step time, activation memory, or decode bandwidth is a larger N, larger D, or more RL/SAE/eval tokens at fixed dollar/carbon budget. Constitutional AI, process rewards, and monosemantic SAEs are hungry for that surplus. A kernel that makes attention 1.5× at 128k context is not a compiler curiosity; it is what makes “many-shot evals,” long-horizon agents, and activation logging at scale affordable. Conversely, an unstable softmax at 100B wastes a cluster-week and produces no science. Anthropic-shaped research engineers are people who can move fluently from “what is the HHH tradeoff on this prompt” to “the accumulator is tf32 and the reduction order changed.”

Suggested 9-month path (20–25 h/week)

Do not consume all papers linearly. For each section: skim the must-reads → do the derivations → do the labs → re-read the papers. The second reading is where you actually learn.

Interview-day checklist (Anthropic RE / kernel-adjacent)

You should be able to do all of these on a whiteboard in one day:

1. \(\mathrm{softmax}(QK^\top/\sqrt{d})V\) + complexity + \(\sqrt{d}\) variance argument.
2. Backprop through attention, matrix form.
3. KV-cache bytes; GQA vs MQA vs MLA.
4. AdamW update; why not L2.
5. DP AllReduce volume; TP AllReduce volume; PP bubble.
6. FlashAttention: tiling, online softmax merge, what’s stored for backward.
7. bf16 vs fp16 vs fp8 E4M3/E5M2; where you accumulate.
8. DPO loss from RLHF+BT in ~10 lines.
9. Constitutional AI’s two stages, concretely.
10. One instability: entropy collapse or logit drift, plus the fix.
11. One bitter-lesson / scaling-law question without becoming a TED talk.
12. One interp object (induction head or SAE loss) well enough to collaborate.

If you miss more than two of these cold, you are not ready.

Resources index (courses & books, not already listed)

• Stanford CS224N — https://web.stanford.edu/class/cs224n/
• Stanford CS25 (Transformers United) — https://web.stanford.edu/class/cs25/
• MIT 6.S191 — https://introtodeeplearning.com/
• Karpathy nanoGPT / Zero to Hero — https://github.com/karpathy/nanoGPT
• EleutherAI GPT-NeoX (real train stack to read) — https://github.com/EleutherAI/gpt-neox
• NVIDIA Megatron-LM — https://github.com/NVIDIA/Megatron-LM
• Dao-AILab flash-attention — https://github.com/Dao-AILab/flash-attention
• vLLM — https://github.com/vllm-project/vllm
• Microsoft μP — https://github.com/microsoft/mup
• Goodfellow, Bengio, Courville, Deep Learning (free) — https://www.deeplearningbook.org/
• Programming Massively Parallel Processors (Kirk & Hwu) — the CUDA textbook.

What this curriculum deliberately does not do

• It does not pretend a blog post replaces Vaswani, Kaplan, Bai, Rafailov, Dao, or Elhage.
• It does not treat “I fine-tuned Llama with QLoRA” as training knowledge.
• It does not give jailbreak recipes; it treats jailbreaks as eval and systems problems (Anil et al., product context limits, ICL).
• It does not pick a winner between DPO and PPO; it makes you derive both.
• It does not claim kernel work is sufficient for AGI. It claims it is on the critical path of the only research program that has empirically compounded.

If you actually do the labs and the derivations, you will be in the small set of people who can walk into an Anthropic research-engineer interview and treat “intelligence,” “RLAIF,” and “accumulator width” as one subject. That is the point.