AKP Anthropic Kernel Path

0/21 read · no cycles yet

First 30 days — concrete, starting tomorrow

Assumption: strong generalist SWE, weak or rusty ML, little or no CUDA. 12-hour template from operating-system.md. If you are already a CUDA person, skip to Day 11 and replace Deep 1 with the take-home from day 1.

Books to have in hand before Day 1 (buy used, PDF if you legally have it):

  • Bryant & O'Hallaron, Computer Systems: A Programmer's Perspective (3e)
  • Programming Massively Parallel Processors (4e or 5e) — start later
  • A linear algebra book you will actually finish: Axler Linear Algebra Done Right if proof-weak, Trefethen & Bau if you already had a real course
  • A dedicated notebook (paper) for derivations. Laptop is for code.

Accounts: GitHub repo kernel-path (public), Anki, the Anthropic take-home cloned. 

git clone https://github.com/anthropics/original_performance_takehome

Do not try to beat 1487 this month. This month you learn to see the machine.

Week 1 — The machine, not the model

Goal: you can talk about bytes, cycles, and caches like you mean it. You have run the take-home baseline and read every line of the simulator you can without optimizing yet.

Day 1 — Baseline + contract

  • Deep 1: Clone take-home. Run python perf_takehome.py / submission tests. Confirm 147734 cycles. Read perf_takehome.py and the Machine simulator end to end. Draw the engines (ALU/VALU/Load/Store/Flow), scratch size, VLEN on paper.
  • Deep 2: CSAPP Ch 1 + start Ch 2 (representing information). Do the practice problems, not the reading.
  • Paper: Hume, Designing AI-resistant technical evaluations. Take notes on design goals, not vibes.
  • Practice: write a 200-word "why this path" for yourself. Not the application essay yet.
  • Artifact: notes/day01-machine.md with a diagram of the simulated machine.

Day 2 — Bits and the cost of a miss

  • Deep 1: CSAPP Ch 2 finish + data lab if you have the CSAPP labs (get them: http://csapp.cs.cmu.edu). Integer and float representation. You will need this for FP8 later.
  • Deep 2: Write a tiny C program that streams through arrays of different sizes and time it. Plot a crude memory-hierarchy curve on your machine.
  • Paper: none. Book only.
  • Practice: implement IEEE-754 encode/decode for fp16 by hand in Python, test against numpy.float16.
  • Artifact: the timing plot + explanation of where L1/L2/DRAM show up.

Day 3 — Machine-level view

  • Deep 1: CSAPP Ch 3 through condition codes and procedures. Read compiler output of your Day 2 program (gcc -O2 -S).
  • Deep 2: Take-home: implement only SIMD vectorization on the scalar kernel. Do not pack VLIW yet. Measure cycles. Write what broke.
  • Paper: none.
  • Practice: 60 min timed: implement a correct in-memory key-value store with GET/SET/DEL and a simple TTL (CodeSignal energy).
  • Artifact: cycle count after SIMD, vs 147734.

Day 4 — Memory hierarchy for real

  • Deep 1: CSAPP Ch 6 (caches). Work every "practice problem" on cache misses.
  • Deep 2: Take-home: hazard-aware VLIW packing of what you have. RAW/WAW. New cycle count.
  • Paper: Williams, Waterman, Patterson, Roofline (2009). You will use this weekly for two years.
  • Practice: given a 3-loop GEMM, compute arithmetic intensity and say whether a 4090 (or an H100) is math- or memory-bound for n=512, n=4096, n=8192. Show the arithmetic.
  • Artifact: roofline sketch in notes/roofline.md.

Day 5 — Friday take-home sport

  • Deep 1+2 combined (5h): take-home only. Goal this week is under 30,000 cycles with a written explanation of each trick. Correctness tests must pass.
  • Paper: none.
  • Practice: rest the interview muscle. Write the weekly log.
  • Artifact: public commit + cycle number.

Day 6 — Long build: bytes of a transformer

  • Deep 1+2: Implement a forward-only GPT-2-small block in numpy: RMSNorm, RoPE, attention, SwiGLU. No training. No PyTorch. Check shapes against a reference (Karpathy nanoGPT or a printed diagram).
  • Paper: Vaswani et al. 2017, Attention Is All You Need. You may skim the NMT experimental sections. You may not skim §3. You will derive Eq. 1 on paper with dimensions labeled.
  • Practice: cost the block: FLOPs and bytes for seq=1024, d=768, heads=12.
  • Artifact: from-scratch/block_numpy.py + a shape sheet.

Day 7 — Off-ish

Anki from the week's holes. Sleep. If you cannot stay off, reread your own notes, do not open a new paper.

Week 1 exit: CSAPP Ch 2/3/6 started, roofline in your bones, take-home < 30k, one numpy transformer block, Vaswani §3 derived.

Week 2 — Linear algebra that you can feel + attention math

Goal: you can derive attention and backprop-through-softmax without looking it up. You can talk about SVD as a tool, not a meme.

Day 8

  • Deep 1: Trefethen & Bau Lectures 1–4 or Axler Ch 1–2 (pick one book and stay). Problem sets, not highlights.
  • Deep 2: Finish numpy GPT block: add causal mask, multi-head, residual. Generate garbage tokens with a random + one learned embedding table if you want dopamine.
  • Paper: none.
  • Practice: whiteboard (phone camera, 25 min): write attention, label every matrix [B,H,S,D], compute FLOPs.

Day 9

  • Deep 1: Linear algebra continues (projections, norms).
  • Deep 2: Implement softmax three ways: naive, max-stabilized, online (Milakov / FlashAttention style running m and l). Prove they match on random tensors to 1e-5.
  • Paper: FlashAttention (Dao et al. 2022) through the algorithm box. Do not read the rest yet.
  • Practice: CodeSignal-style: implement online softmax in a timed 40 min.

Day 10

  • Deep 1: Linear algebra (eigen / SVD intro).
  • Deep 2: Take-home. Target < 15,000 cycles.
  • Paper: FlashAttention §2 IO analysis. Reproduce the IO complexity argument on paper.
  • Practice: system-design 45 min out loud: "design a batching loop for variable-length prefill." Record yourself.

Day 11

  • Deep 1: CSAPP Ch 5 (optimizing program performance) + Ch 4 start if you have steam.
  • Deep 2: Backprop through attention on paper, then in numpy for a tiny (B=2,S=8,D=16) case. Check against torch.autograd on the same tensors.
  • Paper: none.
  • Practice: timed coding 60 min.

Day 12 — Friday sport

  • Take-home 5h. Target < 10,000. Write a scheduler note: how you pack VALU vs Load.

Day 13 — Long build

  • Start from-scratch/gpt.py: train a 10–20M model on TinyStories or Shakespeare with AdamW, no framework magic beyond numpy or a thin torch for GPU later. This will take multiple Saturdays. Today: data loader, tokenizer decision (use a pretrained BPE, do not write BPE this month), train loop, loss plot.

Day 14 — Off-ish

Week 2 exit: online softmax you can write from memory, attention backward checked, FlashAttention IO argument reproduced, take-home < 10k, training repo born.

Week 3 — CUDA week 1 (or Triton if no NVIDIA on the box)

Goal: first real kernel. Not attention. SAXPY, then tiled GEMM.

If you have no NVIDIA GPU, do the CUDA C++ programming guide chapters and write kernels that you will run on a $0.30/hr cloud box at the end of the week. The take-home continues regardless.

Day 15

  • Deep 1: CUDA C++ Programming Guide: programming model, memory hierarchy. PMPP Ch 1–3 if you have the book.
  • Deep 2: Write and run SAXPY. Then a naive GEMM. Time it. Compute % of peak (even if peak is a laptop GPU).
  • Paper: none.
  • Practice: Anki from CUDA vocabulary (warp, SM, occupancy, coalescing).

Day 16

  • Deep 1: Tiled GEMM with shared memory. Bank conflicts. Measure.
  • Deep 2: CSAPP Ch 8 or Ch 9 (virtual memory) — 90 min, keep the systems brain alive.
  • Paper: none.
  • Practice: timed coding.

Day 17

  • Deep 1: Triton intro. Reimplement your tiled GEMM in Triton. Compare.
  • Deep 2: Take-home < 7,000 if you can.
  • Paper: OpenAI Triton paper (Tillet et al. 2019) as background, then current Triton docs for the language you will actually write.
  • Practice: design drill: KV cache memory for a 70B-class GQA model, 8k context, batch 32, fp16 vs fp8. Numbers on paper.

Day 18

  • Deep 1: Profile naive vs tiled vs Triton GEMM. Write the story of the counters you looked at (even if it is just ncu or torch.profiler).
  • Deep 2: numpy GPT training continues: get a loss curve that goes down.
  • Paper: Kaplan et al. 2020 scaling laws (the figure, the exponent, the caveats) + Hoffmann et al. 2022 Chinchilla (the 20-token rule as a claim, not a law of nature).
  • Practice: explain out loud why decode is memory-bound.

Day 19 — Friday sport

  • Take-home. Target < 5,000. If you stall, write a list scheduler on paper before touching code.

Day 20 — Long build

  • Fuse RMSNorm+residual in Triton (or CUDA). Unit test vs a PyTorch reference. This is a real PE-shaped kernel.

Day 21 — Off-ish

Week 3 exit: one GEMM you tiled, one Triton kernel, one fused norm, loss curve exists, take-home < 5k, you can cost a KV cache.

Week 4 — Attention as a kernel + Anthropic literacy

Goal: a correct (not fast) attention kernel, and you can explain Constitutional AI as a procedure.

Day 22

  • Deep 1: Implement standard attention in Triton (not Flash). Test vs torch.nn.functional.scaled_dot_product_attention on small tensors.
  • Deep 2: Linear algebra or CSAPP, whichever you dropped.
  • Paper: FlashAttention-2 (Dao 2023) algorithm changes only.

Day 23

  • Deep 1: Online-softmax attention kernel (the FlashAttention idea, even if not FA2-fast). Correctness first.
  • Deep 2: Take-home.
  • Paper: Bai et al. 2022, Constitutional AI. Read the actual procedure: principles, critique, revision, SL-CAI, RL-CAI. Write it as a flowchart from memory after.

Day 24

  • Deep 1: Backward of your attention kernel, or a written plan if the backward is too big this week. Do not fake a backward you did not test.
  • Deep 2: From-scratch GPT: checkpoint, sample, log val loss.
  • Paper: Christiano et al. 2017 RLHF + the DPO paper (Rafailov 2023) enough to contrast "reward model + RL" vs "direct preference."
  • Practice: 45 min "Why Anthropic?" draft. Be specific. Throw it away if it is generic.

Day 25

  • Deep 1: Inference literacy day. Implement a numpy/torch KV-cache decode loop for your tiny GPT. Measure the difference vs redoing prefill every token.
  • Deep 2: System design writeup: design Claude-like chat inference for 1e5 concurrent generations. Prefill/decode split, paging, batching. 2 pages.
  • Paper: Kwon et al. 2023 vLLM / PagedAttention.
  • Practice: timed coding 60 min.

Day 26 — Friday sport

  • Take-home. Stretch < 3,000. Write what you think the next 2× is.

Day 27 — Long build + month checkpoint

Ship Month 1 artifact:

  • Public repo with: take-home current cycle count and changelog, numpy/torch from-scratch trainer + loss plot, Triton GEMM + fused RMSNorm + attention (correct), KV-cache decode demo, 1000-word writeup.
  • Title the writeup as if it were the Greenhouse paragraph: "the most impressive low-level thing so far (month 1)."

Day 28 — Off

Day 29–30 — Review and patch

  • Re-derive attention and online softmax with a blank page.
  • Re-read Hume.
  • Fix the worst bug in the public repo.
  • Plan month 2: CUTLASS/CuTe or FA2 closer, CSAPP labs finish, start NCCL mental model.

Month 1 exit criteria (all required):

If more than two boxes are unchecked, repeat week 4. Do not "go on to MoE."

Next: 06-days/month-02.md (CSAPP finish, LA, PMPP/CUDA proper, FA through the algorithm, take-home < 3k). If this file and 05-interview/full-prep.md disagree, prefer this file.