Month 3 — FA1 in Triton, Hopper intro, a tiny GPT that trains, CAI from memory

Continues month 2. Resources: 03-kernels/curriculum.md Layer 2–3 (GEMM already started; FA1 is the job this month), 04-llm-theory/curriculum.md §C + §E procedure, 04-llm-theory/high-to-low.md L2.

Month goals

Implement FlashAttention-1-style forward in Triton (tiled, online softmax, never materialize S). Correct vs F.scaled_dot_product_attention on random QKV (fp32 first, then fp16). Causal. Head dim 64 or 128.
Hopper intro from docs, even without an H100: TMA, WGMMA, 228 KB SMEM, clusters. 3-page note. (03-kernels L1 resources 5, 9.)
Tiny GPT trains to a real loss curve and samples. Not a notebook dump of Karpathy.
Constitutional AI flowchart from memory (SL-CAI + RLAIF). Recite, then check Bai 2022.
Take-home < 2,000.

Week 9 — FA1 forward, correctness first

Day	Deep 1	Deep 2	Paper / book
Mon	Triton FA1: Q-tile loop, KV-tile loop, online m/ℓ, O rescale. Tiny shapes only (S=128).	Unit test harness vs SDPA. First mismatch dump.	Dao 2022 reread algorithm only.
Tue	Causal mask without wrecking occupancy.	GQA path: repeat KV heads.	None.
Wed	Debug day. Numerical: fp32 vs fp16, row-max range 0 / 20 / 80 (`02-foundations` module 5 softmax lab).	Take-home.	Milakov & Gimelshein 2018 online softmax if the rescale is lying.
Thu	SRAM budget on paper: what tiles fit.	Tiny GPT: train loop clean enough someone else can run.	Hopper in-depth blog (read; do not pretend you ran TMA).
Fri	Take-home. Target < 2,500.
Sat	Long build: grow FA1 to S=1024, head=64, causal. Table of errors vs SDPA.
Sun	Off-ish.

Week 9 exit: FA1 forward exists and is correct on small tensors.

Week 10 — Make it a kernel, not a prototype

Day	Deep 1	Deep 2	Paper / book
Mon	Autotune BLOCK_M/N + num_warps. Heatmap (`03-kernels` L4 ex. 2 energy).	Hopper note: contrast `mma.sync` / `wgmma` / (later) `tcgen05`. Cite PTX ISA + Colfax, not vibes.	None.
Tue	Profile FA1. Name the limiter. Compare to `flash_attn` if you can install it; else vs SDPA math-path.	Tiny GPT: checkpoint + sample + val loss.	FlashDecoding blog (decode ≠ prefill).
Wed	Debug / correctness on more shapes (non-multiple-of-block S).	Take-home.	None.
Thu	Fused RMSNorm or RoPE if FA1 is waiting on compile.	Interview drill 60 min.	Bai 2022 CAI — write flowchart then open the paper.
Fri	Take-home. Target < 2,200.
Sat	Long build: public writeup draft of FA1 (tiles, SRAM, what broke).
Sun	Off-ish.

Week 10 exit: FA1 profiled; Hopper note exists; CAI flowchart exists.

Week 11 — GPT that trains + CAI as procedure

Day	Deep 1	Deep 2	Paper / book
Mon	Tiny GPT: finish train to a curve you would show. Tokens/sec logged.	KV-cache decode on this model (month 1 loop, now on the trained net).	None.
Tue	FA1: one more correctness or speed pass, not a rewrite.	DPO vs PPO contrast (`04-llm-theory` §E). Rafailov 2023 enough to derive the loss shape.	InstructGPT (Ouyang 2022) skim of the procedure.
Wed	Debug day.	Take-home.	None.
Thu	Blank-page: CAI two stages. Record 8 min voice memo.	Why-Anthropic 200 words, throw away if generic.	2026 constitution (https://www.anthropic.com/news/claude-new-constitution) — priority order only.
Fri	Take-home. Target < 2,000.
Sat	Long build: CAI Stage-1 toy (`04-llm-theory` Lab E4 energy) or improve FA1 writeup. Pick one.
Sun	Off-ish.

Week 11 exit: trained GPT + samples; CAI from memory; take-home approaching 2k.

Week 12 — Close month 3

Day	Deep 1	Deep 2	Paper / book
Mon	FA1 vs SDPA full harness; freeze the kernel.	Re-derive attention backward (prep for month 4 FA2).	FA2 paper: algorithm changes only (Dao 2023).
Tue	PMPP / CUDA leftover from month 2.	Interview: KV-cache bytes for a named 70B-class GQA config.	None.
Wed	Debug / measure.	Take-home polish.	None.
Thu	Blank-page exam: FA1 algorithm, CAI flowchart, Hopper three nouns (TMA, WGMMA, mbarrier).	Timed 90 min incremental system.	None.
Fri	Take-home. Official month target < 2,000.
Sat	Month 3 artifact.
Sun	Off.

Month 3 exit criteria

Triton FA1-style forward, causal, tested vs SDPA (rel 1e-4 fp32 / honest fp16 table).
Writeup: tiles, SRAM, online softmax rescale, first bug.
Hopper intro note (docs, not fake H100 numbers).
Tiny GPT trained, sampled, tokens/sec logged.
CAI two-stage flowchart from memory (checked once against Bai).
Take-home < 2,000.
FA2 algorithm-changes list written (implementation is month 4).

Artifact

Public repo: triton_fa1/ + tests + error table + Hopper note + train loss plot + CAI flowchart image or markdown + take-home cycles. 1000–1500 word Greenhouse-shaped paragraph update.

If FA1 is correct but slow, ship correct. Speed is month 4–5.