Month 5 — FA3 / Hopper, FP8/MX, collectives, DeepSeek-V3 systems, weekly inference design

03-kernels/curriculum.md Layer 3 (FA3, FP8) + Layer 5 (NCCL playground). 04-llm-theory/curriculum.md §H + DeepSeek-V3. Roadmap P2.

Month goals

Read FA3 and write the three ideas: warp-specialized TMA overlap, GEMM/softmax interleave, FP8 block quant. If you have H100, one TMA load experiment. If not, docs + Colfax + FA3 paper; no fake TFLOPS.
FP8 / MX literacy: E4M3/E5M2 bits, OCP MX, MXFP8 recipes (arXiv:2506.08027), Micikevicius FP8. Where you accumulate.
Collectives: all-reduce = RS + AG. NCCL playground or a simulated ring. Busbw vs algobw if you have 2 GPUs.
DeepSeek-V3 systems appendix (not the whole model card tourism). MLA + EP + training numerics they actually used.
Inference system design weekly (45 min recorded). Prompt: "Design an inference API for a frontier chat model." Push yourself on the list in 05-interview/drills.md.

Take-home: Friday sport continues. Aim to see a path under 1,800.

Week 17 — FA3 read + Hopper features

Day	Deep 1	Deep 2	Paper / book
Mon	FA3 paper end-to-end. Notes: three ideas, % of peak claims, FP8 error claim.	Hopper tuning guide: TMA 1D–5D, mbarrier, cluster size.	https://arxiv.org/abs/2407.08608
Tue	If H100: TMA load of a 2D tile + mbarrier (`03-kernels` L1 ex. 7). Else: PTX ISA pages for `cp.async` / WGMMA; annotated note.	Your FA2 kernel: what FA3 would change (warp specialization you do not have).	Colfax Hopper GEMM / WGMMA (one tutorial).
Wed	Debug / reread. No new kernel scope.	Take-home.	None.
Thu	Inference design #1 (45 min recorded): prefill/decode split only.	KV numbers recap from month 4 in 10 min.	None.
Fri	Take-home sport.
Sat	Long build: `notes/fa3-three-ideas.md` good enough to teach.
Sun	Off-ish.

Week 18 — FP8 / MX literacy

Day	Deep 1	Deep 2	Paper / book
Mon	Bit layouts: fp32, tf32, bf16, fp16, E4M3, E5M2. Print them (`04-llm-theory` §H table).	Micikevicius FP8 2022.	https://arxiv.org/abs/2209.05433
Tue	OCP MX + MXFP8 recipes 2506.08027. What a block scale is.	Cast path: a tiny GEMM in fp16-in/fp32-acc vs a fake fp8 scale (even if the SKU cannot run MX). Residual vs f64.	Transformer Engine FP8 primer (docs).
Wed	Debug numerics: softmax in fp16 that NaNs; fix.	Take-home.	None.
Thu	Inference design #2: quantization + correctness eval (Inference Systems req: output regression is a perf bug).	Postmortem skim (full reread month 6): sampling dtype.	None.
Fri	Take-home.
Sat	Long build: `notes/formats.md` + one numerical table.
Sun	Off-ish.

Week 19 — Collectives

Day	Deep 1	Deep 2	Paper / book
Mon	Derive AR = RS + AG. Draw ring vs tree.	NCCL hello (2 GPU) or numpy simulated ranks with byte counts.	NCCL overview docs.
Tue	Overlap thought experiment: GEMM stream vs comm stream. When the tensor is too small.	TP linear from month 4: add a comm volume table that matches a trace or an honest "I only have 1 GPU" calculation.	Korthikanti / Megatron SP reminder.
Wed	Debug.	Take-home.	None.
Thu	Inference design #3: multi-tenant + one autoscaling signal that is not GPU util.	Ring Attention abstract (Liu 2023) — comm vs compute.	None.
Fri	Take-home.
Sat	Long build: collectives note + (if 2 GPU) busbw table.
Sun	Off-ish.

Week 20 — DeepSeek-V3 systems + close

Day	Deep 1	Deep 2	Paper / book
Mon	DeepSeek-V3 paper: systems / infra appendix and MLA recap.	Expert-parallel all-to-all volume for a named MoE layer.	https://arxiv.org/abs/2412.19437
Tue	DeepSeek hardware reflections 2505.09343 (read as taste, not gospel).	Grouped GEMM idea (Triton tutorial 08 if time).	https://arxiv.org/abs/2505.09343
Wed	Debug / write.	Take-home.	None.
Thu	Inference design #4: 1e5 concurrent generations. Prefill/decode, paging, streaming, failure. 2 pages after the recording.	Blank-page FA3 three ideas.	None.
Fri	Take-home. Path under 1,800 written if not hit.
Sat	Month 5 artifact.
Sun	Off.

Month 5 exit criteria

FA3 three ideas from memory. Hopper features named. No fake H100 numbers.
Format table you can draw. MX block-scale in one paragraph.
Collectives: AR=RS+AG, one volume number for TP MLP, overlap story.
DeepSeek-V3 systems appendix notes (MLA + EP at minimum).
Four recorded inference designs.
Take-home: number + next experiment.

Artifact

notes/fa3-three-ideas.md, notes/formats.md, collectives note, V3 systems notes, four design recordings (or transcripts), writeup. Portfolio now has "I can talk Hopper and FP8 without lying."