Build 08 · Blocked
FlashAttention, by hand
Implement FlashAttention — online softmax, tiling, causal masking — on whatever hardware is available. This is the goal the doctrine came from.
Cost — The Blackwell-specific path needs hardware that may not be to hand. The algorithm does not.
Done when
A state a stranger could checkIt matches a reference output, beats the naive version on wall-clock, and every line can be explained without notes.
Why it is harder — The first build that will genuinely stall for days. That is the point of it.
Where it sits
After Build 07 · Attention, implemented, then made fast — You cannot make fast a thing you have not first made correct.
What it reads
Just enough to unblock the next step- GPU Kernels / MLSys Part IV. Parts II–III only if the build stalls on the DSL rather than the algorithm.
- Inference Engineering ch6For where FlashAttention sits among the other serving techniques.
What it ships
- BuildA FlashAttention kernel that beats the naive baseline
- NotebookThe kernel, explained from first principles
Teach it back
If it cannot be explained plainly, it was not learned- 1Explain itExplain what you built to a friend who has never seen it. Plain words, no jargon, five sentences at most.
- 2Predicted vs realWhat you expected to happen, what actually happened, and why the two differ.
- 3Where it breaksThe question about it you still cannot answer, and the point where your explanation stops working.
- 4KeepThe one number, rule or picture worth keeping. It becomes a bottle and gets re-tested.
No notes on this build yet.
A note is a Learning Log entry with the tag study-flash-attention. It is written in Connor's words, filed as a draft, and appears here once it is published from the admin queue. One note can carry several build tags.