Build 10 · Blocked
Train a model across devices, and account for every second
Compose data, tensor and pipeline parallelism into one working training run. Then account for where the wall-clock went.
Cost — Multi-GPU rental, hours at a time. Budget before starting.
Done when
A state a stranger could checkA model trains across devices to a target loss, and the time is accounted for between compute, communication and idle.
Why it is harder — The largest build on the ledger, and the first where the bottleneck is a system rather than a function.
Where it sits
After Build 09 · What each collective operation actually costs — Parallelism is collectives arranged in a pattern. Learn the primitive first.
What it reads
Just enough to unblock the next step- Building a distributed training framework from first principles ↗The parallelism dimensions, in build order.
What it ships
- BuildA working distributed training framework
- NotebookTraining FLOPs to training dollars, from a run you paid for
Teach it back
If it cannot be explained plainly, it was not learned- 1Explain itExplain what you built to a friend who has never seen it. Plain words, no jargon, five sentences at most.
- 2Predicted vs realWhat you expected to happen, what actually happened, and why the two differ.
- 3Where it breaksThe question about it you still cannot answer, and the point where your explanation stops working.
- 4KeepThe one number, rule or picture worth keeping. It becomes a bottle and gets re-tested.
No notes on this build yet.
A note is a Learning Log entry with the tag study-distributed-train. It is written in Connor's words, filed as a draft, and appears here once it is published from the admin queue. One note can carry several build tags.