Build 09 · Blocked
What each collective operation actually costs
Implement the collective operations by hand — all-reduce, all-gather, reduce-scatter, all-to-all — then measure what each costs in time and bandwidth on real devices.
Cost — Needs more than one GPU — rented.
Done when
A state a stranger could checkA table of measured costs per collective exists, and predicted cost matches measured within a stated tolerance.
Why it is harder — First build spanning more than one device. The failure modes stop being logical and start being physical.
Where it sits
After Build 07 · Attention, implemented, then made fast — Distributing a model you have not built is distributing a black box.
Shipping it opens 10 Train a model across devices, and account for every second
What it reads
Just enough to unblock the next step- Building a distributed training framework from first principles ↗The process-groups and collectives portion. Watched WHILE building, not before.
What it ships
- NotebookThe number nobody publishes — communication cost per collective
- BuildThe collectives, measured
Teach it back
If it cannot be explained plainly, it was not learned- 1Explain itExplain what you built to a friend who has never seen it. Plain words, no jargon, five sentences at most.
- 2Predicted vs realWhat you expected to happen, what actually happened, and why the two differ.
- 3Where it breaksThe question about it you still cannot answer, and the point where your explanation stops working.
- 4KeepThe one number, rule or picture worth keeping. It becomes a bottle and gets re-tested.
No notes on this build yet.
A note is a Learning Log entry with the tag study-collectives-cost. It is written in Connor's words, filed as a draft, and appears here once it is published from the admin queue. One note can carry several build tags.