Build 02 · Open
An evaluation you would bet on
Build a small eval harness for a task you define, then deliberately break it — find the prompt, the metric or the sample that makes it report a number you know is wrong.
Time — about a week
Done when
A state a stranger could checkThe harness scores a real model run, AND you can demonstrate at least one concrete way it can be made to lie.
Why it is harder — Adds measurement you have to trust downstream. Everything after this depends on the number this produces.
Where it sits
Shipping it opens 03 What a marginal dollar of test-time compute actually buys
What it reads
Just enough to unblock the next step- Build a Reasoning Model From Scratch ch3Model evaluation — the chapter that runs on the laptop.
What it ships
- BuildThe eval harness, and the ways it breaks
- BottleEval failure modes, spaced
Teach it back
If it cannot be explained plainly, it was not learned- 1Explain itExplain what you built to a friend who has never seen it. Plain words, no jargon, five sentences at most.
- 2Predicted vs realWhat you expected to happen, what actually happened, and why the two differ.
- 3Where it breaksThe question about it you still cannot answer, and the point where your explanation stops working.
- 4KeepThe one number, rule or picture worth keeping. It becomes a bottle and gets re-tested.
No notes on this build yet.
A note is a Learning Log entry with the tag study-eval-you-trust. It is written in Connor's words, filed as a draft, and appears here once it is published from the admin queue. One note can carry several build tags.