Build 11 · Blocked
Teach a small model to reason, with rewards you chose
Run GRPO on a small model against a task you defined. Find where the reward gets gamed.
Cost — Rented GPU (Modal), hours per run. The cost is the forcing function.
Done when
A state a stranger could checkThe model measurably improves on your task, AND you can demonstrate a case where it improved the metric without improving the behaviour.
Why it is harder — First build where the failure mode is the model outsmarting your objective rather than your code being wrong.
Where it sits
After Build 05 · A refinement loop, and the point where it starts drifting — Reward-driven improvement is only legible against a measured non-RL baseline.
What it reads
Just enough to unblock the next step- Build a Reasoning Model From Scratch ch6 · ch7GRPO and advanced GRPO.
- RLHF & Post-Training The objectives and their failure modes.
- CS 185/285 — Deep Reinforcement Learning ↗Policy gradients and actor-critic ONLY, and only if the maths blocks the build.
What it ships
- NotebookWhere GRPO breaks, from a run you paid for
- BottleGRPO mechanics and reward hacking, spaced
Teach it back
If it cannot be explained plainly, it was not learned- 1Explain itExplain what you built to a friend who has never seen it. Plain words, no jargon, five sentences at most.
- 2Predicted vs realWhat you expected to happen, what actually happened, and why the two differ.
- 3Where it breaksThe question about it you still cannot answer, and the point where your explanation stops working.
- 4KeepThe one number, rule or picture worth keeping. It becomes a bottle and gets re-tested.
No notes on this build yet.
A note is a Learning Log entry with the tag study-grpo-run. It is written in Connor's words, filed as a draft, and appears here once it is published from the admin queue. One note can carry several build tags.