nanolm — a language model from scratch
The flagship credibility artifact of the whole study ladder: a working language model built from a byte-pair tokenizer through the Transformer and optimizer to an actual training run, following Stanford CS336 Assignment 1. Building the thing that creates all the compute demand is the strongest possible grounding for reasoning about the labs.
ExternalTrains on PyTorch + a GPU — code lives in the menfem-lab repo. An in-browser tokenizer / inference demo may land here later.
Stack
The problem
You cannot reason about the frontier labs with an engineer's certainty from the outside. The model layer has to be built to be understood.
The approach
Follow CS336 A1 end-to-end with minimal scaffolding: a BPE tokenizer, Transformer blocks, an AdamW optimizer, and a small training loop on a modest corpus — then profile the result against the inference roofline math.
Decisions & trade-offs
- —From scratch, no Hugging Face Trainer — the point is to touch every part.
- —Small enough to train on accessible hardware; correctness over scale.
- —Instrument it so the KV-cache / arithmetic-intensity lessons become measurable.
Where it stands
Target: a trained mini-LM + a /notebook build-log — "I built a language model from scratch."
Build log
The running diary will appear here as /notebook entries once the build starts.