Explained · Models
Quantization
A model's knowledge is stored as billions of numbers. Quantization rounds those numbers to a coarser precision — like compressing a photo — so the model takes far less memory and runs faster and cheaper. Done carefully, the quality loss is barely measurable, which is why almost every model you use is quantized.
Where it breaksThe damage is uneven and does not show up on an average score. A heavily rounded model usually holds its general ability while degrading on exactly the narrow things you cared about — long arithmetic, rare names, a language it barely knew, strict output formats. Push past four bits and the failures arrive suddenly rather than gradually, so the safe move is always to test on your own cases.
4× smaller memory cut from storing weights at 4 bits instead of 16Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (ICLR 2023) · 2022-10-31
What is being rounded
A model is a very large table of numbers called weights, and each weight is normally stored in sixteen bits — enough precision for about five significant digits. Quantization stores them in fewer bits: eight, or four, sometimes fewer. Four-bit storage allows only sixteen distinct values, so the trick is not to round every weight to the same coarse grid but to split the weights into small groups and give each group its own scale, chosen so the sixteen available values land where that group's numbers actually cluster. The saving is direct and easy to compute: at sixteen bits a seven-billion-parameter model needs about fourteen gigabytes just to hold its weights; at four bits it needs about four. That is the difference between a model that fits on a laptop and one that does not.
Why it makes things faster, not just smaller
The instinct is that this is a storage trick. It is really a bandwidth trick. Writing each token of a reply requires streaming the entire set of weights out of memory and through the chip, and modern accelerators can do arithmetic far faster than they can fetch the numbers to do it on — the chip spends most of its time waiting for memory. Quarter the bytes and you quarter the waiting, so generation speeds up roughly in proportion even though the amount of arithmetic is unchanged. This is also why quantization matters most for single-user, one-request-at-a-time serving, where the memory road is emptiest, and why it is the first thing anyone reaches for when running a model on their own hardware.
Where it breaks
Average scores hide the damage. A four-bit model typically holds its benchmark numbers within a point or two while getting distinctly worse at the specific things that were already marginal for it: multi-step arithmetic, rare proper nouns, low-resource languages, and strict machine-readable output where one malformed bracket fails the whole request. The loss is also non-linear. Eight bits is nearly free, four bits is usually a good trade, and below four the model tends not to degrade gracefully but to fall over — coherent one moment, repeating itself the next. A third trap is that a small model squeezed hard is often worse than a larger model squeezed gently at the same memory budget, so the right comparison is at equal gigabytes rather than equal parameter count.
The decision it changes
Quantization is what moves a model across a hardware boundary — from a rented data-centre GPU to a consumer card, or from a workstation to a laptop — and boundaries are where the economics change, not the percentages. If four-bit weights let the model fit in the memory you already own, the marginal cost of running it drops towards the electricity bill. If it merely makes a rented GPU slightly less busy, the saving is real but modest. So the sequence is: pick the largest model that fits your memory after quantization, then verify on your own evals that the narrow things you depend on survived. Fit first, then measure — never the other way round.