Skip to content

AI, explained simply. No jargon.

Explained the way you'd tell a sharp friend: one number each, and where it breaks.

30 terms

Fine-tuning takes a trained model and gives it a short second education on your own examples, your tone, your formats, your field, so the behaviour sticks without teaching anything from scratch. Think of hiring a brilliant generalist, then spending a week training them on your house style.

Where it breaks

It teaches behaviour, not facts, and people reach for it to fix the wrong problem. A model fine-tuned on your documents will start sounding like them without reliably knowing them, so answers get more confident and no more correct. And the moment your examples change or a better base model ships, you own a training pipeline you have to run again.

The number

10,000× fewer trainable parameters: reduction LoRA reports when adapting GPT-3 175B instead of retraining every weight.

Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" · 17 Jun 2021

The longer read

The context window is the model's working memory, how much text it can hold in mind at once, counting your question, the conversation so far, and anything you pasted in. When it fills up, the oldest material falls out, which is why very long chats start to drift. Bigger windows exist; they cost more to run.

Where it breaks

A large window is a capacity, not a promise of attention. Models reliably lose material buried in the middle of a very long input, so a fact on page forty can be present and unused. And every token in the window is re-read and re-billed on every turn, so a long chat gets slower and more expensive with each message even when nothing new is said.

The number

128,000 tokens: context window of GPT-4o, roughly a 300-page book.

OpenAI API model documentation · 13 May 2024

The longer read

A model's knowledge is stored as billions of numbers. Quantization rounds those numbers to a coarser precision, like compressing a photo, so the model takes far less memory and runs faster and cheaper. Done carefully, the quality loss is barely measurable, which is why almost every model you use is quantized.

Where it breaks

The damage is uneven and does not show up on an average score. A heavily rounded model usually holds its general ability while degrading on exactly the narrow things you cared about, long arithmetic, rare names, a language it barely knew, strict output formats. Push past four bits and the failures arrive suddenly rather than gradually, so the safe move is always to test on your own cases.

The number

4× smaller: memory cut from storing weights at 4 bits instead of 16.

Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (ICLR 2023) · 31 Oct 2022

The longer read

Models do not read letters or words. Text is first cut into tokens, common word-pieces from a fixed vocabulary, and every token is billed, counted against the context window, and produced one at a time. A rough rule for English: a token is about four characters, or three-quarters of a word.

Where it breaks

The rule of thumb only holds for ordinary English prose. Code, JSON, long numbers, rare names and most non-English languages fragment into far more tokens per character, so the same content can cost several times more in one language than another. Character-level tasks, counting letters, reversing a word, fail for the same reason: the model never saw the letters.

The number

~4 characters: per token, for ordinary English text.

OpenAI help centre, "What are tokens and how to count them" · 1 Jan 2024

The longer read

The transformer is the design nearly every modern AI model uses. Its key move is attention: while processing each word, the model looks at every other word at once and weighs which ones matter. That parallel look is what made models trainable on enormous amounts of text.

Where it breaks

Attention compares every position with every other, so the work grows with the square of the input length: ten times the text is a hundred times the reading work. That is why long documents are slow and costly, why context limits exist at all, and why a large slice of AI research is spent finding ways to avoid doing the full comparison.

The number

8 attention heads: parallel attention comparisons per layer in the original transformer.

Vaswani et al., "Attention Is All You Need" (NeurIPS 2017) · 12 Jun 2017

Instead of one dense model where every part works on every word, a mixture of experts holds many specialised sub-networks and uses a small router to wake only a couple of them per token. The model is enormous on paper and cheap per answer, because most of it stays asleep.

Where it breaks

You still have to hold the whole model in memory even though only a fraction runs, so the hardware bill tracks total size while the speed tracks active size. Routing also makes serving lumpy: which experts a batch needs varies request by request, so a busy server can end up shuffling weights around instead of computing.

The number

46.7B total, 12.9B active per token: parameter split in Mixtral 8x7B.

Mistral AI, "Mixtral of Experts" · 8 Jan 2024

Distillation trains a small model to copy a large one. The big model answers a pile of questions, and the small model learns to reproduce those answers, ending up far cheaper to run while keeping most of the ability on the work you care about.

Where it breaks

The student inherits the teacher's mistakes and only covers ground the teacher was asked about, so it looks excellent on familiar work and falls apart just off the training distribution. Provider terms are the other constraint: several forbid using their outputs to train a competing model, which makes the cheapest route the one you may not legally take.

The number

40% smaller, 97% of performance: DistilBERT's size cut and retained GLUE score against BERT.

Sanh et al., "DistilBERT, a distilled version of BERT" · 2 Oct 2019

A model that does not know something does not stop, it produces the most plausible continuation, which can be a fluent, well-formatted, entirely invented fact. It is not lying or malfunctioning; generating likely text is the only thing it does, and truth is not a separate check inside it.

Where it breaks

The invented material is indistinguishable in tone from the correct material, so it survives a skim and gets caught only by someone who already knows the answer. It clusters exactly where verification is hardest: citations, case numbers, prices, quotes and dates. Retrieval reduces it and does not remove it, because a model handed the wrong page will still answer confidently from it.

The number

58% truthful: best-performing model's score on TruthfulQA against 94% for humans.

Lin et al., "TruthfulQA: Measuring How Models Mimic Human Falsehoods" (ACL 2022) · 8 Sep 2021

A raw model predicts likely text, which is not the same as being helpful. RLHF shows people pairs of answers, records which they prefer, trains a scoring model on those judgements, and then tunes the model to score well. It is the step that turns a text predictor into an assistant.

Where it breaks

The model learns what raters rewarded, which is what looked good rather than what was right, so it drifts towards long, confident, agreeable answers, and flattery is a trained behaviour rather than an accident. Whose preferences were collected also becomes policy for every user, invisibly.

The number

1.3B beat 175B: labelers preferred the small RLHF-tuned InstructGPT over the 100× larger base GPT-3.

Ouyang et al., "Training language models to follow instructions with human feedback" · 4 Mar 2022

Training builds the model; inference is using it. Every time you ask a chatbot a question, a data centre runs your words through the model and generates the reply, one word-piece at a time. Training happens once, inference happens billions of times a day, and every run costs money.

Where it breaks

The bill scales with use, not with success. A product that gets ten times more popular gets a compute bill roughly ten times larger, which is the opposite of how software normally behaves. Teams price a feature off a demo, ship it to everyone, and discover the margin was never there, long answers, long chat histories and retried requests all multiply the same per-token cost.

The number

$0.15 / $0.60 per million tokens: listed input / output price of GPT-4o mini at launch.

OpenAI, "GPT-4o mini: advancing cost-efficient intelligence" · 18 Jul 2024

The longer read

While writing a reply, the model keeps a running set of intermediate numbers for every token so far, so each new token does not require re-reading the whole conversation. That store is the KV cache. It is what makes generation fast, and it is also what fills up the GPU.

Where it breaks

The cache grows with every token and lives in the accelerator's fast memory, so on long conversations it can occupy more space than the model itself. When it does not fit, the server either evicts and recomputes, a visible stall, or refuses new users, which is why concurrency on a long-context product collapses long before the chips run out of arithmetic.

The number

up to 24× throughput: gain vLLM reports over HuggingFace Transformers by managing KV cache memory in pages.

Kwon et al., "Efficient Memory Management for LLM Serving with PagedAttention" (SOSP 2023) · 12 Sep 2023

The longer read

Latency is how long one person waits; throughput is how many people the machine serves per second. They pull against each other, the main way to raise throughput is to make each request wait a little so it can be processed alongside others.

Where it breaks

Teams optimise the average and ship a product that feels slow, because users experience the worst few per cent of requests, not the mean. The two numbers also hide behind one word: an "improvement" that doubles throughput while adding half a second before the first word appears is a win on the invoice and a loss in the chat window.

The number

TTFT 2,000 ms, per-token 200 ms: latency constraints MLPerf Inference imposes on the Llama 2 70B server scenario.

MLCommons, MLPerf Inference v4.0 results and rules · 27 Mar 2024

Running one request through a GPU wastes most of the chip, because the weights have to be fetched either way. Batching processes many requests together against the same fetched weights, so serving twenty people costs barely more than serving one, which is where the margin in inference comes from.

Where it breaks

Batches are only free while there is memory for everyone's KV cache, so long conversations shrink the batch and the cost per request climbs. Naive batching also makes everyone wait for the slowest reply in the group, which is why serving stacks swap finished requests out mid-flight rather than running fixed batches.

The number

up to 36.9× throughput: gain from continuous batching at the same latency, versus fixed-batch serving.

Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022) · 11 Jul 2022

A small fast model guesses the next few tokens, and the big model checks them all in one pass. Guesses it agrees with are kept, the first disagreement is corrected. The output is identical to running the big model alone, it just arrives sooner.

Where it breaks

The saving depends entirely on how often the small model guesses right, and acceptance collapses on the hard, unusual text where you most wanted the speed. Every rejected guess is wasted work, so on a busy server the technique can raise the cost per request even while making a single request feel faster.

The number

2–3× faster: speedup reported with identical output distribution.

Leviathan et al., "Fast Inference from Transformers via Speculative Decoding" (ICML 2023) · 30 Nov 2022

A GPU is a general-purpose parallel processor that happens to be excellent at AI; a TPU is a chip built for one job and nothing else. Custom silicon can be cheaper per unit of work, and it locks you to one vendor's software and one company's supply.

Where it breaks

The comparison is rarely about the chips. It is about the software everyone already writes against, the availability of parts, and whether your workload matches what the custom chip was designed for. A chip that is twenty per cent better on paper and requires rewriting your serving stack is not twenty per cent better.

The number

80 GB at 3.35 TB/s: memory capacity and bandwidth of the NVIDIA H100 SXM.

NVIDIA H100 Tensor Core GPU datasheet · 21 Mar 2023

HBM is memory stacked vertically and mounted right next to the processor, giving it a far wider road to the chip than ordinary memory. It is what lets an accelerator feed itself fast enough to be useful, and it is one of the scarcest, most expensive parts of an AI machine.

Where it breaks

It is the supply bottleneck and the capacity ceiling at the same time. Only a few firms can make it, packaging capacity is booked far ahead, and once a model plus its caches exceed what is on the card you are splitting the job across chips and paying in network hops.

The number

141 GB at 4.8 TB/s: HBM3E capacity and bandwidth on the NVIDIA H200.

NVIDIA H200 Tensor Core GPU datasheet · 13 Nov 2023

The longer read

Processors got much faster at arithmetic than memory got at delivering numbers to work on, so modern accelerators spend most of their time waiting rather than computing. Almost every optimisation in AI serving is an attempt to move fewer bytes, not to do less maths.

Where it breaks

It makes intuition about performance wrong. Buying a chip with more arithmetic performance changes nothing if the workload is starved for memory bandwidth, and a model that looks twice as expensive by parameter count can run at nearly the same speed. Judge serving hardware on bytes per second and capacity before teraflops.

The number

~300 FLOPs per byte: ratio of BF16 arithmetic (989 TFLOPS) to memory bandwidth (3.35 TB/s) on an H100 SXM.

NVIDIA H100 Tensor Core GPU datasheet · 21 Mar 2023

The longer read

AI capacity is limited less by chips than by electricity and the ability to connect to a grid. A modern AI hall draws as much power as a small town, and the wait for a grid connection is now measured in years, which is why builders chase places with spare power.

Where it breaks

Announced capacity is not delivered capacity. A site can have chips ordered and still sit idle waiting on transmission, transformers or turbines, and those queues do not respond to money. Local grids also push back, so power procurement, not silicon, increasingly sets when capacity actually arrives.

The number

460 TWh in 2022, 620–1,050 TWh by 2026: global data centre electricity consumption and forecast range.

IEA, "Electricity 2024" report · 24 Jan 2024

PUE is the ratio of everything a data centre draws to the part that reaches the computers; 1.5 means half again is spent on cooling and losses. AI racks run so hot that air cooling is running out of room, which is pushing the industry to plumb liquid directly to the chips.

Where it breaks

PUE flatters an efficient building and says nothing about whether the work inside was worth doing, a hall at 1.1 running idle servers beats a hall at 1.4 doing useful work on the metric. It also excludes water, which is where the local objection usually lands.

The number

1.10: Google fleet-wide trailing twelve-month power usage effectiveness.

Google data centre efficiency disclosures · 31 Dec 2023

Instead of hoping the model memorised the right facts, you hand it the right documents at question time. The system searches your files, pastes the best passages into the prompt, and the model answers from those. It is why a chatbot can cite your company wiki without being retrained on it.

Where it breaks

The weak link is the search, not the model. If retrieval pulls the wrong five paragraphs, the model answers fluently and wrongly, with a citation attached, which is more dangerous than no answer at all. Questions that need the whole corpus ("what changed this quarter?") fail quietly, because fetching the nearest few passages is structurally the wrong move for them.

The number

21 million passages: size of the Wikipedia index the original retrieval systems searched per question.

Karpukhin et al., "Dense Passage Retrieval for Open-Domain QA" (EMNLP 2020) · 30 Sep 2020

The longer read

A chatbot answers you; an agent acts for you. Give it a goal and it runs a loop, pick a step, use a tool like search or code, check the result, adjust, repeat until the job is done. The bet is delegation: you review finished work instead of doing the work yourself.

Where it breaks

Errors compound. A step that is right nine times in ten is right barely a third of the time over ten steps, and an agent that took a wrong turn keeps going confidently rather than stopping. The practical failures are loops that never terminate, tool calls made on stale assumptions, and a bill that quietly multiplies because every step is a fresh model call carrying the whole history.

The number

500 tasks: size of SWE-bench Verified, the human-validated set of real GitHub issues agents are graded on.

OpenAI, "Introducing SWE-bench Verified" · 13 Aug 2024

The longer read

Evals are the test suite for an AI system: a fixed set of questions with known good answers, scored automatically every time something changes. Without them, you are guessing whether a new model or prompt actually made things better. Teams with sharp evals ship improvements; teams without them ship vibes.

Where it breaks

An eval measures what it measures, and a system tuned against it gets better at the test rather than the job. Public benchmarks leak into training data, so a high score can mean the answers were memorised. And the questions people most want graded, was that summary any good, have no single right answer, so teams grade them with another model and inherit its biases.

The number

57 subjects: breadth of MMLU, the benchmark most model launches still quote.

Hendrycks et al., "Measuring Massive Multitask Language Understanding" (ICLR 2021) · 12 Jan 2021

The longer read

An embedding turns a piece of text into a long list of numbers that stands for its meaning, positioned so that passages about similar things end up near each other. Search then becomes geometry: find the nearest points. It is the machinery behind semantic search, recommendations and the retrieval half of RAG.

Where it breaks

"Near" means similar-sounding, not correct or relevant. A policy and its exact opposite sit close together because they discuss the same subject, so retrieval happily returns the wrong one. Embeddings also carry no sense of time or authority, so last year's superseded document scores exactly as well as this week's replacement.

The number

1,536 dimensions: default vector length of OpenAI text-embedding-3-small.

OpenAI, "New embedding models and API updates" · 25 Jan 2024

The longer read

Guardrails are the checks placed around a model rather than inside it: filters on what goes in, validators on what comes out, and limits on what the system is allowed to do. They exist because you cannot make a model refuse reliably, but you can refuse on its behalf.

Where it breaks

Guardrails written as instructions in the prompt are advice, not enforcement, anything the model can be talked out of, it will be. Real enforcement has to sit outside the model, in code that checks the output and holds the permissions. The other failure is over-blocking: filters tuned for safety quietly refuse legitimate work, and nobody reports the refusals they did not see.

The number

free: price of the OpenAI moderation endpoint, the standard input filter.

OpenAI platform documentation, Moderation guide · 25 Jan 2024

A model cannot tell your instructions apart from the text it is reading, both arrive as the same stream of words. So a web page, email or document can carry instructions of its own, and a system that reads it may follow them. It is the defining security problem of AI that touches the outside world.

Where it breaks

The damage scales with permissions, not cleverness. A chatbot that only writes text can be made to say something embarrassing; an agent with mail access, a browser and a database can be made to exfiltrate data by fetching an attacker-chosen URL. There is no known reliable filter, because the malicious input is ordinary language.

The number

ranked LLM01: prompt injection's position at number one in the OWASP Top 10 for LLM applications.

OWASP Top 10 for Large Language Model Applications, 2025 edition · 17 Nov 2024

The longer read

MCP is a common plug format between AI assistants and the tools and data they use. Write one server for your system and any assistant that speaks the protocol can use it, instead of every product building a bespoke connector for every service.

Where it breaks

A protocol standardises the wiring, not the judgement. Tool descriptions arrive as text the model reads, so a malicious or careless server can steer an assistant that connected to it. And an easy plug encourages connecting many servers at once, which grows the permission surface faster than anyone audits it.

The number

open-sourced 25 Nov 2024: date Anthropic published the Model Context Protocol specification.

Anthropic, "Introducing the Model Context Protocol" · 25 Nov 2024

Benchmarks are the standard exams models are graded on, and they are how launches get compared. They are useful for tracking the field over years and close to useless for choosing a model for your particular job, which your own test cases decide.

Where it breaks

Questions published on the open web end up in training data, so a score can measure memory rather than ability. Scores are also reported under conditions that differ between labs, and a benchmark near saturation stops separating anyone, the last few points are noise, not progress.

The number

34%: accuracy of skilled non-experts with unrestricted web access on GPQA questions, against 65% for domain PhDs.

Rein et al., "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" · 20 Nov 2023

Not every question needs the biggest, most expensive model. A router reads each request and sends easy ones to a small, cheap model and hard ones to the heavyweight. Users rarely notice the difference, the bill notices, which is why most serious AI products quietly do this.

Where it breaks

The router has to decide how hard a question is before anyone has answered it, which is the hard part of the problem. Misroute a genuinely difficult request to the small model and the user gets a confidently wrong answer at a discount. Worse, routing quality drifts silently: nothing alerts you when last month's rules started sending the wrong third of your traffic to the cheap tier.

The number

~17× cheaper input: gap between GPT-4o at $2.50/M and GPT-4o mini at $0.15/M input tokens.

OpenAI API pricing page · 18 Jul 2024

The longer read

Most requests to an AI system start with the same long preamble, instructions, examples, a document. Prompt caching stores the model's half-finished reading of that fixed part so the next request skips re-reading it, cutting both the price and the wait before the first word appears.

Where it breaks

The cache only hits on an exact prefix match, so putting a timestamp or a user name at the top of the prompt silently destroys it for every request. Caches also expire in minutes, which means low-traffic endpoints pay the write premium and rarely collect the discount.

The number

10% of base input price: what Anthropic charges for a cache read against a written prompt cache.

Anthropic, "Prompt caching with Claude" · 14 Aug 2024

An open-weights model is one whose trained numbers you can download and run on your own hardware. That is not the same as open source, the training data and code usually stay private, and the licence often carries conditions, but it does mean nobody can take the model away from you.

Where it breaks

Downloading is free; running is not. You take on the GPUs, the serving stack, the updates and the security work an API provider was absorbing, and the break-even against per-token pricing arrives later than people expect. Licences also vary sharply, so "open" can still forbid your specific commercial use.

The number

700 million monthly users: threshold above which the Llama 3.1 licence requires a separate agreement.

Meta, Llama 3.1 Community License Agreement · 23 Jul 2024