Explained · Models
Context windows
The context window is the model's working memory — how much text it can hold in mind at once, counting your question, the conversation so far, and anything you pasted in. When it fills up, the oldest material falls out, which is why very long chats start to drift. Bigger windows exist; they cost more to run.
Where it breaksA large window is a capacity, not a promise of attention. Models reliably lose material buried in the middle of a very long input, so a fact on page forty can be present and unused. And every token in the window is re-read and re-billed on every turn, so a long chat gets slower and more expensive with each message even when nothing new is said.
128,000 tokens context window of GPT-4o, roughly a 300-page bookOpenAI API model documentation · 2024-05-13
What the window actually is
There is no memory inside a chatbot in the way people imagine. Each time you send a message, the entire conversation — your first question, every reply, every document you pasted — is sent to the model again as one block of text, and the model reads all of it from scratch before writing a word. The context window is the maximum size of that block. It is measured in tokens, the word-pieces the model reads in, and a rough conversion is four characters or three-quarters of a word per token. So a hundred-thousand-token window is a few hundred pages. When the conversation grows past the limit, something has to go: most products silently drop or summarise the oldest turns, which is why a long chat can forget an instruction you gave it at the start.
Why bigger windows are expensive
While the model reads the block, it keeps a running set of intermediate numbers for every token — the key-value cache — sitting in the chip's fast memory so that the next token can be produced without re-reading everything from the beginning. That cache grows in direct proportion to the conversation. A long context therefore occupies a large slice of a very expensive GPU for the whole session, and that slice cannot be sold to anyone else, which is the real reason long context costs money. On top of that, the attention step that lets each token look at every other token grows with the square of the length, so doubling the input more than doubles the work of reading it. Providers pass both through as input-token pricing and as slower first responses on long prompts.
Where it breaks
The headline number is a ceiling, not a guarantee. Measured carefully, models retrieve facts placed at the start and end of a long input far more reliably than facts placed in the middle, so material can be inside the window and effectively invisible — the "lost in the middle" effect. The second failure is economic and catches teams by surprise: every turn re-sends the whole history, so message thirty in a long chat pays for all twenty-nine before it, and a conversation that felt cheap at the start quietly becomes the most expensive thing in your product. The third is dilution. Padding a prompt with an entire manual so that the answer is definitely in there tends to lower answer quality, because the relevant paragraph is now competing with fifty irrelevant ones.
What to do instead of filling it
Treat the window as scarce even when it is large. Retrieve the few passages that matter rather than pasting the source, because a short precise prompt beats a long complete one on both quality and cost. Summarise old turns rather than carrying them verbatim. Put the instruction that must be obeyed at the very end, where models attend most reliably. And if a workflow genuinely needs a hundred pages every time, use prompt caching so the fixed part is paid for once rather than on every turn. The engineering question is never "does it fit" — it is "what does this cost on every message from here on".