The Data Stock and When It Runs Out

A model learns from text people have written. There is a finite amount of it on the public internet, and every year's biggest training run eats more of it. "Running out of data" means one thing precisely: the point where a frontier training run would need more public human-written text than exists — not the internet going quiet, and not every model stopping.

The best-known estimate of that date is Epoch AI's, and the most useful fact about it is that it moved:

• 2022: high-quality text would be used up before 2026 — median around 2024.
• 2024: the effective stock (about 300 trillion tokens once low-quality and duplicate text is removed and reuse is allowed) is fully used between 2026 and 2032, central date 2028 if models are trained compute-optimally, 2027 if they are overtrained 5×, and as early as 2025 at 100× overtraining.

Epoch's own reason for the move: filtered web text turned out to train as well as curated text, and training on the same data several times turned out to be worth nearly as much as fresh data — together enlarging the usable stock 2–5× (Epoch AI). As of today (2026-09-24) the window has opened; the "we run out in 2026" line still in circulation misreads both estimates.

How big the stock is

Epoch's 2030 report puts the indexed web at about 500 trillion tokens after deduplication (range 100T to 3,000T, depending on whether you count only compiled corpora or private data too), against training sets "on the order of 15 trillion tokens" — about 30× headroom as of August 2024. At the 4× a year growth in training compute, that headroom is used up in about five years. Images, video and audio push the effective stock to 400 trillion–20 quadrillion tokens; synthetic data is left out of the estimate on purpose.

How it reaches the token price

Running out of free text does not make serving a model dearer. It makes building the next one dearer: the missing data has to be licensed (see the price of training data), taken from private sources, or generated — Epoch's rough estimate is that generating a training set synthetically about doubles the compute to train on it. That cost is paid once and spread over every token the model later serves. So the data wall shows up in the token price only through the amortised cost of training, and only for the frontier models that actually hit it. Epoch's 2030 report, notably, ranks data behind power and chips as the limit most likely to bind first.

Key Claims

• Epoch's exhaustion window moved four years later between 2022 and 2024. "Our 2022 paper predicted that high-quality text data would be fully used by 2024, whereas our new results indicate that might not happen until 2028." Evidence: moderate (the forecaster's own revision; a forecast, not a measurement) (Epoch AI)
• Current window: 2026–2032 (80% interval), central 2028 compute-optimal. How models are trained moves the date by years: overtraining 5× brings it to 2027. Evidence: moderate (single research group; June 2024) (Epoch AI)
• ~500T tokens of indexed web text against ~15T used. Evidence: moderate, with a wide range (100T–3,000T) (Epoch AI)
• Data is not the first limit on scaling to 2030 — power is, then chips. Evidence: moderate (one report's ranking) (Epoch AI)
• Copyright mostly threatens the best data, not the amount. Epoch expects restrictions to fall hardest on books and reputable news — the same kind of text Bartz v. Anthropic priced. Evidence: weak (stated as expectation) (Epoch AI)

Benchmarks & Data

Figure	Value	As-of	Source
Largest known training sets	~15T tokens	2024-08	Epoch AI
Indexed web text, deduplicated	~500T tokens (100T–3,000T)	2024-08	Epoch AI
Effective public text stock	~300T tokens	2024-06	Epoch AI
Fully used (80% interval)	2026–2032	2024-06	Epoch AI
Fully used, compute-optimal / 5× / 100× overtrained	2028 / 2027 / 2025	2024-06	Epoch AI
Previous estimate (high-quality text)	before 2026, median ~2024	2022-11	Epoch AI
Effective stock with image, video, audio	400T–20 quadrillion tokens	2024-08	Epoch AI

Open Questions

• Has Epoch revised the window again since June 2024? Training sets have grown past "15T"; this KB has not read a later update.
• Does synthetic data actually work at frontier pretraining scale? Epoch calls it "very speculative" and excludes it; the answer decides whether the wall matters at all.
• Where do licensed and private corpora sit against the public stock — and at what price per token?

Related Concepts

• The price of training data — what the data costs once it is no longer free

Backlinks

Pages that reference this concept:

• The price of training data
• Epoch AI