LLM Inference Handbook (Modular)

What this is, and what it is NOT

It is a definitional standard and a calculator set. It is not a benchmark source.

Read across the landing page and the metrics chapter 2026-08-06: every numerical example is illustrative. No measured hardware results, no model names, no deployment figures. The one worked example — 220 output tokens, TTFT 80 ms, E2EL 2200 ms, giving TPOT 9.68 ms/token — is a playground scenario, not a measurement.

So it does not move a number on this rung, and was ingested on Connor's explicit approval with that understood. Its value is different and real: it supplies the vocabulary a cost claim needs to be checkable, which is directly what the rung's cost-measurement-problem concept is about. You cannot compare cost-per-task across harnesses without agreeing what TTFT and goodput mean first.

The formulas (the actual reason to keep it)

Metric	Definition
TTFT	Time to generate the first token after sending a request
E2EL	Total latency — request sent to final token received
TPOT	(E2EL − TTFT) / (Total Output Tokens − 1) — mean gap between consecutive tokens, excluding the first
ITL	The exact pause between two consecutive tokens. For a single request mean ITL = TPOT. Across requests: Average ITL = Sum of all ITLs / Total Output Tokens
RPS	Total completed requests / (T1 − T2)
TPS	Tokens per second, split into input TPS and output TPS
Goodput	Requests completing per second while meeting defined SLOs

Goodput is the one that matters for this rung. Throughput counts work done; goodput counts work done acceptably. A cost-per-token figure computed against throughput rather than goodput flatters any system that is fast and wrong, or fast and unusably laggy — which is exactly the measurement failure the rung tracks.

Note the − 1 in TPOT. It is easy to drop, and dropping it inflates the denominator and understates per-token latency on short generations.

Stated thresholds

• Chatbot TTFT: "under 500 milliseconds to feel responsive"
• Code completion TTFT: "below 100 milliseconds for seamless developer experience"
• Example SLO: TTFT ≤ 200 ms and E2EL ≤ 3000 ms

⚠️ These are unsourced rules of thumb, not measured UX research. Usable as convention; not citable as a finding.

The latency-throughput tradeoff

Stated cleanly, and it is the mechanism behind most "our inference is cheaper" claims:

"Maximize throughput … usually means using larger batch sizes" but "can slow down responses for individual users." Conversely, "Minimize latency … often involves small batches and isolated compute resources, but it means you'll use GPUs less efficiently."

"larger batches … can improve aggregate TPS by keeping the GPU busy, but may increase queueing time, TTFT, or per-user TPOT."

Any vendor cost-per-token number is a point chosen on this curve. Without the batch size and the SLO, the number is not comparable to another vendor's.

Calculators (apparatus)

Genuinely useful tools, worth knowing exist:

• GPU Memory Calculator — /getting-started/calculating-gpu-memory-for-llms/
• KV Cache Memory Calculator — /inference-optimization/kv-cache-offloading/#how-to-calculate-the-kv-cache-size
• Batching Strategy Simulator — /inference-optimization/static-dynamic-continuous-batching/
• Chunked Prefill Scheduler — /inference-optimization/static-dynamic-continuous-batching/#chunked-prefill
• Quantization Memory Impact Visualizer — /model-preparation/llm-quantization/#quantization-formats
• GPU Comparison Table — /getting-started/choosing-the-right-gpu/#matching-gpus-to-open-source-llms

Structure

Foundations · Planning your deployment · Model preparation · Model interaction · Inference optimization · Kernel optimization · Infra and operations.

Evidence grade

Vendor-published reference. Authoritative for definitions, empty of measurements, and conflicted by construction — Modular sells an inference stack, so its framing of what to measure is not neutral. Cite it for a formula or a definition. Never cite it for a performance or cost figure, because it does not contain one.

Sections not yet read: kernel optimization, infra and operations, and the deployment-planning chapters. If a later claim depends on those, read them first rather than assuming this close read covers them.