LLM inference and serving

Why the bill is mostly output tokens

Two calls to the same model, same price sheet, wildly different cost. Summarizing a 4,000-token document into 100 tokens is cheap and fast. Expanding a 100-token prompt into 2,000 tokens of prose is neither. Input and output tokens are not priced the same, and they are not produced the same way.

Input tokens are read in the prefill step: the model processes the whole prompt at once, so a long prompt is a one-shot cost. Output tokens come out of the decode step, one at a time, each generated from all the tokens before it. Providers charge more for output (often several times more, for example 5x on some models) because it is the expensive phase, and because it is sequential, response time grows with how many output tokens you ask for, not with how long the prompt was.

Wrong "Our prompts are huge, so trimming the system prompt is where the savings are." Right a 5,000-token prompt read once in prefill is usually cheaper and faster than 1,500 generated tokens. Cap max_tokens to what the answer needs and stop over-generating before touching anything else.

TTFT, TPOT, and why streaming helps

Users feel two different delays. Time to first token (TTFT) is how long until the first token appears: queue time plus prefill over the whole prompt. Time per output token (TPOT), also called inter-token latency, is the gap between tokens after that. Total latency is roughly TTFT plus TPOT times the number of output tokens.

Streaming sends each token as it is produced instead of waiting for the full response. It does not make generation faster, but it hides everything after the first token: the user starts reading while the model is still writing, so perceived latency drops to about TTFT. Any serious chat UI streams. The cost is backpressure: if your client cannot consume tokens as fast as they arrive, you buffer them or the connection stalls.

What prompt caching pays for

If every request begins with the same large block (a fixed system prompt, policies, few-shot examples), you are paying to process those identical tokens on every call. Prompt caching lets the provider reuse the already-processed prefix. On Anthropic a cache read costs about 0.1x the normal input price, and the first write costs a little more than a normal read (1.25x for the 5-minute cache). OpenAI caches automatically for prompts of at least 1,024 tokens and discounts the cached portion.

The catch is that caching is a prefix match. The cached block has to be byte-identical and sit at the front. Anything that changes per request (a timestamp, the retrieved documents, the user's question) goes after the stable prefix. One volatile value near the top invalidates the cache and you pay full price again.

Wrong "Caching will cut our output costs too." Right caching only touches input tokens. Output is always generated fresh and billed in full.

Continuous batching, and why cheaper can feel slower

A serving engine runs many requests on one GPU at once. Continuous batching interleaves them: as one sequence finishes it drops out and a waiting one joins, so the GPU stays busy instead of idling until the slowest request in a fixed batch completes. This is how a server reaches a low cost per token.

The tradeoff is that throughput and per-user speed pull against each other. Pack in more concurrent sequences and total tokens per second climbs, but each user's share of the GPU shrinks, so TPOT rises. If you raise concurrency to cut cost and users report slower responses, you are watching that trade play out. There is a ceiling: once the batch saturates the hardware, adding more requests just grows the queue and latency climbs.

The KV cache is the real memory budget

Every token the model has already seen is kept as a set of key and value vectors so the next token does not recompute attention over the whole sequence. This KV cache is what makes decode fast, and it is also what runs you out of memory. Its size grows linearly with both sequence length and batch size, roughly batch_size * sequence_length * 2 * num_layers * hidden_size * bytes_per_value. A 7B model at batch size 1 can want a couple of GB of KV cache on top of the weights; multiply by concurrency and long contexts and it dominates the memory bill.

Two things follow. Long context windows are expensive in memory and in prefill time, not only in input-token price, because the cache and the attention work both scale with length. And the KV cache, rather than raw compute, usually caps how many requests you can batch: you run out of room for keys and values before you run out of arithmetic throughput.

PagedAttention and continuous batching

Allocate one contiguous KV region per sequence sized to its maximum length and you waste memory: you reserve for the worst case and fragment the rest. vLLM's PagedAttention borrows the operating-system idea of paging. It splits the KV cache into fixed-size blocks and maps a sequence's logical positions to scattered physical blocks through a table. Blocks are allocated on demand, fragmentation nearly vanishes, and identical prefixes across requests can point at the same physical blocks. vLLM hashes each block by its tokens plus the prefix before it, keeps a global table, and evicts by reference count then least-recently-used when it runs short.

More free KV memory means larger batches, which feed continuous batching (also called in-flight batching): finished sequences are evicted from the running batch mid-flight and queued ones are admitted, so the GPU never waits for the longest request. When memory does run short the engine preempts a sequence, recomputing or swapping its cache rather than failing, which surfaces as a latency spike, not an error.

Splitting the latency budget

TTFT and TPOT have different physics, so tune them apart. Prefill (which sets TTFT) is a matrix-matrix operation, highly parallel and compute-bound; it saturates the GPU and scales with prompt length. Decode (which sets TPOT) is a matrix-vector operation, memory-bandwidth bound: the bottleneck is moving weights and the KV cache out of memory, so it underuses compute and is where batching buys throughput.

Chunked prefill trades between them. Splitting a long prefill into chunks that ride alongside ongoing decodes lowers those users' TTFT contention while slightly slowing the in-flight decodes. Prompt caching (and vLLM's automatic prefix caching) attacks TTFT directly by skipping prefill for a reused prefix, which is why providers advertise large TTFT reductions on cached prefixes.

Prompt caching mechanics and what breaks it

Provider caching is a prefix match with real economics. On Anthropic a cache read is about 0.1x input; a write is 1.25x for the default 5-minute TTL or 2x for the 1-hour TTL, so caching pays as soon as the prefix is reused: the 5-minute cache amortizes its 0.25x write premium on the first cache read, and the 1-hour cache (2x write) after roughly one to two reads. There is a minimum cacheable length (roughly 1,024 tokens on several providers, higher on some models) and entries expire after minutes of inactivity, so short prompts or low-traffic endpoints may never register a hit.

What silently breaks it is any change in the prefix. A datetime.now() or request ID in the system prompt, retrieved documents placed above the fixed prompt, a reordered tool list, or an unsorted JSON serialization all move bytes and invalidate everything after the change. Keep the stable content frozen and first, put volatile content last, and confirm with the provider's cache-read token counter that hits are actually landing.

Self-host vs API, and the real break-even

The pitch for self-hosting is a lower price per token. That number only shows up at high sustained utilization, because a rented GPU costs the same idle or busy, and you now own scaling, upgrades, and on-call. A box at 20% utilization can cost more per real token than the API. Steady, predictable, high-volume load favors self-host; spiky or low-average load favors the API, which spreads its hardware across many tenants.

Wrong "Self-hosting removes our rate limits." Right it swaps a shared quota you can ask to raise for a fixed GPU capacity ceiling you must provision and queue behind. Hosted APIs enforce per-organization requests-per-minute and tokens-per-minute by usage tier, and a 429 means back off, smooth bursts, and budget tokens per call, then raise the tier. Self-hosting turns that soft quota into a hard wall at your provisioned throughput. Capacity planning stays the same either way: estimate peak concurrent requests, tokens in and out per request, and the KV-cache memory that concurrency implies, then size to your P99 rather than your average.