The New Unit Economics of AI Inference: Batching, KV Cache, and Tokens

GPU hours are only the first line item

Teams often estimate inference cost by dividing an hourly accelerator price by a vendor's advertised tokens-per-second number. That is a capacity estimate, not a product cost. Prompt length, idle time, queueing, retries, cache misses, and the difference between prefill and decode decide what users actually pay for.

A production estimate should use:

cost per output token = total serving cost / successful output tokens

The numerator includes compute, memory, networking, storage, orchestration, observability, and failed or abandoned requests. The denominator should exclude tokens produced for requests that users never received.

Prefill and decode have different economics

Prompt processing, or prefill, reads a large input and builds attention state. Decode generates output incrementally. A workload with short prompts and long answers behaves differently from one with huge documents and short answers.

Measure at least these four quantities:

• Time to first token
• Inter-token latency
• Input tokens per request
• Output tokens per request

A system tuned only for output throughput may have unacceptable first-token latency. A system tuned only for first-token latency may waste capacity by running tiny batches.

Batching is a queueing problem

Continuous batching lets a serving engine add and remove sequences as requests arrive. It can increase utilization, but it also creates a tradeoff: waiting briefly for a fuller batch may improve total throughput while making an individual request slower.

Use a latency budget to choose the batching window. For example, if the product promises a first token within a fixed target, the scheduler cannot wait indefinitely for more work. The benchmark must report p50, p95, and p99 latency, not just averages.

A simple test matrix is more informative than a single benchmark:

KV cache changes the capacity calculation

During generation, the key-value cache stores attention state for active sequences. Its memory use grows with context length and concurrency. A server that looks healthy with short prompts can run out of memory when users paste long documents or keep conversations open.

Useful controls include:

• Maximum context length per route
• Maximum concurrent sequences
• Prefix caching for repeated system prompts
• Eviction policy for inactive sessions
• Separate pools for interactive and batch traffic

Cache reuse can reduce work dramatically, but it should be measured. A cache that is too small creates churn; a cache that is too broad can protect low-value entries while evicting hot prefixes.

Routing is part of the model decision

Not every request needs the largest model. A router can send simple classification, extraction, or summarization tasks to a smaller model and reserve a larger model for requests that need deeper reasoning or broader context.

Routing must be evaluated on quality, not just cost. Track escalation rate, correction rate, user retries, and downstream task success. A cheap answer that causes a retry may cost more than a slightly more expensive first attempt.

Cost per successful task

Tokens are a useful operational unit, but they are not the business outcome. For an agent or workflow, calculate:

cost per successful task = total cost / tasks completed without human repair

Include tool calls, retries, moderation checks, retrieval, and failed validations. This measure prevents teams from celebrating token efficiency while the product still requires expensive manual correction.

Spend optimization effort in this order

Optimize in this order:

1. Remove unnecessary prompt and tool output tokens.
2. Add request limits and cancellation handling.
3. Measure prefill and decode separately.
4. Tune continuous batching against latency SLOs.
5. Add prefix or KV-cache reuse where traffic is repetitive.
6. Route simple tasks to smaller models.
7. Quantize only after quality and latency tests exist.
8. Buy more accelerators only after utilization and queueing are understood.

The implication for capacity planning

AI inference economics are an application performance problem. The accelerator matters, but so do context limits, cache policy, batching, routing, and the definition of success.

Before buying capacity, instrument one request from queue to completed task. If the record cannot explain where time, tokens, and failures went, the next GPU is a guess. Measure first, then make the hardware follow the workload.