Economics 6 min read Updated 2026-11-01
Token Pricing Economics: Input, Output, Prompt Caching & Batch Discounts
Deconstructing API unit economics: why output tokens cost 3x–5x more than inputs, and how prompt caching slashes billing by 90%.
Authored by Cadle Research Team · Strictly peer-reviewed with verifiable equations and benchmarks.
EXECUTIVE SUMMARY
API pricing is denominated in dollars per million tokens ($/MTok). Understanding the asymmetric hardware cost structure between prefill (input) and decode (output) phases is essential for architecting cost-effective AI systems.
1. The Hardware Reality: Compute-Bound vs. Memory-Bound
Why do LLM API providers charge $3.00/MTok for input but $15.00/MTok for output on the same model?
- **Prefill Phase (Input Tokens)**: The model processes the entire prompt concurrently in a single forward pass. Matrix multiplications utilize the tensor cores at peak FLOPS efficiency (**Compute-Bound**).
- **Decode Phase (Output Tokens)**: Tokens are generated sequentially, one at a time. Each generated token requires streaming the entire model's parameters (e.g. 140 GB for a 70B model) from HBM3 VRAM to the processor cores just to calculate one vector (**Memory-Bandwidth Bound**). Output generation leaves GPU compute cores sitting idle waiting for memory transfer, making it 3x to 5x more expensive to serve.
2. Prompt Caching: Slashing Prefill Costs by 80%–90%
Prompt caching allows API providers to store the intermediate KV-cache states of static prompt prefixes (such as system prompts, API documentation, or few-shot examples) in GPU memory or fast host RAM.
- **Anthropic Prompt Caching**: Writing to cache costs 1.25x the base input price, but subsequent cache hits cost only **10% of base input price** (a 90% discount).
- **OpenAI Cached Prompts**: Automatically applies a 50% discount to prompt tokens that match recent requests without requiring explicit breakpoint markers.
- **DeepSeek Cache Hits**: Input cache hits cost as low as $0.014/MTok, representing a **95% discount** over non-cached inputs.
3. Batch API Processing
For non-real-time workloads (evaluations, classification pipelines, synthetic data generation), both OpenAI and Anthropic offer 24-hour SLA Batch APIs at a flat **50% discount**. These requests execute during off-peak datacentre hours when compute utilization dips, maximizing provider gross margins and cutting developer expenditure.