Optimizing AI in Production
The cost model
Every API call to an LLM has two independent cost components: input tokens and output tokens. These are priced separately, at different rates, because they represent different compute operations. Input tokens are processed in a single forward pass; output tokens are generated autoregressively — one token at a time, each requiring a full forward pass. This is why output tokens cost more than input tokens, typically two to five times more per token.
Input tokens
Input tokens are everything you send to the model: the system prompt, the conversation history, retrieved documents, tool definitions, and the current user message. You pay for all of them on every request, because the model processes the full context window on every call. There is no differential pricing between "important" and "boilerplate" context — a token in the system prompt costs the same as a token in a retrieved document. This is what makes context engineering a cost engineering problem as much as a quality problem.
For Claude Sonnet pricing (as of mid-2025): approximately $3 per million input tokens, $15 per million output tokens. Exact prices change with model releases — always check the current pricing page rather than relying on remembered figures.
Cache hits
Cached input tokens are priced at 10% of the normal input token rate. A 50K token system prompt that costs $0.15 per request uncached costs $0.015 per request when cached. Cache writes cost 25% more than uncached input tokens — the first request that populates the cache pays a premium. The breakeven on any cached prefix is at the second request.
Output tokens and max_tokens
You only pay for output tokens actually generated, not for max_tokens. Setting a high max_tokens does not increase cost; it is an upper bound. The cost is determined by how many tokens the model generates. This means that prompt engineering to produce shorter outputs (when brevity is acceptable) directly reduces output token costs.
Prompt caching ROI
Prompt caching is the highest-ROI optimization available for production AI systems with static context. It is almost always worth implementing. The math is straightforward.
Breakeven calculation
Let P be the number of tokens in your cacheable prefix, and C be the input token price per million. The cache write cost is 1.25 × (P / 1M) × C. The cache read cost is 0.1 × (P / 1M) × C. The savings per request after the first is 0.9 × (P / 1M) × C. The extra cost on the first request (cache write vs normal) is 0.25 × (P / 1M) × C. Breakeven is at the second request — the savings on request two ($0.9) exceed the premium on request one ($0.25).
For a 100K token prefix at $3/MTok: cache write costs $0.375, cache read costs $0.030, uncached input costs $0.30. Net result: the first request costs $0.075 more than uncached ($0.375 vs $0.30), but every subsequent request saves $0.27 ($0.30 vs $0.03). A system making 100 requests per day saves roughly $27/day on input token costs from this prefix alone.
What to cache
System prompts over 1,000 tokens. If your system prompt is more than roughly 1,000 tokens — detailed instructions, embedded documentation, persona definition — it is worth caching. Below 1,000 tokens, the absolute savings are small enough that the complexity may not be worth it.
Tool definitions. If you have more than ten tools, their combined token cost is significant. Cache the tool definitions before the conversation history.
Static reference documents. Documents that are the same across many requests — product documentation, API specifications, a codebase that the model reasons over — are prime cache candidates.
Do not try to cache conversation history. Conversation history changes on every turn and will never hit the cache. Putting it before the cache breakpoint will invalidate the cache on every request.
Cache miss rate
Monitor your cache hit rate in production. A low hit rate — below 80% — indicates that something in your cacheable prefix is varying across requests when it should not. Common causes: a timestamp or request ID accidentally included before the cache breakpoint, a whitespace character that varies by environment, a Unicode normalization inconsistency. These are subtle bugs that the cache hit rate makes visible.
cache_read_input_tokens and cache_creation_input_tokens in the usage object. Log these per request and track the ratio over time. If cache creation tokens are unexpectedly high, investigate what is changing in the prefix.Model routing
Not every task requires a frontier model. Routing simpler tasks to smaller, cheaper, faster models is the most impactful latency and cost optimization after prompt caching — and unlike prompt caching, it can reduce output token costs as well.
When smaller models work
Classification and routing. Deciding which category an input belongs to, or which handler should process a request. A small model performs classification at near-frontier accuracy for most categories, at a fraction of the cost and latency.
Simple extraction. Pulling well-defined fields from structured or semi-structured text. The task does not require deep reasoning — it requires following instructions reliably. Small models handle this well.
Format conversion. Converting between formats — JSON to markdown, a table to prose, a list to a narrative — does not require advanced reasoning. Small models do it reliably.
Short factual lookups. If the answer is in the context and the question is direct, a small model will find and state it. Reserve frontier models for tasks that require synthesis, multi-step reasoning, or generating novel content.
Latency vs capability tradeoffs
Smaller models are faster. Claude Haiku generates tokens roughly three to five times faster than Claude Sonnet. For user-facing applications where response latency matters, routing classification and simple extraction steps to a faster model reduces the total time for multi-step pipelines even if the final synthesis step still uses a frontier model.
The risk of model routing is routing complex tasks to a model that cannot handle them. Build an eval for your routing logic itself — a dataset of tasks labeled by required capability tier, and verify that your router classifies them correctly before deploying. A misrouted task that produces wrong output is more expensive than a correctly routed task on the more expensive model.
Cascade routing
A cascade router first attempts the task with a smaller model, then retries with a larger model if the output does not meet quality requirements. The quality check can be a validation rule (did the output parse as valid JSON?), a confidence signal from the model (low-confidence outputs get escalated), or a separate classifier. Cascade routing captures the cost savings of small models for the tasks they handle well while ensuring frontier-model quality on the tasks that need it.
Batch API
The Batch API is a 50% discount in exchange for asynchronous processing with a 24-hour turnaround window. It is purpose-built for workloads where latency does not matter — jobs that can run overnight, bulk processing tasks, evaluation runs, and data enrichment pipelines.
Use cases
Eval runs. An eval suite against a few hundred cases is an ideal batch job. It is latency-insensitive, the volume is predictable, and the 50% cost reduction directly reduces the overhead of running evals frequently.
Data enrichment. Classifying, tagging, or extracting fields from a large corpus of documents. If you have 100,000 documents to process and the results are not needed immediately, batch processing at half price is the obvious choice.
Offline analysis. Weekly report generation, nightly summaries, periodic content moderation sweeps. These are naturally batch workloads that do not need synchronous API responses.
Synthetic data generation. Generating training data, test cases, or eval datasets at scale. The volume is high, the latency tolerance is high, and the cost savings are significant at scale.
Throughput math
The batch API accepts up to 100,000 requests per batch. If your workload is 500,000 items, submit five batches. Results are available via polling or webhook when processing completes. The 24-hour window is a maximum — most batches complete faster, especially during off-peak hours. Plan workflows that tolerate overnight turnaround; do not design batch processing into anything that needs a sub-hour SLA.
When it does not make sense
Any user-facing task, any task with a latency SLA shorter than a few hours, any task where failures need immediate retry and correction. The batch API trades latency for cost; if latency matters, it is not the right tool.
Cost-per-unit thinking
The wrong way to think about AI costs is cost per API call. API calls are infrastructure; they have no direct meaning to your business. The right way to think about AI costs is cost per unit of value delivered — cost per ticket resolved, cost per document processed, cost per email drafted, cost per order placed. This reframes the optimization target and makes the economics visible.
Define your unit
Start by defining what the unit of value is for your system. For a support agent: a resolved ticket. For a document intelligence system: a processed document. For a code generation tool: a merged pull request or a completed task. Once you have the unit, measure the cost per unit rather than the cost per API call. This immediately reveals whether your AI system is economically viable and where the cost levers are.
Cost drivers by unit
Once you have the cost-per-unit number, break it down into its components. For a typical multi-turn agent: what fraction of cost is the system prompt (addressed by prompt caching)? What fraction is retrieval (addressed by better chunking or query routing)? What fraction is agent reasoning steps (addressed by task decomposition or model routing)? What fraction is output generation (addressed by more concise prompting)? The breakdown tells you where to optimize first.
Comparing to alternatives
Cost-per-unit also enables comparison to alternatives. If your AI support agent costs $1.50 per resolved ticket and your human support team costs $12 per resolved ticket, the economic case is clear. If the agent costs $11 and handles only 60% of tickets without escalation, the math is tighter and the improvement levers are obvious: reduce cost per ticket (optimization levers above), or increase the resolution rate (product and model improvements).
Tracking over time
Track cost-per-unit over time, alongside quality metrics. As models improve, costs typically fall — newer, better models often cost the same or less as prior models. As your system improves, the cost-per-unit should fall while quality holds or improves. If cost-per-unit is rising, something in the system — prompt bloat, increasing retry rates, growing context windows — is degrading efficiency. The metric surfaces it.