LLM Cost Optimization Strategies for AI Engineering
Cut LLM spend with AI engineering techniques that actually move the needle: model routing, prompt caching, and context trimming beat chasing cheaper per-token rates.
LLM bills have a way of arriving before anyone budgeted for them. A team ships a prototype, usage grows, and the monthly invoice triples. The instinctive response is to ask the provider for a discount or hunt for a cheaper model. That's the wrong first move.
Most overspend on LLM infrastructure comes from three fixable patterns: sending every request to the largest model regardless of complexity, re-computing prompts that haven't changed since the last call, and stuffing context windows with tokens the model doesn't need. None of those problems are solved by a lower per-token rate. They're solved by architecture.
Solid AI engineering practice treats LLM spend the way mature backend teams treat database queries: profile first, identify the waste, fix the structure. The per-token rate is a multiplier on whatever waste is already present. Fix the waste and the rate stops mattering as much.
What you'll learn
- Why per-token rate is the wrong place to start
- Model routing: match task complexity to model tier
- Prompt caching: stop paying to re-read the same text
- Context trimming: only send what the model needs
- Batching and async request patterns
- Semantic caching for repeated queries
- Comparing cost levers side by side
- Building a cost instrumentation layer
- Frequently Asked Questions
Why per-token rate is the wrong place to start
Per-token pricing is the headline number vendors compete on, and it gets most of the attention in cost discussions. It deserves the least.
Here's the math. If your application is calling gpt-4o for tasks that a smaller model handles correctly 95% of the time, you might be spending 10x more per token than necessary on the majority of your traffic. Negotiating a 15% discount on that rate saves you 15% on a number that could have been 90% smaller with routing.
The same logic applies to re-sent context. A system prompt that's 2,000 tokens long, sent with every request in a chatbot receiving 100,000 daily calls, costs hundreds of dollars a day just for the static portion. Prompt caching can drop that line item to near zero.
The real cost drivers in most AI engineering systems:
- Model selection: using a frontier model for classification, summarization, or extraction tasks that a mid-tier model handles accurately.
- Context repetition: re-sending system prompts, retrieved documents, or conversation history that hasn't changed.
- Context bloat: including documents, examples, or history items that don't improve the response for a given query.
- Retry waste: retry loops on soft failures that burn 2-3x the tokens of a successful first call.
Fix those four first. Then revisit pricing.
Model routing: match task complexity to model tier
Model routing is the practice of classifying each incoming request and sending it to the cheapest model capable of handling it correctly. It's the single highest-leverage cost lever in most production AI systems.
The operational insight: LLM tasks form a natural complexity hierarchy. Extraction, classification, and short-form summarization are well within the capabilities of smaller, cheaper models. Multi-step reasoning, nuanced instruction following, and long-form generation benefit from larger models. Routing correctly means you're paying frontier prices only for frontier tasks.
from enum import Enum
import re
class RequestTier(Enum):
SIMPLE = "simple" # classification, extraction, short Q&A
MEDIUM = "medium" # summarization, structured generation
COMPLEX = "complex" # multi-step reasoning, long-form synthesis
MODEL_MAP = {
RequestTier.SIMPLE: "gpt-4o-mini",
RequestTier.MEDIUM: "gpt-4o-mini",
RequestTier.COMPLEX: "gpt-4o",
}
def classify_request(prompt: str, context_tokens: int) -> RequestTier:
# Simple heuristics — replace with a lightweight classifier in production
if context_tokens > 8000:
return RequestTier.COMPLEX
lowered = prompt.lower()
simple_signals = ["classify", "extract", "is this", "true or false", "yes or no"]
complex_signals = ["analyze", "compare", "write a", "explain in detail", "step by step"]
if any(s in lowered for s in complex_signals):
return RequestTier.COMPLEX
if any(s in lowered for s in simple_signals):
return RequestTier.SIMPLE
return RequestTier.MEDIUM
def route_request(prompt: str, context_tokens: int) -> str:
tier = classify_request(prompt, context_tokens)
return MODEL_MAP[tier]
The classification logic above is intentionally simple. In practice, a lightweight embedding-based classifier trained on your own labeled request data will outperform keyword heuristics. The inference cost of that classifier is tiny compared to the savings it generates.
One real trade-off: routing introduces latency from the classification step and operational complexity from maintaining multiple model integrations. For low-volume applications, a single mid-tier model is often cheaper overall once you account for engineering time. Routing pays off at meaningful scale: roughly 50,000 or more requests per day where the cost differential justifies the plumbing. On the AI engineering projects the Laxaar team has shipped, routing alone has reduced model spend by 60% or more for mixed-complexity workloads.
Prompt caching: stop paying to re-read the same text
Prompt caching is a provider-level feature that charges reduced rates (or nothing) for the portion of a prompt that hasn't changed since the last request with the same prefix. Both Anthropic and OpenAI support versions of this; the mechanics differ slightly but the economics are similar.
The typical candidate for caching is the system prompt. If your agent has a 3,000-token system prompt that describes tools, persona, and instructions, and that prompt is identical across all requests from a given user in a session, you're paying full price to re-read it on every call. Prompt caching makes the static prefix cheap after the first read.
import anthropic
client = anthropic.Anthropic()
SYSTEM_PROMPT = """
You are a technical support agent for Acme Software. You have access to the following tools...
[3000 tokens of tool descriptions, instructions, and context]
"""
def chat_with_caching(user_message: str, conversation_history: list[dict]) -> str:
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}, # mark for caching
}
],
messages=conversation_history + [
{"role": "user", "content": user_message}
],
)
return response.content[0].text
The discipline here is structure. Place the static, cacheable content at the top of your prompt (system instructions, tool definitions, reference documents that don't change per request) and the dynamic content at the bottom (user message, retrieved context specific to this query). Providers cache prefix matches. Vary the cached portion and you break the cache hit.
Retrieved documents in RAG pipelines are another major caching candidate. If your retriever returns the same top-k documents for semantically similar queries, and you're re-sending those full documents with each request, caching the document block saves a significant fraction of input tokens on repeat queries.
Context trimming: only send what the model needs
Context trimming is the practice of actively removing tokens from the prompt that don't improve the model's response for a specific request. It's the most direct way to reduce input token spend, and it's also the optimization most teams skip because it requires understanding what context actually matters.
Three categories of context bloat to check first:
Stale conversation history. Multi-turn chat applications often append every prior turn to the context. By turn 20, the first 15 turns may be irrelevant to the current question. Sliding window strategies (keep the last N turns) and summarization strategies (compress older history into a compact summary) both reduce token count without losing meaningful context.
Retrieved documents that don't match the query. RAG pipelines retrieve k documents; not all k are equally relevant. A relevance threshold filter (drop any retrieved chunk below a minimum similarity score) keeps context tight. In our testing, dropping chunks below cosine similarity 0.75 from a well-tuned retriever has negligible impact on response quality and cuts average context length by 20-40%.
Redundant few-shot examples. If your prompt includes 10 examples to demonstrate output format, but 3 examples achieve the same format reliability, the other 7 are dead weight.
from dataclasses import dataclass
@dataclass
class RetrievedChunk:
content: str
score: float
def trim_context(
history: list[dict],
retrieved_chunks: list[RetrievedChunk],
max_history_turns: int = 6,
min_chunk_score: float = 0.72,
) -> tuple[list[dict], list[str]]:
# Keep only the most recent turns
trimmed_history = history[-max_history_turns * 2:] # each turn = user + assistant
# Drop low-relevance chunks
relevant_chunks = [
chunk.content for chunk in retrieved_chunks
if chunk.score >= min_chunk_score
]
return trimmed_history, relevant_chunks
Context trimming has an honest trade-off: you can trim too aggressively. Dropping relevant history causes the model to repeat questions or miss context the user already provided. Set trim thresholds empirically against your eval set, not by intuition.
Batching and async request patterns
Batching is a technique where multiple independent LLM requests are grouped and submitted together, often at a reduced rate. OpenAI's Batch API offers 50% cost reduction for requests that don't need real-time responses. Anthropic offers similar asynchronous processing at lower rates.
The practical constraint: batching only applies to tasks where latency tolerance is high. Background processing (nightly report generation, bulk classification, offline document processing) is a natural fit. Interactive user-facing applications aren't.
import openai
import json
client = openai.OpenAI()
def submit_batch_classification(texts: list[str], batch_id_prefix: str) -> str:
requests = [
{
"custom_id": f"{batch_id_prefix}-{i}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "Classify the sentiment as positive, negative, or neutral."},
{"role": "user", "content": text},
],
"max_tokens": 10,
},
}
for i, text in enumerate(texts)
]
# Write to JSONL for batch submission
with open("batch_requests.jsonl", "w") as f:
for req in requests:
f.write(json.dumps(req) + "\n")
batch_file = client.files.create(
file=open("batch_requests.jsonl", "rb"),
purpose="batch",
)
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
return batch.id
The 50% discount on batch APIs is substantial for high-volume background work. If you have any offline processing pipeline currently using real-time endpoints, switching it to batch is often the fastest path to a meaningful bill reduction.
Semantic caching for repeated queries
Semantic caching is an application-level pattern where you store previous LLM responses and return them for new queries that are semantically similar, without calling the model at all. The architecture: embed each incoming query, search a cache of prior query embeddings, and return the cached response if similarity exceeds a threshold.
This is distinct from prompt caching (a provider feature for static prefixes). Semantic caching happens before the API call and avoids it entirely for cache hits.
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, SearchParams
import hashlib
import time
model = SentenceTransformer("all-MiniLM-L6-v2")
cache_client = QdrantClient(":memory:")
cache_client.create_collection(
"query_cache",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
SIMILARITY_THRESHOLD = 0.92 # tune per application
def cache_lookup(query: str) -> str | None:
vec = model.encode(query).tolist()
results = cache_client.search(
collection_name="query_cache",
query_vector=vec,
limit=1,
search_params=SearchParams(hnsw_ef=64),
)
if results and results[0].score >= SIMILARITY_THRESHOLD:
return results[0].payload["response"]
return None
def cache_store(query: str, response: str):
vec = model.encode(query).tolist()
cache_client.upsert(
collection_name="query_cache",
points=[PointStruct(
id=hashlib.md5(query.encode()).hexdigest()[:16],
vector=vec,
payload={"query": query, "response": response, "stored_at": time.time()},
)],
)
Semantic caching works best in applications with predictable, repetitive query patterns: FAQ bots, document Q&A systems, support agents. It's less effective for highly variable creative or analytical tasks where every query is genuinely distinct.
The threshold tuning is important. A threshold of 0.92 is conservative (only near-duplicate queries hit the cache). A threshold of 0.80 returns more cache hits but risks returning slightly mismatched responses. Validate the threshold against real query pairs from your application logs before going live.
Comparing cost levers side by side
| Technique | Typical Savings | Latency Impact | Engineering Effort | Best Fit |
|---|---|---|---|---|
| Model routing | 60-85% on routed traffic | Adds classifier latency | Medium | High-volume mixed-complexity apps |
| Prompt caching | 50-90% on static prefix tokens | None (or slightly lower) | Low | Any app with static system prompts |
| Context trimming | 20-50% on input tokens | None | Low to medium | RAG apps, multi-turn chat |
| Batch API | 50% flat | High (hours, not ms) | Low | Offline processing pipelines |
| Semantic caching | 30-70% cache hit rate | Lower for hits | Medium | FAQ, support, repetitive queries |
The highest-ROI combination for most AI engineering teams is prompt caching plus context trimming. Both are low effort and carry no meaningful latency trade-off. Model routing is the next layer once you have enough traffic volume to justify the classifier maintenance.
Building a cost instrumentation layer
None of these techniques can be tuned without measurement. Before optimizing, instrument your LLM calls to capture token usage per request, model used, cache hit/miss status, and estimated cost. Without this data, you're guessing at where the spend is.
import time
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class LLMCallRecord:
request_id: str
model: str
prompt_tokens: int
completion_tokens: int
cached_tokens: int
latency_ms: float
estimated_cost_usd: float
cache_hit: bool = False
tags: dict = field(default_factory=dict)
# Approximate pricing per 1M tokens (update as providers change rates)
PRICING = {
"gpt-4o": {"input": 2.50, "output": 10.00, "cached_input": 1.25},
"gpt-4o-mini": {"input": 0.15, "output": 0.60, "cached_input": 0.075},
"claude-sonnet-4-5": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
}
def estimate_cost(model: str, prompt_tokens: int, completion_tokens: int, cached_tokens: int = 0) -> float:
prices = PRICING.get(model, {"input": 1.0, "output": 3.0, "cached_input": 0.5})
input_cost = ((prompt_tokens - cached_tokens) / 1_000_000) * prices["input"]
cached_cost = (cached_tokens / 1_000_000) * prices["cached_input"]
output_cost = (completion_tokens / 1_000_000) * prices["output"]
return input_cost + cached_cost + output_cost
Run a weekly cost breakdown by model, by feature area, and by request type. The distribution is almost always more concentrated than expected. A small fraction of request patterns drives the majority of spend, and those are the ones worth optimizing first.
At Laxaar, our standard practice when auditing an existing AI system is to instrument first, spend a week collecting data, then prioritize the cost levers by impact. Teams that skip instrumentation end up optimizing the wrong things and are surprised when the bill barely moves.
The Laxaar team builds AI infrastructure across a range of production use cases. If you're scaling an LLM-powered application and costs are outpacing value, we can identify where the spend is concentrated and which fixes will move the number. Our custom software development practice includes cost-aware architecture review for AI systems.
Frequently Asked Questions
Should we switch to a cheaper model first or optimize the architecture?
Optimize the architecture first. Switching to a cheaper model without fixing context bloat or routing means the cheaper model processes the same unnecessary tokens, so you save less than expected and often introduce quality regressions that are hard to attribute. Fix prompt caching and context trimming, measure the result, and then re-evaluate whether a model switch is still needed. Often it isn't.
How do we know which requests are safe to route to a smaller model?
Build an offline evaluation set from real production requests, labeled with the quality threshold your application needs. Run both models over this set and measure accuracy, format compliance, or whatever metric matches your use case. The requests where the smaller model meets the threshold are safe to route. Don't rely on intuition. The boundary is task-specific and the only honest way to find it is empirical testing.
What's a realistic cost reduction expectation from these techniques combined?
Teams that implement model routing, prompt caching, and context trimming together typically see 50-80% reductions in LLM spend without measurable quality loss. The range is wide because it depends heavily on the current baseline. An application that sends every request to a frontier model with a bloated 10,000-token context and no caching has much more room to improve than one that already uses a mid-tier model with lean prompts. Instrument first. Your actual baseline tells you more than any benchmark.
Does semantic caching risk returning stale or wrong answers?
Yes, if the threshold is too low or the application domain changes. Responses cached when the underlying data was different may become incorrect over time. Mitigation strategies: add a TTL so cached responses expire, include a version tag in the cache key when your reference data changes, and track cache hit rates vs. user feedback signals to catch silent staleness. For applications where correctness is non-negotiable on every response, semantic caching should apply only to stable factual queries, not dynamic or time-sensitive ones.
How does prompt caching interact with streaming responses?
Prompt caching is a server-side feature applied to the input tokens before generation begins. It's fully compatible with streaming: the cache hit reduces the cost of processing the input prefix, and the streamed output tokens are billed normally. Enabling prompt caching doesn't change how you handle streamed responses in your client code; it only changes what you're charged for on the input side.
Building a production LLM application and watching costs climb? The Laxaar team has instrumented and optimized AI systems across generative AI development and AI automation services projects. Reach out through our contact page and we'll help you find where the spend is and which levers to pull first.
Working on something like this?
Get a fixed scope, timeline, and price within one business day — no obligation.
