Caching Strategies for Faster, Cheaper LLM Applications
Cut LLM latency and token spend with semantic caching and prompt-prefix caching — key AI infrastructure patterns every production AI team should know.
LLM API bills have a habit of arriving much larger than expected. The culprit is almost never a single expensive request. It's the pattern of sending the same prompt context repeatedly across thousands of calls that individually look cheap. A product with a 2,000-token system prompt, 500-token conversation history, and 100-token user query is billing you for 2,600 input tokens every single turn. Multiply that by 50,000 daily active users and the math stops being theoretical.
Caching is the obvious fix, but LLM caching is not the same as caching a database query. Exact-match key lookups work only for a narrow class of prompts. Most user inputs are phrased differently each time, which means naive caching has near-zero hit rates in practice. Two techniques actually move the needle: prompt-prefix caching and semantic caching. They work on different assumptions, and each requires you to accept a specific kind of fuzziness.
Building robust AI infrastructure means treating caching as a first-class design concern, not a bolt-on optimization. The teams at Laxaar who see the largest cost and latency reductions are the ones who architect caching into their LLM pipelines from the start, not after the bill arrives.
What you'll learn
- Why exact-match caching fails for LLM apps
- How prompt-prefix caching works and when to use it
- Semantic caching: the fuzzy-match approach
- Comparing caching strategies across key dimensions
- Implementing a production semantic cache
- Response caching and TTL design
- Common mistakes that kill cache hit rates
- Frequently Asked Questions
Why exact-match caching fails for LLM apps
Exact-match caching is straightforward: hash the full prompt string, store the response, return the cached response when the same hash appears again. It works perfectly for REST APIs with stable, repeated request patterns. It fails for LLM apps because user queries are almost never byte-for-byte identical across sessions.
"What's the refund policy?" and "How do I get a refund?" are semantically equivalent for a customer support agent, but their hashes are completely different. An exact-match cache treats them as unrelated requests and pays full inference cost for both.
The practical hit rate for naive exact-match caching across a typical conversational AI product hovers under 5%. You're paying for cache infrastructure and getting almost nothing back. The problem isn't caching. "Same request" means something different for language than it does for HTTP.
Two patterns solve this: prefix caching (targeting the repeated static portions of prompts) and semantic caching (targeting equivalent user queries regardless of phrasing). They're not alternatives. They're complementary layers that address different parts of the token cost.
How prompt-prefix caching works and when to use it
Prompt-prefix caching is a feature offered by major LLM providers (Anthropic calls it "prompt caching," OpenAI calls it "cached input tokens") that detects when the beginning of a prompt is identical to a recently processed prompt and reuses the KV-cache state from that prior computation rather than reprocessing the tokens from scratch.
The mechanics: when you send a 4,000-token system prompt followed by a 200-token user message, the provider's infrastructure can recognize that the 4,000 tokens haven't changed since the last request and skip their attention computation. You're billed at a reduced rate for cached input tokens, typically 50–90% less than standard input token pricing.
import anthropic
client = anthropic.Anthropic()
SYSTEM_PROMPT = """You are a helpful assistant for Acme Corp...
[4000 tokens of product documentation, policies, and context]
"""
def chat(user_message: str, conversation_history: list[dict]) -> str:
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}, # Mark for prefix caching
}
],
messages=conversation_history + [{"role": "user", "content": user_message}],
)
return response.content[0].text
Prefix caching delivers the most value when:
- Your system prompt is long (over 1,024 tokens for Anthropic, over 1,024 for OpenAI).
- The same system prompt is reused across many requests, which is true for virtually all production applications.
- You have multi-turn conversations where earlier turns are stable and only the latest message changes.
The key constraint: the cached prefix must be byte-identical. If you interpolate dynamic values into your system prompt (current date, user name, account tier), each unique interpolation creates a separate cache entry and degrades hit rates. Move dynamic context out of the system prompt and into the user turn, or into a separate message, so the static system prompt can cache cleanly.
At Laxaar, prompt-prefix caching is the first optimization we apply when building generative AI applications for clients. It requires almost no code change and delivers immediate cost reduction on every long-system-prompt application.
Semantic caching: the fuzzy-match approach
Semantic caching embeds user queries as vectors and runs approximate nearest-neighbor search against a store of prior query-response pairs. When a new query lands close enough to a cached one (above a configurable similarity threshold), the cached response is returned without touching the LLM at all.
This directly addresses the exact-match problem. "What's the refund policy?" and "How do I get a refund?" will produce similar embeddings. If the first query has a cached response, the second query can retrieve it without a new inference call.
The tradeoff is honest: you're betting that semantically similar queries deserve the same answer. That's true for factual Q&A over stable content. It's false for queries where small phrasing differences carry real intent differences, or where the answer depends on dynamic context the cache doesn't see.
import openai
import numpy as np
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, SearchRequest
import hashlib, uuid, json
openai_client = openai.OpenAI()
qdrant = QdrantClient(":memory:")
qdrant.create_collection(
collection_name="semantic_cache",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)
SIMILARITY_THRESHOLD = 0.92 # Tune this per use case
def embed(text: str) -> list[float]:
resp = openai_client.embeddings.create(
model="text-embedding-3-small",
input=text,
)
return resp.data[0].embedding
def cache_lookup(query: str) -> str | None:
query_vector = embed(query)
results = qdrant.search(
collection_name="semantic_cache",
query_vector=query_vector,
limit=1,
score_threshold=SIMILARITY_THRESHOLD,
)
if results:
return results[0].payload["response"]
return None
def cache_store(query: str, response: str):
vector = embed(query)
qdrant.upsert(
collection_name="semantic_cache",
points=[PointStruct(
id=str(uuid.uuid4()),
vector=vector,
payload={"query": query, "response": response},
)],
)
def llm_with_semantic_cache(query: str) -> str:
cached = cache_lookup(query)
if cached:
return cached # Cache hit — no LLM call
# Cache miss — call the LLM
resp = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": query}],
)
response_text = resp.choices[0].message.content
cache_store(query, response_text)
return response_text
The similarity threshold is the most consequential configuration decision. Too low (e.g., 0.80) and you'll return cached answers for queries that meant something different, producing wrong results. Too high (e.g., 0.99) and your hit rate collapses back toward zero. The right value varies by domain: factual support bots can tolerate 0.90–0.93; creative or personalized applications should be more conservative at 0.95+.
Comparing caching strategies across key dimensions
| Strategy | What it caches | Hit rate driver | Latency saving | Cost saving | Best for |
|---|---|---|---|---|---|
| Exact-match | Full prompt string | Identical repeated requests | Very high (when hit) | High (when hit) | Batch jobs, deterministic queries |
| Prompt-prefix (provider) | Static prompt prefix tokens | Long, reused system prompts | Moderate | High (50–90% on prefix tokens) | All production LLM apps |
| Semantic cache | User query intent | Query volume and topic clustering | High (when hit) | High (when hit) | Q&A, support, search |
| Response CDN cache | Full response content | Identical outputs at scale | Very high | High | Public, non-personalized content |
These strategies compose. A production LLM application should layer all applicable approaches: prefix caching enabled at the provider level for system prompt savings, semantic caching at the application layer for user query deduplication, and response caching for any outputs that are truly static.
Implementing a production semantic cache
A toy semantic cache works in a notebook. A production semantic cache needs namespace isolation, TTL-based expiry, cache invalidation hooks, and observability. Here's what each of those means in practice.
Namespace isolation means scoping cache lookups to a context boundary. A cached answer for "what's the return policy?" should only be returned for queries in the same product domain, not across unrelated tenants or applications. Store a namespace key in your cache metadata and filter on it at query time.
TTL-based expiry is non-negotiable for any cache over dynamic content. A semantic cache for a customer support bot backed by a product knowledge base should expire entries when the underlying documentation changes. Without expiry, you'll serve cached answers about old policies long after they've changed. The failure will be silent.
Observability means tracking cache hit rate, similarity score distributions, and latency delta between cache hits and misses. Without this data you can't tune the similarity threshold or know whether the cache is actually helping.
from datetime import datetime, timedelta
def cache_store_with_ttl(query: str, response: str, namespace: str, ttl_hours: int = 24):
vector = embed(query)
expires_at = (datetime.utcnow() + timedelta(hours=ttl_hours)).isoformat()
qdrant.upsert(
collection_name="semantic_cache",
points=[PointStruct(
id=str(uuid.uuid4()),
vector=vector,
payload={
"query": query,
"response": response,
"namespace": namespace,
"expires_at": expires_at,
},
)],
)
def cache_lookup_with_ttl(query: str, namespace: str) -> str | None:
query_vector = embed(query)
results = qdrant.search(
collection_name="semantic_cache",
query_vector=query_vector,
query_filter={"must": [{"key": "namespace", "match": {"value": namespace}}]},
limit=1,
score_threshold=SIMILARITY_THRESHOLD,
)
if not results:
return None
payload = results[0].payload
if payload["expires_at"] < datetime.utcnow().isoformat():
return None # Expired — treat as cache miss
return payload["response"]
One architectural note: embedding inference has its own latency cost. For the semantic cache lookup to be faster than a direct LLM call, the embedding call plus vector search must be cheaper and faster than the LLM inference. That's almost always true (embedding models run in under 50ms and ANN search is sub-10ms), but it means you shouldn't use a slow or rate-limited embedding endpoint as your cache key generator.
Response caching and TTL design
Semantic and prefix caching reduce what you send to the LLM. Response caching stores complete LLM outputs and returns them without any LLM call when conditions match. It's the most aggressive form of caching and appropriate for a narrower set of use cases.
Response caching works well for:
- Static FAQ responses where the answer is stable and not personalized. Cache the full response for 24–48 hours.
- Batch summarization jobs where the same source document gets summarized repeatedly. Cache by document hash.
- Public-facing explanations of fixed concepts or product features that don't change with user context.
Response caching breaks down for personalized outputs, real-time data lookups, or any response where user-specific context meaningfully changes the answer. Returning a cached response to the wrong user (or a stale response after the underlying data changed) is harder to debug than a slow but fresh LLM call.
TTL design is the most domain-specific part of caching strategy. A reasonable starting framework:
- Product documentation answers: 6–24 hours (expire on doc update)
- General factual Q&A: 48–72 hours
- Dynamic data summaries: under 1 hour, or cache-by-version
- Personalized recommendations: do not cache, or cache per user with sub-1-hour TTL
Common mistakes that kill cache hit rates
Injecting dynamic values into the system prompt. Date strings, user IDs, account tiers in the system prompt create a unique cache entry per interpolation. The prefix cache can't help you. Move dynamic context to the user turn.
Setting the similarity threshold too aggressively. A threshold of 0.99 is essentially exact-match. You're paying for embedding infrastructure and getting almost no benefit. Start at 0.90 and raise it only if you observe incorrect cache hits.
Not scoping the cache namespace per domain or tenant. Without namespacing, cached answers bleed across unrelated query populations. A support bot for product A returns cached answers that were written for product B's policies.
Ignoring cache invalidation. A semantic cache that doesn't expire is a liability. Every change to your knowledge base, product documentation, or policy creates a window where the cache returns outdated information. Build cache invalidation into your content update workflow from day one.
Caching before you've measured. Before building a semantic cache, pull your actual query logs and cluster them by topic. High query diversity with low repetition (creative writing tools, code generation for novel tasks) means caching will give you a poor return on engineering investment. Cache where the patterns are, not everywhere.
For teams building out AI infrastructure or custom software with embedded LLM capabilities, the caching layer is where a 30–60% cost reduction lives. It's not glamorous engineering, but the economics are hard to argue with.
Frequently Asked Questions
Does prompt-prefix caching require any special configuration?
Most providers make it opt-in either through a flag or a specific cache-control annotation in the request (as in Anthropic's API). The prefix must be byte-identical across requests for the cache to activate, and there's typically a minimum prefix length. Check your provider's documentation for the current token minimum and supported model list, since availability varies by model tier.
What similarity threshold should we start with for semantic caching?
Start at 0.92 for most factual Q&A applications. Run your cache in shadow mode for a week, logging both the retrieved cached response and what the LLM would have returned, then compare them. If you see divergence, raise the threshold. Hit rate below 15% with a well-clustered query set? Lower it slightly. There's no universal right answer; the threshold is a product quality dial, not just a performance knob.
Can we use pgvector instead of a dedicated vector store for semantic caching?
Yes, and it's a good starting point. Postgres with the pgvector extension gives you ANN search alongside the relational columns you need for TTL, namespace filtering, and metadata storage. A purpose-built vector database will handle higher query throughput with better latency at scale, but for most teams shipping their first LLM product, pgvector is a pragmatic way to avoid adding another service to the stack.
How do we know if our caching layer is actually working?
Track three metrics: cache hit rate (percentage of queries served from cache), latency p50 and p95 split by cache hit vs miss, and cost per query split by cache hit vs miss. If hit rate is above 20% and latency on cache hits is under 100ms compared to 1–3 seconds for LLM inference, the cache is doing its job. Most teams find hit rates between 25–45% once the application has been live for a few weeks and the cache has warmed.
Does semantic caching work for multi-turn conversations?
With care. Caching the full conversation history as the cache key defeats the purpose: every conversation is unique. The pattern that works is caching at the individual turn level for self-contained questions within a conversation, or caching retrieval results (in a RAG system) rather than final responses. For deeply contextual conversational queries, semantic caching is usually not the right tool; prefix caching of the stable context window is more reliable.
What's the risk of returning a wrong cached response?
It's real and worth designing against. The mitigation strategies are: a higher similarity threshold, namespace isolation to prevent cross-domain cache hits, short TTLs over dynamic content, and logging cache hits alongside user feedback signals so you can detect silent quality degradation. Some teams add a lightweight LLM-judge step that scores whether the cached response actually answers the new query before returning it. This adds latency but reduces incorrect hits for safety-sensitive applications.
Working on an LLM product and trying to get token costs under control before they scale out of hand? The Laxaar team builds production AI infrastructure including caching, routing, and observability layers for teams who need their AI applications to be both fast and economical at scale. Explore our AI engineering services or get in touch with a project quote.
Working on something like this?
Get a fixed scope, timeline, and price within one business day — no obligation.
