AI Gateway Tools Compared: Routing, Cost, Failover
Compare AI gateway tools on multi-provider failover, semantic caching, and spend governance so your production LLM traffic stays fast and cost-controlled.
Most teams discover they need an AI gateway the hard way: a provider outage takes down a feature at 2 AM, a batch job that should cost $40 bills $400 because someone forgot to cap token usage, or latency spikes to four seconds because every request hits the same overloaded region. A gateway sits between your application and your LLM providers, handling routing, failover, caching, and spend governance so those incidents don't repeat.
The category has matured quickly. LiteLLM, Portkey, and the cloud-native options from AWS and Azure each solve the core problem but make very different bets on where complexity should live. Pick the wrong one and you're paying a migration tax later, because your gateway ends up touching request paths, cost dashboards, and often CI pipelines.
At Laxaar, we've routed production LLM traffic through each of these options across different client workloads. This post covers what actually differs once you're past the getting-started guide.
What you'll learn
- What an AI gateway does and why it matters
- LiteLLM: the open-source workhorse
- Portkey: the developer-experience play
- Cloud-native gateways: AWS Bedrock and Azure AI Foundry
- Failover and routing compared
- Caching strategies across gateways
- Spend governance and cost controls
- How to choose for your stack
- Frequently Asked Questions
What an AI gateway does and why it matters
An AI gateway is a proxy layer that receives LLM API calls from your application and forwards them to one or more model providers, handling retries, load balancing, caching, and observability in between. Think of it as an API gateway purpose-built for the shape of LLM traffic: long-lived streaming connections, token-based billing, non-deterministic outputs, and provider-specific quirks.
Without a gateway, each application component calls providers directly. Fine at low volume. At scale, though, three problems compound. Provider reliability becomes your reliability: if OpenAI has a regional issue, your feature breaks. Cost visibility degrades to reconciling invoices per provider rather than per feature or team. And adding a new provider means touching application code everywhere instead of updating a routing rule in one place.
The real value isn't any single feature. A gateway turns provider diversity from a coordination problem into a configuration problem.
LiteLLM: the open-source workhorse
LiteLLM is a Python library and proxy server that translates the OpenAI API format into calls to over 100 model providers, from Anthropic and Google to Replicate and Together AI. You deploy it yourself (Docker, Kubernetes, or bare metal), and your application talks to it exactly as if it were the OpenAI API: same endpoint, same request shape.
# LiteLLM proxy config — litellm_config.yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: gpt-4o # same logical name, fallback provider
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
routing_strategy: least-busy
num_retries: 3
fallbacks:
- gpt-4o:
- claude-3-5-sonnet-20241022
# Your application — no changes needed, just point to your LiteLLM proxy
import openai
client = openai.OpenAI(
api_key="your-litellm-api-key",
base_url="http://your-litellm-proxy:4000"
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Summarise this document: ..."}]
)
LiteLLM's strength is breadth. It handles load balancing across multiple deployments of the same model, tracks cost per request using current provider pricing, and supports Redis-backed caching out of the box. The proxy includes a basic admin UI and exposes Prometheus metrics for your existing observability stack.
The trade-off is operational overhead. You own the infrastructure. You manage updates, scaling, and the Redis instance for caching. When a new provider API changes, you wait for a LiteLLM release or patch it yourself. For teams with existing Kubernetes infrastructure and strong DevOps capacity, that's a fair deal. For smaller teams, it's a maintenance burden that sneaks up on you.
Portkey: the developer-experience play
Portkey is a managed AI gateway with a similar routing and failover model but a fundamentally different operational philosophy. You send requests to Portkey's cloud endpoint (or a self-hosted instance if you need data residency), configure routing through a dashboard or API, and get observability, cost tracking, and version-controlled prompt management without running infrastructure.
from portkey_ai import Portkey
client = Portkey(
api_key="your-portkey-api-key",
virtual_key="openai-virtual-key", # Portkey stores your provider keys
config={
"strategy": {
"mode": "fallback"
},
"targets": [
{"virtual_key": "openai-virtual-key", "model": "gpt-4o"},
{"virtual_key": "anthropic-virtual-key", "model": "claude-3-5-sonnet-20241022"}
],
"cache": {"mode": "semantic", "max_age": 3600}
}
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What's our refund policy?"}]
)
Portkey's semantic cache is worth calling out specifically. It returns cached responses not just for exact prompt matches but for semantically similar prompts, which matters for support bots, FAQ answerers, and any workload where users rephrase the same question. The cache hit rate on a customer support workload the Laxaar team ran through Portkey was around 34%, which translated directly to cost reduction.
The managed model means Portkey handles provider API changes, SLA monitoring, and infrastructure scaling for you. The honest cost: you're trusting a third party with your provider credentials and request data. For regulated industries or workloads with strict data residency requirements, that's a real constraint. Portkey does offer a self-hosted enterprise tier, but at that point the operational simplicity advantage narrows.
Cloud-native gateways: AWS Bedrock and Azure AI Foundry
AWS Bedrock and Azure AI Foundry both include gateway-like capabilities for teams already committed to a cloud provider. Bedrock's model routing and Azure AI Foundry's model catalog offer multi-model access, IAM-based access control, and native integration with each cloud's observability, audit logging, and cost management tooling.
The value here is integration depth, not feature breadth. If your data already lives in S3 and your team manages IAM policies for everything else, routing LLM calls through Bedrock means your AI cost attribution lands in the same Cost Explorer view as your EC2 and RDS spend. Audit trails go to CloudTrail. Access policies fit the IAM model your security team already reviews.
The limitation is obvious: you're locked into one cloud provider's model catalog. Bedrock gives you Anthropic, Amazon Titan, Meta Llama, Stability AI, and a few others. Azure AI Foundry covers OpenAI, Meta, Mistral, and Azure-hosted models. Neither gives you clean access to the full cross-provider selection that LiteLLM or Portkey expose.
Failover within these platforms is also constrained to models within the same cloud. If you need to fail from Claude on Bedrock to Claude on Anthropic's direct API, you're writing custom logic. Cross-cloud failover isn't a supported pattern.
Failover and routing compared
Failover is the gateway feature that justifies the whole architecture for most teams. Here's how the options stack up on the dimensions that matter in production.
| Dimension | LiteLLM (self-hosted) | Portkey (managed) | Bedrock / Azure AI Foundry |
|---|---|---|---|
| Cross-provider failover | Yes — any provider to any provider | Yes — any provider to any provider | Limited to models within the same cloud |
| Failover trigger conditions | Error code, timeout, or custom | Error code, timeout, status codes | Provider error or quota limit |
| Load balancing strategies | Least-busy, round-robin, latency-based | Round-robin, weighted, fallback | Limited — provider-managed |
| Routing by model capability | Manual config | Config + dashboard | Model catalog selection |
| Custom routing logic | Python callbacks | Config-driven, no custom code | IAM policies and quotas only |
LiteLLM gives the most routing flexibility because it's code. You can write a Python function that inspects the request and routes based on prompt length, user tier, time of day, or any other signal. That flexibility has a price: you're maintaining routing logic in code rather than config.
Portkey's routing is config-driven, so it's auditable and changeable without a deployment. For teams where routing rules change often (tiered model access by customer plan, say), that's a meaningful advantage over code-based routing.
Caching strategies across gateways
Caching is where gateways recover their cost most visibly. LLM calls are expensive. Returning a cached response for a repeated or near-identical request is the fastest win on the cost line.
LiteLLM supports exact-match caching (Redis-backed) and semantic caching (using embedding similarity). The semantic cache requires you to configure and host an embedding model alongside your LiteLLM proxy. Setup is documented but adds another component to maintain.
Portkey ships semantic caching as a managed feature. You set a similarity threshold (0.0 to 1.0) and a TTL, and Portkey handles the embedding and lookup. The trade-off is that you're trusting Portkey's caching infrastructure and you have less control over which embedding model runs the similarity check.
Neither cloud-native option ships meaningful semantic caching. Bedrock and Azure AI Foundry have response caching for exact prompts in limited configurations but it's not a primary feature.
Our honest take: for most production workloads, exact-match caching alone recovers significant cost on anything with repeated system prompts or templated queries. Semantic caching adds more value for freeform user input but introduces a real false-positive risk: a cached response to a similar but not identical question can be wrong in subtle ways. Set your similarity threshold conservatively (0.85 or higher) and log cache hits for spot-checking.
Spend governance and cost controls
Nothing ages a gateway deployment faster than a finance surprise. Per-request cost logging is table stakes. What actually matters in production: budget caps that fire before you hit a threshold, spend attributed by team or feature rather than by provider invoice, and hard stops that don't require an on-call engineer to enforce them.
LiteLLM tracks cost per request using its own provider pricing database and exposes it via API and the admin UI. You can set budget limits per API key and configure soft alerts via webhook. The pricing database can lag provider changes, so spot-check it if you've recently changed models.
Portkey's cost tracking is real-time, attributed by virtual key (which you can map to a team, feature, or user), and visible in a dashboard without any setup. Budget guardrails with hard stops per virtual key work out of the box. For product teams that want to give individual squads an LLM budget, Portkey's model here is the most practical we've used.
Cloud-native gateways lean on the existing cloud cost infrastructure. AWS Cost Explorer and Azure Cost Management both handle LLM spend attribution if you tag your Bedrock or Azure AI requests. The tooling is mature, but it requires cloud cost management discipline that many product teams don't have yet.
How to choose for your stack
The decision comes down to three questions: how much infrastructure do you want to own, whether you need cross-cloud provider flexibility, and how much operational simplicity you're willing to pay for.
Choose LiteLLM when your team has Kubernetes infrastructure and DevOps bandwidth, you need maximum routing flexibility or custom routing logic, you want the full model provider catalogue without vendor dependency, or you're operating in an air-gapped or strict data-residency environment where managed services aren't viable.
Choose Portkey when you want to ship fast without managing infrastructure, your team benefits from a dashboard for cost attribution and routing changes, semantic caching for customer-facing features is a priority, or you need per-team budget controls without custom tooling.
Choose a cloud-native gateway when your stack is already committed to a single cloud provider, IAM-based access control and native audit logging matter more than provider diversity, or you need deep integration with existing cloud cost management workflows.
The AI automation services we build at Laxaar typically land on LiteLLM for clients with existing platform engineering teams, and Portkey for product teams that want gateway benefits without a new infrastructure component. The cloud-native options earn their place in compliance-heavy environments where centralising on one cloud is already the decision.
If you're building a custom software development project that routes LLM traffic across multiple providers, start with the simplest option that covers your failover and cost requirements. You can migrate gateways. It's annoying, but it's not catastrophic. Don't let gateway selection block shipping.
Frequently Asked Questions
Do you need an AI gateway when using only one LLM provider?
You probably don't need failover routing, but you still benefit from centralized cost tracking, per-feature attribution, and caching. Even single-provider setups see meaningful cost reduction from exact-match caching on templated prompts. If you're planning to add providers later, setting up a gateway early is far cheaper than retrofitting one after your application is calling the provider API in twenty places.
How does semantic caching affect response quality?
Semantic caching returns a stored response when the incoming prompt is similar enough to a prior prompt, according to an embedding-based similarity score. At a high threshold (0.90+), false positives are rare but cache hit rates drop. At a lower threshold (0.75), hit rates improve but you risk returning a response that was correct for a slightly different question. Log cache hits and sample them regularly, especially in the first few weeks of deployment. Don't rely on any similarity threshold being universally safe.
Can you use an AI gateway with streaming responses?
Yes, and all three options we've covered support streaming. The gateway proxies the stream directly to your client without buffering the full response first. The practical consideration is that streaming responses can't be cached in the same way as non-streaming calls. The gateway needs to see the full response to cache it, so caching works on the buffered copy, not the stream itself. Verify your gateway's specific streaming-plus-caching behavior before relying on it.
How should provider credentials be secured through a gateway?
LiteLLM reads credentials from environment variables at startup, so standard secrets management via your orchestrator (Kubernetes secrets, AWS Secrets Manager, etc.) applies. Portkey stores credentials as virtual keys on their platform; check their security documentation for encryption-at-rest and access control specifics. For cloud-native gateways, IAM roles replace credential management entirely, which is often the cleanest model if you're already on AWS or Azure. Never commit provider API keys to config files in any of these setups.
What's the operational cost of running LiteLLM at scale?
LiteLLM's proxy is a Python FastAPI application. At low to medium volume it runs fine on a small container instance. At high volume (hundreds of requests per second), you need horizontal scaling, a reliable Redis cluster for caching, and a database for cost tracking. The LiteLLM team publishes guidance on production deployments, but budget for the same operational care you'd give any stateful service. The hidden cost isn't compute. It's the engineering time to handle upgrades, monitor the proxy's own health, and stay current with provider API changes.
Choosing an AI gateway is one of the earlier infrastructure decisions that's easy to defer and annoying to fix later. If you're routing LLM traffic in production and want a second opinion on your gateway setup, talk to the Laxaar team. We've seen the failure modes, and we're happy to help you skip them.
Working on something like this?
Get a fixed scope, timeline, and price within one business day — no obligation.
