LiteLLM vs OpenRouter: Which LLM Gateway to Run

Small team, mixed providers, nobody who wants to babysit a proxy? OpenRouter. It's managed, failover works without you configuring it, and the fee at that scale is smaller than a day of anyone's time.

Healthcare, finance, or anything where prompts contain data that isn't allowed to leave your VPC? LiteLLM, self-hosted. This isn't a cost decision and the break-even math doesn't apply to it.

Spending five figures a month on inference? LiteLLM. At that point the percentage fee is a salary line and the proxy is a rounding error.

One provider, one model, one service? Neither. You need retry logic, not infrastructure. I'll come back to this, because it's the most common wrong answer.

A gateway is one endpoint in front of many providers. You send an OpenAI-shaped request, it translates and forwards. Four things come with that, and they're worth naming separately because most teams only need two of them:

One integration instead of five. Anthropic, OpenAI, Google and the open-weight hosts all have different SDKs and different quirks. A gateway collapses that into one client.

Fallback during outages. This is the one people underrate until the first bad Tuesday. Vercel published incident counts for December 2025 alone: 22 OpenAI incidents totalling 182.7 hours, and 20 Anthropic incidents totalling 184.5 hours. Not all of those were full outages, but that's a lot of degraded surface across one month. In their own production traffic, fallback routing rescued 3.5% of requests before any team wrote retry logic.

Spend controls and per-team keys. Issuing a virtual key per team, with a budget attached, is genuinely hard to retrofit later.

One place to look when something is slow. Related to but not the same as real tracing — I'd still run proper agent observability alongside it, because a gateway sees requests, not reasoning.

If your honest answer is “I just want it to stop dying on 429s,” you don't need any of this. That's client-side backoff, and I wrote up the specific case of handling Claude's 429s and overload errors without adding a hop.

The break-even. At $200/month of infrastructure, a 5.5% fee costs more than hosting once model spend passes roughly $3,600/month. That number moves fast with your setup — a genuinely production-grade LiteLLM deployment with replicas, a managed Postgres, Redis and alerting is more like $500 to $2,000/month, which pushes break-even into the five-figure range.

And the arithmetic quietly omits the expensive part. Nobody bills you for the afternoon spent debugging why the proxy's connection pool wedged. If your team has zero people who want to own a piece of infrastructure, the honest infrastructure cost isn't $200 — it's $200 plus the on-call rotation you don't have.

One footgun on the OpenRouter side: that $0.80 minimum on card top-ups means a $10 top-up carries roughly an 8% effective rate, while $500 carries the headline 5.5%. If you're testing with small top-ups and extrapolating the percentage you're seeing, you'll overestimate the fee at scale.

Every comparison of these two argues about a 5.5% fee. Meanwhile prompt caching is a 90% discount, and routing decisions are what decide whether you get it.

Caches are scoped to one provider and one model. That's the whole problem in one sentence. A router that sends turn three of a conversation somewhere other than turn two finds no cache, and the entire prefix reprices from cache-read rates — roughly a tenth of standard input — back to full input.

Run the arithmetic on a normal agent. A 20,000-token cached system prefix at $3 per million input tokens costs about $0.006 to read from cache and $0.06to process cold. That's $0.054 of avoidable cost per request. At 400 requests a day — a modest internal agent — that's roughly $650 a month of pure waste. On the same workload, the 5.5% platform fee everyone is arguing about is worth tens of dollars.

The cache decision is an order of magnitude more expensive than the gateway decision. I went through the mechanics of the discount itself in the prompt caching post; the point here is that a gateway is the layer most likely to silently destroy it.

To be fair to both products: neither breaks caching on its own. LiteLLM passes cache_controlthrough and can inject cache breakpoints for you, and OpenRouter supports provider caching. Two things cause the damage. First, letting a cost- or latency-based router move a cached multi-turn conversation between providers mid-session — the router is optimizing per-request price while destroying a 10x discount on the prefix. Second, passthrough routes: LiteLLM's own caching layer is bypassed on provider passthrough paths, so the behavior you tested on one route isn't the behavior you get on another.

The rule I'd apply: pin cached multi-turn conversations to one provider and one model for the life of the session, and give the router its freedom on stateless work — classification, extraction, one-shot generation. Then verify. Every provider returns cache-read token counts in the usage block. If that field is zero across a conversation that should be hitting cache, your gateway is the first place to look, not the last.

Both products “support failover,” and the phrase hides the actual difference. Retrying a 500 against the same provider is table stakes. Switching to a different provider when the first one is having a bad hour is a separate capability, and it's the one that matters during an incident.

OpenRouter does it by default. Its router deprioritizes providers that returned errors in the last 30 seconds and moves traffic without you writing anything.

LiteLLM gives you more control and less default. Six routing strategies ship in the box — weighted pick, latency-based, rate-limit-aware, least-busy, lowest-cost, and custom Python. That's a strictly richer toolkit, and rate-limit-aware routing in particular is something the managed option can't match on your own keys. But cross-provider fallback is config you write, deploy and test.

Which is really a question about your team, not the software. Untested failover config is indistinguishable from no failover. If nobody owns it, the default that works beats the option that could be better. If someone does own it, LiteLLM's rate-limit-aware routing will save you more grief than outage-based routing ever will — in my own stack the failures I actually hit are 429s and per-key quota exhaustion, not providers going dark.

Self-hosting wins here and it isn't close. A four-instance LiteLLM deployment adds about 2ms at the median, 8ms at P95 and 13ms at P99; drop to two instances and the median climbs to around 12ms. A managed gateway adds a network hop, and OpenRouter measures in the 40–55ms range at P50.

Now weigh that against a model that takes 2 to 30 seconds to answer. Forty milliseconds on a four-second generation is one percent. Nobody has ever noticed it. The exceptions are real but narrow: a voice agent where every millisecond of time-to-first-token is audible as a pause, and high-volume classification where you fire thousands of sub-second calls and the fixed overhead starts to dominate the request. If you're building either of those, self-host. Otherwise, don't let a 40ms number decide a piece of infrastructure.

Four questions, in this order. The first one that returns a hard yes ends the decision.

1. Can prompt data legally leave your network? If no, LiteLLM. Stop reading, the rest doesn't apply.

2. Is monthly model spend above ~$5,000? If yes, LiteLLM, and hire the ownership along with it. Below that, the fee is cheaper than the attention.

3. Does anyone on the team want to own a proxy? If no, OpenRouter, regardless of what question two said. An unowned self-hosted gateway becomes a single point of failure in front of every AI feature you have.

4. Do you actually route across providers? If no, skip both and write the retry loop. Adding a gateway to a single-provider stack buys you one more thing that can break and no capability you didn't already have.

For what it's worth, my own setup lands on managed. I run Claude as the primary model with cheaper models behind the same endpoint for long, low-stakes work — and the routing savings there are real, but small compared to what I'd lose the first week the proxy went down while I was asleep. The same instinct that makes me push agent cost work toward caching and model selection rather than infrastructure applies here: the gateway is rarely where the money is.

Cache-read tokens are non-zero in your usage logs. Check this on a real multi-turn conversation through the gateway, not a single request. It is the cheapest check on this list and the most expensive to skip.

Cached sessions are pinned to one provider and one model. Routing freedom belongs on stateless calls.

You've tested failover by actually breaking something. Revoke a key in staging and watch what happens. Config that has never fired is a hypothesis.

Per-team keys and budgets exist from day one. Retrofitting attribution after six months of shared-key traffic is miserable work.

You know what happens when the gateway itself is down. Self-hosted means it's your outage; managed means it's theirs. Either way, a direct-to-provider escape hatch is worth the hour it takes to write.