Geekflare
Open coreMITTested by Geekflare

LiteLLM review

Python SDK and proxy server that puts 100+ LLM providers behind one OpenAI-format API, with virtual keys, budgets and spend tracking

Website GitHub Docs
Data refreshedMethodologyHow we tested
Founder of Geekflare

Chandan Kumar is a founder of Geekflare. He is a technology enthusiast and entrepreneur who loves to help businesses and people around the world. Chandan has worked at BNP Paribas, Citibank, Deutsche Bank, Motorola, and HP. He’s got an in-depth understanding of business software and IT infrastructure.

Verdict

LiteLLM covers more providers and endpoints than most gateways, and its free proxy already handles keys, team budgets, spend tracking and fallbacks for a whole company. The catches are a Python proxy that wants 4 GB of RAM per worker while the Rust core is still in beta, SSO and audit logs behind a licence, and two compromised releases on PyPI and Docker in 2026.

Best for

Platform teams that need one API and per-team budgets across many providers

Not for

Teams that cannot pin versions and verify release artifacts before upgrading

Pros

  • One OpenAI-format API for 100+ providers, including Bedrock, Vertex AI and Azure
  • Virtual keys, team budgets and spend tracking in the free version
  • Works as an in-process Python library or a central proxy
  • Exact and semantic caching with several backends

Cons

  • Credential-stealing releases reached PyPI in March and PyPI and Docker in October 2026
  • SSO beyond 5 users, audit logs and team admin roles need an Enterprise licence
  • The Python proxy needs 4 GB of RAM per worker and the Rust core is still partial
What people say
Ask Sparkian

Is LiteLLM right for your project?

Describe what you need and AI will check the fit against this review.

Or try:

LiteLLM features and capabilities

Virtual keys with budgets

Each developer, team or app gets its own key with model access, rate limits and a hard budget, so a runaway job or leaked key stops at its cap.

Spend tracking per key and team

Every request's cost is recorded against its key, user and team, which gives finance a chargeback report without extra tooling.

Router with fallbacks

Requests can be spread across several deployments of a model and retried on another model or provider when one fails or hits a rate limit.

MCP and agent gateway

MCP servers and A2A agents can be registered once on the gateway and reached with the same virtual keys used for model calls.

Opt-in Rust core

Setting rust true on a model sends supported routes through the new Rust core and falls back to Python on any error, so you can try it one model at a time.

FallbacksSwitches to another model or provider when a call fails
Yes
Load balancingSpreads traffic across keys, deployments or providers
Yes
Budgets and limitsSpending caps or rate limits per key, user or team
YesPer key, user and team; tag and per-model key budgets need Enterprise
Cost trackingRecords spend per request and per key, user or team
Yes
CachingExact or semantic response caching
YesExact and semantic, Redis, Qdrant, Valkey, S3 or GCS
GuardrailsPII redaction, content filters or prompt-injection checks
YesPresidio and custom guardrails free; some integrations need Enterprise
MCP gatewayExposes or governs MCP servers and tools
Yes

At a glance

Type
Python SDK with a proxy server
Stack
Python / Rust
Databases
PostgreSQL, Redis
Licence terms
The SDK and proxy are MIT, but code in the enterprise/ folder needs a paid licence key in production, which gates SSO beyond 5 users, audit logs, org and team admin roles, secret managers and some guardrail integrations.
SSO
Paid only
Roles and permissions
Free edition
Audit logs
Paid only

Free vs paid editions

What you get when you self-host, and what needs a paid plan.

The MIT-licensed proxy includes virtual keys, teams, budgets, rate limits, spend tracking, fallbacks, caching, the MCP gateway and the admin UI. An Enterprise licence key on the same image adds SSO for more than 5 users, SCIM, audit logs, org and team admin roles, secret managers, tag budgets and support with SLAs. Enterprise is priced per deployment on request, with a free 30-day trial, and there is no published managed cloud plan.

Self-hosting LiteLLM

LiteLLM's production guide asks for 1 vCPU and 4 GB of RAM per proxy worker, plus PostgreSQL for keys and spend, and Redis 7 or newer once you run more than one instance.

New cloud server to ready

1 min 1 s

Memory in use at idle

642 MB of 4 GB

Median request

18.1 ms

Requests/s, 32 clients

55

Peak memory

772 MB

Version tested

1.106.2 · Oct 21, 2026

  • The quickstart compose file started the proxy and Postgres in 61 seconds. Together they used 642 MB at idle, about 700 MB of it the proxy, which still runs as Python with a Prisma query engine beside it.
  • Against the mock provider it added 18 ms per request, and with 32 clients at once it managed 55 requests a second at a median of 574 ms. That is one worker, as the production guide recommends per pod with 1 vCPU and 4 GB, so you scale by adding replicas.
  • The request to a model that doesn't exist came back from Claude Haiku 5.5, with x-litellm-attempted-fallbacks: 1 and the cost of the call in the response headers.
  • On the day of our test, LiteLLM's 1.107.0 release on PyPI and its main-stable Docker tag carried a credential stealer for a few hours. We pulled main-stable after the fix, checked the image digest against LiteLLM's notice and found the safe 1.106.2. Pin an exact version and check digests.
Config we ran (with LiteLLM's quickstart compose and Postgres)
model_list:
  - model_name: mock            # the stand-in provider for the overhead test
    litellm_params: { model: openai/mock-model, api_base: "http://mock/v1", api_key: none }
  - model_name: haiku
    litellm_params: { model: openrouter/anthropic/claude-haiku-5.5, api_key: os.environ/OPENROUTER_API_KEY }
  - model_name: broken          # a model that doesn't exist, to force a fallback
    litellm_params: { model: openrouter/anthropic/claude-model-that-does-not-exist, api_key: os.environ/OPENROUTER_API_KEY }
router_settings:
  fallbacks: [{ "broken": ["haiku"] }]
OutputPassed1.106.2 · fell back to Claude Haiku 5.5 · 1.0 s
Request: POST /v1/chat/completions with model "broken", whose upstream model doesn't exist; asked "In one sentence, what does an LLM gateway do?" with max_tokens 60
x-litellm-model-group: haiku
x-litellm-attempted-fallbacks: 1
x-litellm-response-cost: 3.25e-05

{
  "model": "anthropic/claude-haiku-5.5",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "An LLM gateway is a proxy layer that sits between applications and one or more large language model providers, centralizing access to them and handling concerns like routing, authentication, rate limiting, cost t"
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 60,
    "total_tokens": 85,
    "cost": 3.25e-05
  }
}

What you can do with LiteLLM

Company-wide LLM access

Give every team a virtual key with its own models and monthly budget, with SSO through your identity provider on Enterprise.

Provider failover for a product

Route an app's calls across OpenAI, Azure and Bedrock deployments and fall back automatically when one is down or rate limited.

Coding agents on any model

Point Claude Code, Codex or Cursor at the gateway so every agent call is logged, costed and capped per developer.

LiteLLM alternatives

BifrostGo gateway that puts many LLM providers behind one OpenAI-compatible API, with failover, virtual keys, budgets, semantic caching and an MCP gatewayagentgatewayRust proxy for LLM, MCP and agent-to-agent traffic, with per-key budgets, guardrails and a built-in UIApache APISIXAPI gateway on NGINX and etcd with AI plugins for LLM proxying, fallbacks, token limits, caching and guardrailsNew APISelf-hosted AI gateway with a web console for routing, quotas, billing and usage logs, built on One APIHigressAPI gateway built on Istio and Envoy, with Wasm plugins for LLM proxying, token quotas, semantic caching and MCP hostingAgent RouterKubernetes control plane for LLM and MCP traffic built on Envoy Gateway, formerly Envoy AI Gateway