LiteLLM

What is LiteLLM?

LiteLLM is an open source LLM gateway that gives you one OpenAI-compatible API in front of 100+ model providers — OpenAI, Anthropic, Google Gemini, Amazon Bedrock, Azure, and more. You call every model with the same request format, so switching providers is a config change, not a rewrite. It ships as both a Python SDK and a standalone proxy server.

What is LiteLLM best for?

Platform teams that need to give many apps and developers governed access to lots of models from one place. It centralizes provider keys, tracks spend per key/team/model, enforces budgets, and load-balances across deployments — so you control cost and routing centrally instead of wiring each provider into every app by hand.

What can LiteLLM do?

• Call 100+ providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and more) through one OpenAI-compatible API
• Run as a Python SDK for app code, or as a proxy server for organization-wide access
• Issue virtual API keys scoped by team, project, or app, with SSO on the admin UI
• Track spend per key, user, team, and model, and set hard daily/monthly budgets that halt requests
• Load-balance, route to the lowest-cost deployment, and fall back automatically when a provider fails
• Cache responses (semantic and exact) with Redis, and log every request for audit
• Add guardrails for PII masking, prompt-injection detection, and secret detection

Where does LiteLLM fall short?

• The proxy is the heavy part. The SDK is a lightweight pip install, but the gateway expects Postgres for keys/spend and Redis for caching and rate limits — a real service to run, not a single binary.
• Some governance features are enterprise-only. The core gateway is MIT-licensed and free, but JWT auth, RBAC, SCIM provisioning, and audit logs sit behind a paid enterprise license.
• Broad provider coverage means uneven edges. Mapping 100+ APIs to one schema means newer or unusual provider features (certain streaming, tool-calling, or vision quirks) can lag or need provider-specific handling.

Is LiteLLM free?

Yes — the core gateway and SDK are open source under the MIT license and free to self-host, with no per-token markup: you bring your own provider keys. A paid enterprise tier adds SSO/JWT auth, RBAC, SCIM, audit logs, and dedicated support (pricing on request). There is no managed cloud — you host it yourself.

What does LiteLLM replace?

LiteLLM is a self-hosted stand-in for hosted routing layers. It’s an OpenRouter alternative that keeps your own provider keys instead of paying a token markup, and it removes single-cloud lock-in from Amazon Bedrock and Azure OpenAI Service by putting every provider — including local backends like Ollama and vLLM — behind one API you control.

FAQ

Is LiteLLM open source? Yes. The gateway and Python SDK are MIT-licensed on GitHub. An optional enterprise tier adds paid features on top, but the core is fully open and free.

Can I self-host LiteLLM for free? Yes. Run the SDK with pip install litellm, or deploy the proxy via Docker, Kubernetes (Helm), or Terraform. The free tier includes routing, virtual keys, budgets, spend tracking, and caching.

Is LiteLLM a good OpenRouter alternative? If you want the same unified-API convenience but with your own provider keys and no per-token markup — and you’re willing to run the service — yes. OpenRouter is a hosted marketplace; LiteLLM is infrastructure you operate.

What do I need to run LiteLLM? The SDK needs only Python. The proxy server needs a container runtime plus Postgres (keys and spend) and Redis (caching and rate limits) for full functionality, and your own API keys for each provider you route to.