Tools · SPBoucher
Menu

LiteLLM

In-depth entry

by BerriAI · Library · also known as litellm (PyPI), LiteLLM Proxy, LiteLLM AI Gateway

Open-source Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format, with routing, fallbacks, virtual keys, budgets, spend tracking and logging callbacks

MIT licenseOpen sourceFreeEnterpriseLocalSelf-hostedSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

LiteLLM, from BerriAI, is an open-source project with two faces. The Python SDK offers one completion() interface that translates OpenAI-format requests to more than a hundred providers (OpenAI, Anthropic, Google, Bedrock, Azure, Mistral, Ollama, vLLM and many more), normalizes responses and exceptions, supports streaming and includes a Router for retries, fallbacks and load balancing across deployments. Callbacks forward inputs, outputs and computed cost to observability tools such as Langfuse, MLflow and Helicone.

The Proxy, also called the AI Gateway, packages the same translation layer as an OpenAI-compatible server for a whole organization: virtual API keys with per-key, per-team and per-user budgets, spend tracking across providers, caching, centralized logging, guardrails for content filtering and PII masking, an admin UI, an MCP gateway for tools and an agent gateway for A2A traffic, plus an auto-router that picks a model per request. A Rust-based gateway core is in beta. The core is MIT-licensed; an enterprise/ directory under a separate license adds SSO/SAML, audit logs and multi-team management, sold as an enterprise offering. Deployment is Docker or CLI with a database for keys and spend.

LiteLLM is the common answer when several teams need governed access to many providers from one internal endpoint, or when an application must stay provider-portable. Running the proxy adds operational components and a hop in the request path, provider-specific features are sometimes exposed with a lag, and the pace of releases means pinning versions in production.

Capabilities

  • Unified OpenAI-format `completion()` across 100+ providers
  • Router with retries, fallbacks and load balancing
  • OpenAI-compatible proxy server with virtual keys and budgets
  • Spend tracking, caching and centralized logging
  • Guardrails for content filtering and PII masking
  • Callbacks to Langfuse, MLflow, Helicone and other observability tools
  • MCP gateway, agent (A2A) gateway and auto-router
  • Admin UI; Docker and CLI deployment; enterprise SSO and audit logs

Typical workflows

  1. Give each team a virtual key with a monthly budget on a shared gateway
  2. Fall back from one provider to another during an outage
  3. Keep an application portable by coding against the OpenAI format only
  4. Forward every call's cost to Langfuse for analysis

Strengths

  • Very broad provider coverage
  • MIT core with self-hosting
  • Governance features (keys, budgets, logging) in one place
  • Works as a library or as infrastructure

Limitations

  • Proxy adds latency and operational components
  • Enterprise features are separately licensed
  • Provider feature parity can lag new releases
  • Fast release cadence requires version pinning

Prerequisites

  • Python 3.8+ for the SDK; Docker and a Postgres database for the proxy
  • Provider API keys

Who it is for

  • Platform teams offering LLM access internally
  • Developers who need provider portability
  • Organizations tracking LLM spend per team

Alternatives & comparison

OpenRouter provides a similar unified API as a hosted service with its own billing, whereas LiteLLM is self-hosted software you run with your own provider keys. LangChain's model interface also abstracts providers but inside an application framework rather than as a gateway. Langfuse is complementary: LiteLLM commonly forwards traces and costs to it.

  • LangChain

    Open-source Python and JavaScript framework whose create_agent harness builds tool-using agents on the LangGraph runtime with swappable model providers and middleware

    Open sourceOpen source
  • Langfuse

    Open-source LLM observability and evaluation platform: tracing, LLM-as-judge and human evals, prompt management, datasets and cost dashboards, self-hostable or as Langfuse Cloud

    Open sourceOpen source
  • OpenAI API

    OpenAI's hosted developer platform: Responses and Chat Completions APIs, Realtime voice, embeddings, images, audio, fine-tuning, evals and managed agents

    ProprietaryUsage-based
  • OpenRouter

    Unified API gateway to hundreds of models from many providers with automatic fallbacks, credit-based pass-through billing, BYOK, free models and data-policy routing

    ProprietaryFreemium
Compare LiteLLM with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

LangSmith

LangChain, Inc. · Prompting, evaluation & observability

LangChain's hosted platform for tracing, evaluating and deploying LLM applications and agents, with prompt tooling, an LLM gateway and cloud, hybrid or self-hosted options

ProprietaryFreemiumPaidSelf-hostedNov 2, 2026

Weights & Biases

CoreWeave · Training & fine-tuning

ML experiment tracking, sweeps, artifacts and model registry plus Weave for LLM tracing and evaluation, now sold by CoreWeave within its Forge platform with a free academic tier

MITFreemiumPaidSelf-hostedBroken

promptfoo

Promptfoo, Inc. · Prompting, evaluation & observability

Open-source, local-first CLI and library for evaluating prompts, models, RAG pipelines and agents, with automated red teaming and CI integration

MITOpen sourceFreeLocalSelf-hostedBroken

DSPy

Stanford NLP and the DSPy community · Prompting, evaluation & observability

Python framework for programming language-model pipelines with typed signatures and modules, then optimizing prompts and weights automatically against a metric

MITOpen sourceFreeLocalSelf-hostedNov 2, 2026

Ragas

Vibrant Labs (formerly Exploding Gradients) · Prompting, evaluation & observability

Open-source Python library for evaluating LLM and RAG applications with LLM-driven and custom metrics, experiments, dataset management and result tracking

Apache-2.0Open sourceFreeLocalBroken

Braintrust

Braintrust Data, Inc. · Prompting, evaluation & observability

Hosted observability and evaluation platform for AI agents with tracing, experiments, pattern detection, a CLI, MCP access and an AI assistant (Loop) that analyzes traces and builds evals

MixedFreemiumPaidNov 2, 2026