What is LiteLLM? The one API in front of every LLM provider

The LLM landscape is fragmented, and the fragmentation has a cost

OpenAI, Anthropic, Google, Grok, open-weight models — every provider has its own API, its own keys, its own billing, its own rate limits. A company that runs AI seriously rarely runs one provider; it runs several, because different models are good at different things and priced differently. That is the source of the problem this article is about.

The fragmentation shows up as four specific pains. Every application needs integration code for every provider it calls. Real API keys get copied into teams and repos, which is a security problem waiting for an incident. Nobody can answer who spent what on which model. And switching a provider means changing application code, which is how vendor lock-in survives even when the contract allows you to leave.

None of these four is a blocker on its own. Together they are the reason a serious AI estate eventually looks for a single point of control.

A gateway is the missing layer between your apps and the providers

An LLM gateway is a proxy server that sits between your applications and the model providers. Your applications call one endpoint; the gateway decides which provider actually serves the request. Everything the four pains describe — keys, limits, cost attribution, model switching — stops living in each application and moves into that one layer.

LiteLLM is the best-known open-source implementation of this idea. It is free, self-hostable, and supports a provider list that covers essentially everything in production use: OpenAI, Google Gemini, Anthropic, Grok, and open-weight models served in your own infrastructure. Because it exposes one OpenAI-compatible API, your application code does not need to know which provider is behind the request at all.

The rest of this article is what that one layer actually does — the four controls that make the gateway more than a routing shortcut.

Executable artefact

One API, three providers, zero changes in the application

This is the whole idea in its smallest form. The application calls chat-default; the gateway holds the provider keys and the routing. Swap a model by editing this file, not the code.

# config.yaml — the application still just calls the one endpoint
model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-4o
      api_key: "os.environ/OPENAI_API_KEY"
  - model_name: chat-alternative
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: chat-local
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama:11434

litellm_settings:
  fallbacks: [{"chat-default": ["chat-alternative", "chat-local"]}]
  num_retries: 2

The fallback list is the cheap resilience: if the primary provider is down or rate-limited, the request fails over to the next route without an application change.

What LiteLLM does

The four controls that define the gateway

1. One API for every provider. A single OpenAI-compatible format in front of all providers, so the application never knows — or cares — which one answered. Model availability is configurable per provider, from a strict allowlist to everything.
2. Teams, permissions and limits. Teams get their own model access, with per-team cost limits (daily, weekly, monthly) and rate limits on tokens or requests. Admin versus user roles with fine-grained permissions on top.
3. Virtual keys instead of real keys. Teams and developers work with virtual keys that LiteLLM mints. Each key can be restricted to specific models, and access can be revoked instantly — without ever rotating the real provider credential.
4. Usage tracking and cost analysis. Every request is logged: which key, which model, what it cost. The same prompt can be compared across providers, so the pricing strategy is a decision made on data, not a guess.

A practice example, anonymised

A mid-size company ran AI applications on two providers: a cheap open-weight model for high-volume internal tasks and a frontier model for the hard cases. Every team had its own API keys, the monthly costs were a mystery, and switching a model meant touching application code.

With LiteLLM, both providers were configured in the gateway with clear model access per team. Teams worked with virtual keys, so no real credential was in circulation. The development team got a monthly budget, and overruns were blocked automatically. Usage tracking showed at a glance which team used which model and how much.

The result, in the company's own words: costs became planable and transparent per team, real API keys stopped leaving the building, model changes run through the gateway configuration instead of a code deployment, and the price comparison inside the gateway showed exactly where moving to a cheaper model would pay. No part of that required exotic engineering.

The Pexon view: the gateway as the cost-control heart of the platform

Our position is that LiteLLM fits naturally into a platform architecture where the customer owns the stack and no single LLM vendor is a bottleneck — technically or commercially. One central gateway, many providers, full control.

In most customer estates, Microsoft AI Foundry serves as the model catalogue. LiteLLM does not replace that; it complements it. It bundles the Foundry models and every other provider — OpenAI, Gemini, open-weight models — behind one API, and adds the usage tracking and cost steering a catalogue alone does not provide.

Combined with multi-harness orchestration and intelligent model routing — the frontier model for the hard ten percent, routing for the rest — the gateway becomes the cost-control layer everything runs through. That is what makes AI economically sustainable rather than a line item that grows every month.

And because LiteLLM is open source and self-hostable, the data-sovereignty argument is concrete: the routing decision and the usage record live on hardware you operate, with no license lock-in and no hidden cost. That is the architecture we implement for customers who want AI they can defend.

Sources: LiteLLM documentation — proxy, virtual keys, budgets · BerriAI/litellm on GitHub. Read 2026-08-28. Vendor documentation changes; verify against the current release.

Keep reading

• LiteLLM proxy in production: setup, security, monitoring — the follow-up to this page — what this explainer describes, built the way an enterprise has to build it
• LiteLLM vs OpenRouter — the gateway decision for multi-model apps — self-hosted gateway against hosted routing, when the data leaves your estate
• GPU inference sizing: matching hardware to a token budget — the cost model for the open-weight side of the routing rule
• LLM gateway for Claude: per-team budgets and audit-grade logging — the same gateway pattern, scoped to the Claude estate
• Private AI hub: open weights on hardware you control — all the posts in this cluster