The LiteLLM proxy, run the way an enterprise has to run it

The quickstart gives you a running proxy. It does not give you a deployable one.

LiteLLM proxy is an open-source gateway that puts one OpenAI-compatible API in front of many model providers. You define the models in a config file, your applications call a single endpoint, and the gateway handles routing, load balancing, fallbacks and cost tracking behind it. The README path to a running instance takes about ten minutes, and it is the wrong path for anything that will outlive a demo, because a default LiteLLM deployment is one master key, one config file and no idea who spent what.

The enterprise setup is not a bigger version of the quickstart. It is a different configuration, and this guide walks through it in the order we run it in customer estates: the docker-compose skeleton, the config with model routes, virtual keys instead of the master key, budgets and limits per team, monitoring before the first incident, and routing rules with fallbacks as the cost control.

One disclaimer up front, because it decides how you read everything below. A self-hosted gateway with real availability is usually more expensive per token than a SaaS aggregator for the first year, and anyone who tells you otherwise is comparing a GPU rental price against a list price. You run LiteLLM for control: your keys, your logs, your infrastructure, your DSGVO story. The engineering below is about making that control real rather than nominal.

Sequence

The setup order we run in, and why it is that order

1. Deploy the container before you write a single route. The image runs anywhere Docker runs. Pin the tag, mount a config directory, and make the health endpoint reachable before any application points at it. A proxy that cannot answer /health has no business being configured.
2. Write the config with provider keys in the environment, never in the file. config.yaml declares model names, providers and routing rules. The keys themselves come from environment variables or a secrets manager, so the config file is safe to commit and review.
3. Create virtual keys and throw the master key away from day one. Every application and every team gets its own virtual key. The master key then exists only to mint and revoke virtual keys, and nobody's code ever sees it. This is the single control that makes every later one possible.
4. Set budgets and limits per team before the first user. LiteLLM supports per-key budgets, per-team limits and model access rules. Configure the containment before the traffic: a limit you add after the incident is a post-mortem, not a control.
5. Stand up monitoring before the second team joins. Cost, tokens, latency and error rate per model and per team, on a dashboard the operator actually opens. The metrics exist in the proxy; the work is deciding who looks at them and when.

Executable artefact

The docker-compose skeleton and the config that goes with it

Versions are placeholders; pin whatever is current when you deploy. The shape is the point: environment for keys, volume for config, healthcheck that makes the proxy observable.

# docker-compose.yml — the enterprise skeleton
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest   # pin a release tag in production
    container_name: litellm
    ports:
      - "4000:4000"
    environment:
      - STORE_MODEL_IN_DB=false
      - DATABASE_URL=postgresql://litellm:${DB_PASSWORD}@postgres:5432/litellm
      # provider keys come from the environment, never from config.yaml
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - AZURE_API_KEY=${AZURE_API_KEY}
    volumes:
      - ./config.yaml:/app/config.yaml
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health/liveliness"]
      interval: 30s
      timeout: 5s
      retries: 3

# config.yaml — model routes only; keys stay in the environment
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
  - model_name: local-llama
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama:11434   # self-hosted fallback on your own GPUs

litellm_settings:
  drop_params: true
  num_retries: 2
  request_timeout: 600

general_settings:
  master_key: ${LITELLM_MASTER_KEY}   # used only to mint virtual keys

The healthcheck and the key separation are the two lines that separate this from the quickstart.

Security is virtual keys, budgets, RBAC and an audit trail — in that order

The security story of a LiteLLM deployment is mostly the story of what the master key is not used for. Virtual keys carry their own budgets, spend limits and model access, so a leaked application key costs you the exposure of one team's routes rather than the whole gateway. The master key, held by the platform team, exists to mint and revoke those keys and to change the config.

Budgets come second because they are the control that makes cost attribution real. A per-team budget turns the monthly invoice from a single line into a set of accounts you can defend at renewal, which is exactly the argument the FinOps team needs when the question is whether the gateway pays for itself. The audit trail comes third: who called which model, from which key, at what cost, on a log you can replay. LiteLLM writes this natively; the enterprise work is deciding retention and who may read it.

For European operators the fourth layer is where the gateway earns its keep. Self-hosted on your own infrastructure, with provider keys held in your environment and logs under your control, the data-sovereignty argument becomes concrete: the routing decision and the usage record live on hardware you operate. That is the difference between a defensible DSGVO story and a hand-wave at a SaaS provider's subprocessor list.

What the gateway cannot do is police the prompts themselves. Prompt injection and guardrail enforcement belong to the layer above — the tracing and guardrail stack the OWASP guidance for LLM applications points at. Treat LiteLLM as the transport control and buy the guardrail layer separately; pretending one product does both is how the incident happens.

Monitoring: the four numbers that decide whether the gateway is working

A LiteLLM deployment produces four numbers worth watching per model and per team, and they answer four different questions. Cost per team tells you whether the budgets are doing their job. Tokens per model tells you whether the routing rules are actually routing. Latency tells you whether a provider degradation is your problem or theirs. Error rate tells you whether the fallbacks are firing or silently failing.

The monitoring work is not in the metrics, which the proxy exposes natively, but in the decisions around them. Who is on call for a gateway that sits between every application and every model? What happens to the dashboard when nobody looks at it for a week? The answer we give customers is that a gateway is infrastructure, not a library: it gets an owner, an on-call rotation and a change process, and the monitoring dashboard is the visible part of that ownership.

The cost control follows from the same data. Routing rules that prefer a cheaper model for a task the expensive one does not need, fallback routes that catch a provider outage without a human being paged at 3am, and a model-priority policy that says which provider wins when two can serve the same request. None of this is exotic. All of it is configuration the quickstart does not have, which is the whole difference between running LiteLLM and running LiteLLM as infrastructure.

The middle path we see most often: OpenRouter for the prototype, LiteLLM for the production estate, and the routing rule between them as the first real configuration. Nobody runs one gateway for long.

Sources: LiteLLM documentation — proxy, virtual keys, budgets · BerriAI/litellm on GitHub. Read 2026-08-18. Vendor documentation changes; verify against the current release.

Keep reading

• Private AI hub: open weights on hardware you control
• LLM quantisation: what four-bit costs in accuracy
• GPU inference sizing: matching hardware to a token budget
• LLM gateway for Claude: per-team budgets and audit-grade logging