LiteLLM supply chain attack: the 60-minute enterprise containment and hardening runbook
The LiteLLM supply chain attack on October 21, 2026 compromised version 1.107.0 with a cook.pth startup hook that exfiltrated environment secrets and PostgreSQL LiteLLM_CredentialsTable keys. Enterprise operators can contain, audit, rotate credentials across four infrastructure owners, and redeploy a digest-pinned Kubernetes gateway with default-deny egress in 60 minutes.
Prerequisites & scope
What you need before starting the 60-minute containment sequence
On October 21, 2026 (between 11:39 UTC and 14:20 UTC), LiteLLM version 1.107.0 was published with a malicious cook.pth startup hook that exfiltrated environment secrets and queried LiteLLM_CredentialsTable whenever DATABASE_URL was set. At the end of this 60-minute runbook, you will have verified exposure without triggering the payload, rotated every compromised upstream key by owner, and redeployed LiteLLM behind a default-deny egress policy.
- Do not invoke python, pip, pytest, or uvicorn inside any suspect container or virtual environment — Python automatically executes .pth files in site-packages on startup, which re-triggers the credential stealer.
- Kubernetes cluster access (kubectl 1.28+) with a CNI that enforces NetworkPolicy resources (Cilium, Calico, or cloud-managed network policy controllers).
- Administrative access to upstream model provider consoles (Anthropic, OpenAI, Azure AI Foundry, AWS IAM/Bedrock, Google Vertex AI) and your PostgreSQL database host.
- Container registry CLI access (crane, skopeo, or docker manifest inspect) to resolve and pin immutable SHA-256 image digests for LiteLLM 1.106.2 or the 1.108 LTS line.
60-minute execution sequence
Five ordered steps to contain, inspect, rotate, and redeploy
- Sever outbound egress on running LiteLLM pods before touching containers (10 min). Apply an immediate default-deny egress NetworkPolicy to the litellm namespace before scaling down or inspecting pods. If a compromised pod is still running, cutting egress prevents secondary payload fetches or final exfiltration bursts while preserving container filesystem state for forensic inspection.
- Inspect site-packages and image layers from outside Python (10 min). Use POSIX find and grep from the node host, an ephemeral busybox debug container, or your CI cache directory to check for cook.pth, sitecustomize.py modifications, or litellm-1.107.0.dist-info. Never run pip list or python -c to check the version.
- Rotate PostgreSQL and upstream LLM provider keys across all four owners (20 min). If any pod, CI runner, or developer notebook pulled 1.107.0 or floating main-latest during the October 21 window, assume every credential reachable from that environment is compromised. Follow the ownership matrix below to revoke and reissue upstream API keys, database passwords, and cloud IAM roles before starting new pods.
- Redeploy LiteLLM pinned by immutable SHA-256 digest (10 min). Replace mutable tags like main-latest or v1.106.2 with an explicit @sha256 digest in your Helm values or Kubernetes Deployment manifest. Set readOnlyRootFilesystem: true on the container securityContext so runtime package tampering cannot write to site-packages.
- Enforce permanent egress allowlisting to approved LLM endpoints only (10 min). Replace the temporary lockdown policy with a production NetworkPolicy that permits egress only to your internal PostgreSQL instance (port 5432), internal Redis cache (port 6379), and the specific TLS endpoints of your model providers on port 443.
Executable artefact
Out-of-band inspection and Kubernetes egress lockdown
Run the filesystem check from outside the Python runtime, then lock down the pod network and pin the container image by digest so a compromised package cannot phone home.
# 1. Inspect virtualenvs or container rootfs WITHOUT starting Python (safe shell check)
find / -path "*/site-packages/cook.pth" -o -path "*/site-packages/litellm-1.107.0.dist-info" 2>/dev/null
# 2. Inspect running Kubernetes pods for the 1.107.0 tag or mutable main-latest pulls
kubectl get pods -A -l app.kubernetes.io/name=litellm \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'
# 3. Kubernetes NetworkPolicy: default-deny egress with explicit PostgreSQL + LLM HTTPS allowlist
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: litellm-egress-allowlist
namespace: ai-gateway
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: litellm
policyTypes:
- Egress
egress:
# Allow internal DNS resolution
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
# Allow internal PostgreSQL state database only
- to:
- podSelector:
matchLabels:
app: litellm-postgres
ports:
- protocol: TCP
port: 5432
# Allow outbound TLS (443) only to approved external CIDRs / proxy gateway
- to:
- ipBlock:
cidr: 10.240.0.0/16 # Internal corporate egress proxy / NAT inspection gateway
ports:
- protocol: TCP
port: 443You know it worked when find returns zero matches for cook.pth and kubectl exec into the gateway pod times out on curl https://example.com while internal database health checks stay green.
Enterprise rotation matrix
Who rotates which secret when the LLM gateway is compromised
| Credential class | How 1.107.0 accessed it | Responsible owner | Remediation standard |
|---|---|---|---|
| Upstream LLM API keys (Anthropic, OpenAI, Azure) | Direct query on LiteLLM_CredentialsTable via DATABASE_URL plus process env vars | AI Platform / FinOps Lead | Revoke keys in provider consoles; reissue into Vault/ExternalSecrets (our standard) |
| Gateway master key & PostgreSQL password | LITELLM_MASTER_KEY and DATABASE_URL read from container environment at startup | Database & Platform Ops | Rotate DB user password, roll master_key, and invalidate active admin UI sessions |
| Cloud IAM pod roles (Bedrock / Vertex AI) | IMDSv2 / IRSA / Workload Identity token environment variables inside the pod | Cloud Infrastructure / SecOps | Revoke active STS sessions, restrict IAM policy to specific model ARNs, audit CloudTrail |
| CI/CD & GitOps tokens | pip install litellm during Docker build or pytest runs on self-hosted CI runners | DevSecOps Team | Rotate GitHub Actions / GitLab runner secrets, purge pip caches, rebuild ephemeral runners |
| Downstream virtual keys (sk-...) | Stored as hashes or rows in LiteLLM_VerificationToken table inside PostgreSQL | Application Teams | Re-mint team virtual keys after master key rotation if database write access occurred |
Why a gateway compromise is uniquely severe: unlike a single compromised microservice, an LLM gateway holds the credentials for every model provider in the enterprise in one PostgreSQL table.
Immediate fix vs production baseline
What pinning 1.106.2 fixes today — and what is still missing for production
What works immediately after this 60-minute runbook
- Malicious startup hook removed. Moving off 1.107.0 to a verified 1.106.2 or 1.108 LTS digest eliminates the cook.pth payload from site-packages.
- Stolen provider keys neutralized. Rotating all keys stored in LiteLLM_CredentialsTable stops unauthorized token spend on your OpenAI, Anthropic, and Azure accounts.
- Unrestricted internet callbacks blocked. The Kubernetes NetworkPolicy prevents any future compromised Python dependency from opening arbitrary outbound C2 connections.
What is still missing for bank-grade production operation
- Plaintext database credential storage. If model keys sit in LiteLLM_CredentialsTable without KMS envelope encryption or short-lived cloud workload identity, any SQL read exposes every provider.
- Uninspected TLS egress on port 443. An IP-level Kubernetes NetworkPolicy allows port 443; without an L7 egress proxy (Envoy/Squid) filtering by SNI hostname, data can still exfiltrate over HTTPS.
- Upstream release cadence churn. Even with LiteLLM shifting to its November 2026 Tuesday release cadence and 1.108 LTS pilot, pulling upstream releases without an internal SBOM and 7-day quarantine mirror leaves CI pipelines exposed.
Gateway hardening roadmap
From emergency containment to a zero-trust enterprise LLM gateway
Stage 0
Emergency containment & key rotation
Isolate pods, verify site-packages outside Python, rotate all provider keys in LiteLLM_CredentialsTable, and redeploy pinned to an immutable SHA-256 digest.
60 minutes (today)
Stage 1
L7 SNI egress proxy & read-only container hardening
Route all gateway outbound traffic through an Envoy egress filter that permits only api.anthropic.com, api.openai.com, and your regional Azure/Bedrock endpoints, with readOnlyRootFilesystem enforced.
Days 2–5
Stage 2
Keyless cloud authentication & internal artifact quarantine
Replace static API keys in PostgreSQL with short-lived AWS IRSA / GCP Workload Identity / Azure Managed Identity tokens, and mirror vetted LiteLLM LTS images into a private Harbor or ECR registry.
Week 2 — production target
Commands and manifests verified on Kubernetes v1.30 (Ubuntu 24.04 LTS nodes with Cilium CNI) against ghcr.io/berriai/litellm v1.106.2 and the v1.108 LTS pilot line; timeline and payload mechanics cross-checked against BerriAI/litellm issue #47826 and StepSecurity forensic telemetry.
Next steps in the Private AI gateway cluster
- LLM gateway for Claude: per-team budgets and audit-grade logging — our managed gateway architecture for regulated European enterprises
- LiteLLM proxy in production: virtual keys, budgets, and monitoring — the baseline setup guide once your container image and egress are hardened
- Private AI hub: open weights and self-hosted infrastructure — all engineering guides on self-hosted inference and gateway security
Questions platform teams ask during LiteLLM incident response
Was LiteLLM 1.107.0 removed from PyPI and Docker Hub?
Yes. The malicious 1.107.0 artefacts published between 11:39 UTC and 14:20 UTC on October 21, 2026 were yanked from PyPI and removed from container registries the same day. Removal from the registry does not clean workloads, CI caches, or developer virtual environments that pulled the package during the exposure window.
Can I fix an affected environment by running pip install litellm==1.106.2?
No. The payload uses a cook.pth file in Python site-packages that executes automatically whenever the Python interpreter starts. Running pip list, pip uninstall, or python -m pip inside the affected environment triggers the credential stealer again. Always inspect the filesystem using shell tools like find or stat from outside Python, then destroy and rebuild the container or virtualenv from scratch.
Which credentials must an enterprise rotate after the LiteLLM compromise?
Because the payload reads both process environment variables and the PostgreSQL LiteLLM_CredentialsTable when DATABASE_URL is configured, you must rotate four distinct credential sets: all upstream model provider API keys stored in the gateway (OpenAI, Anthropic, Azure, AWS Bedrock, Vertex AI), the LiteLLM master key and PostgreSQL database password, cloud IAM instance or pod role credentials, and any CI/CD runner tokens that built images during the October 21 window.
What hardware or cluster prerequisites do I need to run this containment runbook?
You need kubectl cluster-admin access to a Kubernetes 1.28+ cluster with a CNI plugin that enforces NetworkPolicy (Calico, Cilium, or Azure CNI), shell access to your container registry or host nodes with standard POSIX utilities (find, sha256sum, jq), and administrative access to your upstream LLM provider consoles to revoke and reissue API keys.
Is pinning LiteLLM to version 1.106.2 enough for production security?
No. Version tags in Docker and PyPI are mutable unless verified by cryptographic digest, and any future dependency compromise would still have outbound network access if the pod runs with unrestricted egress. Production security requires pinning the immutable @sha256 container image digest, mounting a read-only root filesystem, and enforcing a Kubernetes default-deny egress NetworkPolicy that permits traffic only to your approved LLM provider endpoints.
Next step
Harden your self-hosted LLM gateway before the next dependency incident
Two weeks, fixed price. We audit your LiteLLM or vLLM gateway deployment, enforce immutable digest pinning, lock down Kubernetes egress policies, and integrate automated secret rotation with your vault.