01 · Routing
Round-robin burns the prefix cache
Agent and RAG traffic reuse long system prompts. Sending each turn to a random replica re-encodes work another pod already holds.
llm-d is traffic control for vLLM, not a second model server
llm-d is an open-source Kubernetes-native stack for distributed LLM inference. Founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, llm-d sits above vLLM: an inference gateway scores load, predicted latency and prefix-cache hits, then can split prefill from decode across pods. Teams reach for llm-d when round-robin on one replica no longer holds the latency SLO.
Three ways a healthy vLLM cluster still misses the SLO
02 · Shape
One replica cannot hold the model
Frontier open-weight checkpoints spill off a single node. Cloning the same fat pod is not the same as splitting the work.
03 · Mix
Prefill and decode fight for the same GPU
Long prompts want compute. Token generation wants memory bandwidth. Sharing one pool makes both worse at peak.
04 · Ops
A custom scheduler becomes an unpaid product
Teams write affinity hacks, then own them. llm-d exists so that layer is a project with founders, not a sidecar you cannot staff.
The two answers on the table both fail, for different reasons
The first bad answer is more replicas of the same vLLM pod, behind a generic Service. That scales copies. It does not know which copy holds the prefix, and it does not split prefill from decode. Google Cloud's founding post on llm-d describes exactly this congestion: uniform round-robin on mixed RAG and agent traffic.
The second bad answer is a bigger node. That helps until the next model, and it still round-robins. llm-d is the third shape: keep vLLM (or SGLang) as the engine, put a Kubernetes-native inference gateway in front, and score endpoints on cache, load and predicted latency before you pay for disaggregation.
What llm-d actually adds above the model server
01
Model server
vLLM or SGLang still run the weights. llm-d does not replace them. If this layer is wrong, the gateway only distributes a bad serving choice.
- vLLM
- SGLang
02
Endpoint picker
The scheduler scores replicas for prefix-cache hits, current load, P/D role and predicted latency. That is the difference between an inference gateway and kube-proxy.
- cache
- load
- SLA
03
Optional disaggregation
Prefill pods and decode pods, KV moved between them (NIXL in the project's architecture). A second pool, not a default. Well-lit paths exist; they are not free.
- prefill
- decode
- NIXL
The picker is the argument. Round-robin has none of these scores.
Illustrative of llm-d's published scheduler themes, not a Helm chart we ran in a lab.
# llm-d endpoint picker (architecture, not a copy-paste install)
scorers:
- prefixCache # same system prompt → warm KV
- load # skip the hot decode pod
- predictedLatency
disaggregation: prefillDecode # only after routing is not enoughThree scorers are the product. Disaggregation is the expensive optional line. Invert that order and you buy RDMA to hide a round-robin bug.
What llm-d covers, and what you still have to build
| Layer | llm-d | Still yours |
|---|---|---|
| Model serving | Orchestrates vLLM / SGLang | Checkpoint choice, quantisation, eval |
| Request routing | Cache-, load- and P/D-aware picker | Auth, tenants, token budgets |
| Scale-out | Well-lit paths, inference gateway | GPU procurement and utilisation |
| Disaggregation | Prefill/decode pools + KV transfer | RDMA, two-pool operations |
Google Cloud reported about 2× time-to-first-token on code-completion-style tests in the founding announcement. That is their measurement, not a Pexon benchmark. llm-d.ai also publishes larger marketing ratios; treat those as project claims.
The breaks that surprise platform teams
- Disaggregation is not step one. The sibling self-hosting post already said this: quantisation and cache-aware routing first. llm-d makes that routing a project instead of a script.
- A fat model is not an llm-d requirement. Small models with heavy prefix reuse still benefit from the picker. Ultra-class checkpoints need the whole stack plus GPUs. Do not confuse the two.
- Founders do not operate your cluster. CNCF sandbox and five founding vendors mean the design is not a one-man Helm chart. They do not mean it is hands-off.
Trade-off
Stay on single-node vLLM when concurrency is modest, prefixes are short, and the model fits. Introduce llm-d when you can name the missed SLO (TTFT, ITL, or GPU idle from cache misses). If you cannot name it, you are installing a control plane for sport.
Where this sits in the cluster
- GPU sizing for LLM inference — the money page: how many cards before you add a control plane
- LLM self-hosting on Kubernetes — the getting-started post; llm-d is the scale-up layer
- open weight model hosting — the models this stack actually serves
- Private AI hub — the cluster this post belongs to
llm-d questions
What is llm-d?
llm-d is an open-source, Kubernetes-native stack for distributed LLM inference. It orchestrates model servers such as vLLM and SGLang: intelligent routing, optional prefill/decode disaggregation, and well-lit Helm paths. It is a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA.
Do we need llm-d if vLLM already runs?
Not if a single replica meets the SLO. llm-d starts to earn its keep when round-robin wastes prefix cache, when the model no longer fits on one node, or when prefill and decode need separate pools. vLLM remains the engine; llm-d is the traffic control.
What is prefill/decode disaggregation in llm-d?
Prefill is compute-heavy prompt processing; decode is memory-bound token generation. llm-d can run those stages on separate pods and move the KV cache between them (NIXL in the project's architecture). It needs a real network and two pools. Most estates should try cache-aware routing first.
Does llm-d require RDMA?
Disaggregated serving wants a fast interconnect for KV transfer. Prefix-cache-aware routing on a co-located pool does not. If you do not have RDMA and do not have a latency SLO that justifies it, stay on one pool and pick endpoints by cache and load.
What if we do not want to operate llm-d ourselves?
Then do not install it as a weekend project. llm-d is platform operation: gateway, scheduler, two pools, observability. Pexon sizes the serving layer first; we only introduce llm-d when the measured workload has already beaten single-node vLLM.
Next step
Measure whether llm-d is the next layer, or just more YAML
Two weeks, fixed price. We instrument prompt length, concurrency and prefix reuse, then say whether cache-aware routing is enough or whether prefill/decode pools are justified. The plan is yours whether or not we build it.