LLM self-hosting on Kubernetes is production reality
LLM self-hosting on Kubernetes is production reality, not an experiment: one platform processes about 650 billion tokens a month for 30 internal tenants on shared GPU servers. It pays off on data protection, cost and control. The four optimizations that make it economical are quantisation, speculative decoding, KV-cache-aware routing and, where justified, prefill-decode disaggregation.
Not an experiment: 650 billion tokens a month on shared GPUs
Self-hosted LLMs on Kubernetes are operated reality, not a lab project. One large platform processes roughly 650 billion tokens a month for 30 internal tenants — on-premises, on shared GPU servers. That is the scale at which the economics of self-hosting become real.
The reasons a company runs its own models are four: data protection — data does not leave the building; cost — own infrastructure instead of variable API fees; fine-tuning — deploy your own models; and control — no vendor lock-in.
The architecture is worth understanding before any optimization: on-premises GPU servers, shared model servers where many users and services share a GPU farm for high utilisation, a gateway layer for auth and token-based rate limiting, and vLLM in the backend for efficient parallel processing.
Know your workloads before you optimize
- Three use cases carry 90 percent of the traffic. Text summarisation for agent memory is the largest, safety classification in real time second, and information extraction third. If you do not know which use cases dominate, you optimise the wrong end.
- Sort workloads by latency sensitivity. Safety classification is latency-sensitive and single-turn; batch extraction is latency-insensitive. Human-in-the-loop assistants are sensitive and multi-turn; autonomous agents are insensitive and long-running.
- Maximise throughput, hold latency in bounds. Throughput rises with concurrency — continuous batching and page attention scale it — but latency suffers. Pack the latency-insensitive workloads hard; bound the sensitive ones.
- Treat agentic workloads as the challenge. Large models need 8–16 GPUs per server, context grows every round and pressures GPU memory, and bursty traffic from harnesses spawning sub-agents breaks naive load balancing. KV-cache affinity matters.
The four optimizations
What makes self-hosting economical — with the honest trade-offs
| Technique | Gain | Trade-off |
|---|---|---|
| Quantisation (FP8/INT8/INT4) | Smaller models, more throughput, less memory | Slight accuracy loss — worth it |
| Speculative decoding | Faster generation, identical quality | Two models in memory; depends on predictability |
| KV-cache-aware routing | TTFT down from 1,200ms to 200ms in multi-turn | Moderate architecture complexity |
| Prefill-decode disaggregation | Independent scaling, better inter-token latency | High complexity — RDMA, separate pools |
The recommendation ladder: quantisation first — it is native in vLLM and wins on both throughput and latency with no new components. Speculative decoding for latency-sensitive tasks. KV-cache-aware routing for agentic and multi-turn workloads. Prefill-decode disaggregation only when the workload and the network justify it.
The four techniques, in the order to adopt them
Quantisation is the simple win. FP8-quantised models beat the baseline on both throughput and latency, and vLLM supports it natively — no new components, clear first step. It is the easiest way to make a self-hosted platform economical.
Speculative decoding is for latency-sensitive tasks: a fast draft model generates tokens ahead, the large model verifies them in parallel, and the output is identical. The acceptance rate depends on predictability — around 51 percent on known data, 15 percent on random data — which decides whether it pays off.
KV-cache-aware routing is the agentic king's path. Requests sharing a prefix — system prompts, agent instructions — get routed to the pod that holds the cache, and time-to-first-token falls from 1,200 ms to around 200 ms in multi-turn use. For agent traffic it is close to mandatory.
Prefill-decode disaggregation is powerful but complex: separating the compute-bound prefill from the memory-bound decode needs RDMA networks and separate pools. Only adopt it when the workload truly justifies the operational cost.
Executable artefact
The routing rule that makes agent traffic fast
Agentic workloads share long prefixes — the same system prompt and instructions on every call. Routing requests with the same prefix to the same pod keeps the KV cache warm and cuts time-to-first-token by a factor of six.
# KV-cache-aware routing — route by prefix affinity, not round-robin
# Requests with the same system prompt go to the pod holding that cache.
- A request arrives for "chat-default" with a known agent system prompt.
- The endpoint picker routes it to the pod whose KV cache already
holds that prefix (cache affinity).
- TTFT: ~1,200ms (cold) -> ~200ms (warm, multi-turn).
# Without it, every round re-encodes the whole prefix — the hidden
# cost of agentic traffic on self-hosted models.The rule is simple and the effect is large: affinity beats round-robin when the prefix is long and reused, which is exactly the agent case.
Self-hosting is not a weekend project — it is platform operation: GPU clusters, vLLM, gateway, monitoring, updates. That is the operational responsibility, not the installation.
The Pexon view: the operated LLM platform
Our position is that self-hosted LLMs are private AI on Kubernetes — exactly the operational responsibility we take on. We build and run the whole stack: shared GPU model servers with resource management on Kubernetes, vLLM with quantisation, continuous batching and page attention, a gateway layer with token-based rate limiting and usage tracking, and KV-cache routing for agentic workloads with SLO-aware autoscaling.
The honest caveat is that workload analysis comes first — optimizing the wrong end is how platforms overspend. There is no single optimisation for every workload; the four techniques are a ladder, not a buffet. And agentic traffic is the future, so KV-cache routing is where the platform earns its keep for the workloads that are coming.
The honest risks. Workload analysis comes first — without knowing which use cases dominate, you optimise the wrong end. There is no uniform optimisation: different workloads need different techniques. Agentic and multi-turn workloads are becoming dominant, so KV-cache routing becomes a required building block. Do not rush prefill-decode disaggregation — it is powerful but complex, and only pays when the workload and network justify it. And self-hosting is operations work: GPU clusters, vLLM, gateway, monitoring and updates are platform operation, not installation.
Keep reading
- llm-d: distributed inference on Kubernetes — the scale-up layer when round-robin on vLLM is no longer enough
- Ollama vs vLLM: the serving engine decision — the serving layer underneath this platform
- GLM 5.3 Flash: frontier behaviour at a flash price — the open-weight models a self-hosted platform runs
- GPU inference sizing: matching hardware to a token budget — the money page for this cluster
- Private AI hub: open weights on hardware you control — the cluster this post belongs to
Questions we get asked about self-hosted LLMs
Does LLM self-hosting really pay off?
At sufficient utilisation, yes — on data protection, cost and control. One production platform processes about 650 billion tokens a month for 30 internal tenants on shared GPU servers, which makes the economics work. The key is shared model servers with high GPU utilisation, not a GPU per tenant.
What is the easiest first optimisation?
Quantisation — FP8 models run natively in vLLM and improve both throughput and latency without new components. It is the clear first step; the more advanced techniques like KV-cache routing and prefill-decode disaggregation come later.
Do we need prefill-decode disaggregation?
Only if your workloads are latency-sensitive enough to justify the complexity — it needs RDMA networks and separate pools. For most estates, quantisation plus KV-cache-aware routing delivers the gains that matter.
Why KV-cache-aware routing for agents?
Because agentic workloads share long prefixes — system prompts and instructions. Routing requests with the same prefix to the same pod cuts time-to-first-token dramatically, from about 1,200 ms to around 200 ms in multi-turn use. It is the key technique for agent traffic.
Next step
Get the self-hosting economics calculated on your workload
Two weeks, fixed price. We classify your inference workload, size the GPU platform, and hand over the cost model with the optimization plan — quantisation first, KV-cache routing for agents, disaggregation only if it pays. The plan is yours whether or not we build it.