Azure AI Foundry vs. Kubernetes — the token volume that decides it

Managed endpoint or your own cluster: the decision is volume, not fashion

Azure AI Foundry and Kubernetes are not competitors in the way the comparison is usually framed. Foundry is a managed model-serving platform: you call an endpoint, Microsoft runs the infrastructure, and you pay per token. Kubernetes on AKS is the self-operated path: you run a serving layer such as vLLM in your own cluster, manage the GPU node pools, and pay only for the compute.

The difference shows up in exactly one number: token volume at steady state. Below a threshold, the managed endpoint is the right answer — the per-token price buys the infrastructure and the operations you would otherwise run yourself. Above it, self-hosted inference on your own cluster runs at roughly a third of the per-token cost, because you stop paying the managed premium on every single token.

Both paths are legitimate. The mistake is choosing without measuring your own volume, model mix and concurrency — which is what the comparison below is built to prevent.

Below 20 million tokens per month, the managed endpoint is usually the honest answer. Above it, the gap compounds: every additional token on the managed path carries the same premium.

What the crossover actually depends on

The 20-million-token figure is a starting point, not a law. Four variables move it in practice. Utilisation is the largest: a GPU node pool that idles at ten percent is more expensive per token than the managed endpoint it was meant to replace. Model size matters because it decides how many tokens fit on a card. Your team's Kubernetes maturity decides the real cost of the self-operated path — the infrastructure bill is only part of it. And data residency can decide the question outright, because a workload that may not leave the estate has only one of the two options.

The failure mode the comparison exists to prevent is buying GPUs before measuring any of the four. Teams read a blog post, order a node pool, and discover in month three that the utilisation curve does not support it. The sizing discipline from our GPU inference work applies here unchanged.

Four numbers that decide the platform

Each number below is measurable in two weeks of instrumented traffic. None of them should come from a vendor slide.

• Token volume per month at steady state — the single number that decides the crossover. Under 20M, Foundry usually wins on total cost; above it, self-hosting compounds.
• Utilisation of the GPU node pool you would actually run. Idle GPUs are the most expensive mistake in self-hosted inference, and the hardest to notice from a dashboard.
• Model size and context window, which decide how many tokens fit on a card and therefore how many cards you need. A 70B model at 4-bit needs different hardware than a 7B model.
• Data residency and contract requirements, which can decide the question outright: if prompts may not leave the estate, the managed endpoint is not an option regardless of volume.

The routing rule that makes both paths work

A LiteLLM gateway in front of both paths: steady load to self-hosted vLLM, bursty traffic to Azure AI Foundry.

# LiteLLM config: steady load to self-hosted vLLM, burst to Foundry
model_list:
  - model_name: chat-default
    litellm_params:
      model: vllm/llama-3.3-70b-instruct
      api_base: http://vllm-inference.aks.svc:8000/v1
  - model_name: chat-burst
    litellm_params:
      model: azure/azure-ai-foundry/llama-3.3-70b
      api_key: os.environ/AZURE_FOUNDRY_KEY

router_settings:
  routing_strategy: usage-based-routing
  model_group_alias:
    chat-default: chat-burst   # failover when self-hosted is saturated

The honest caveat: self-hosted is not free operations

The per-token saving on the self-hosted path is real, and it is not the whole story. Someone has to operate the cluster: node upgrades, GPU driver versions, serving-layer releases, alerting, and the runbook for when a card fails at 2 a.m. For a team that already runs AKS, that cost is incremental. For a team that would learn Kubernetes for this project, it can exceed the token saving.

That is why the engagement structure is the same as everywhere else on this site: the two-week blueprint measures the workload first, and the handover includes the runbook and the operations story — not just the deployment.

The managed endpoint and the self-operated cluster are not rivals. They are two lanes of the same gateway, and the routing rule is a configuration file.

Size the GPUs before you buy them

• GPU inference sizing — The two-week blueprint turns the crossover into a hardware plan: how many cards, which generation, and what it costs to run your actual workload.