Ollama vs. vLLM — the serving layer decides the throughput
Ollama vs vLLM is a serving-layer decision: Ollama wins on developer ergonomics and setup speed, vLLM wins on production throughput, continuous batching and KV-cache management. The crossover sits near sustained multi-user load. Pexon deploys both behind a LiteLLM gateway, so the serving layer is a configuration choice rather than a rewrite.
Two tools with the same job and completely different design targets
Ollama and vLLM both expose an OpenAI-compatible API in front of open-weight models. That is where the similarity ends. Ollama is built for developer ergonomics: install, pull a model, run it. vLLM is built for production throughput: continuous batching, paged KV cache, and as many tokens per second as the hardware can deliver under concurrent load.
The right question is not which one is better. It is which one your workload actually needs — and the answer is a measurement, not a preference.
Ollama wins on setup speed and single-user ergonomics. vLLM wins on sustained multi-user throughput on the same hardware. The crossover sits where concurrency starts to matter — typically above a handful of simultaneous users.
Ollama versus vLLM, side by side
| Dimension | Ollama | vLLM |
|---|---|---|
| Time to first inference | Minutes | Hours to days |
| Continuous batching | Limited | Core strength |
| Throughput at concurrency | Drops under load | Sustained, several times Ollama on same GPUs |
| Paged KV cache | No | Yes |
| Operational surface | Minimal | Real — needs a runbook |
| Best for | Prototyping, single-user, desktop | Production serving, multi-user |
Throughput claims come from the PagedAttention paper (Kwon et al., SOSP 2023) and our own serving benchmarks; the number that matters is tokens per second at your concurrency, measured on your hardware.
The mistake is choosing before measuring: teams standardise on Ollama for a pilot and hit the throughput wall at production, or buy vLLM infrastructure for a workload that never needed it.
What the measurement actually looks like
Two weeks of instrumented traffic answers the question with numbers. First, the concurrency profile: how many simultaneous requests your users actually generate. Second, tokens per second at that concurrency on both serving layers, on the same hardware. Third, the model mix — a 7B model behaves differently from a 70B model under load.
The result is usually one of three: Ollama is sufficient and the project stays simple; vLLM is needed and the migration is a configuration change behind the gateway; or the workload is bursty enough that neither self-hosted option is right and a managed endpoint wins.
The gateway that makes the serving layer a config choice
Both behind one LiteLLM gateway: Ollama for interactive prototyping, vLLM for production load.
# LiteLLM config: two serving layers, one API
model_list:
- model_name: chat-dev
litellm_params:
model: ollama/llama3.1:8b
api_base: http://localhost:11434
- model_name: chat-prod
litellm_params:
model: vllm/llama-3.3-70b-instruct
api_base: http://vllm-inference:8000/v1
api_key: os.environ/VLLM_KEYBoth are open source, both run on your own hardware, and neither locks you in. The serving layer is the layer most teams replace first — which is exactly why it belongs behind a gateway.
Three numbers that decide the serving layer
Each number comes out of the two-week measurement. None of them can be read off a benchmark leaderboard.
- Concurrent requests at peak — the single number that decides whether vLLM's continuous batching pays for its operational complexity.
- Tokens per second at that concurrency on both layers, on the same hardware. This is the number that survives a budget review.
- Model size and context length, which decide how much KV-cache memory each request consumes and therefore how many requests fit on a card.
Ollama is not a toy and vLLM is not overkill. They are two lanes of the same gateway, and the routing rule is a configuration file.
Go deeper on serving and hardware
- GPU inference sizing — The arithmetic behind how many GPUs your serving layer actually needs.
- LiteLLM enterprise setup — The gateway that makes the serving layer a configuration choice.
Serving layer questions
What is the difference between Ollama and vLLM?
Ollama is a developer-friendly model runner: install, pull a model, run an OpenAI-compatible API in minutes. vLLM is a production serving engine built for throughput: continuous batching, paged KV cache and higher tokens-per-second under concurrent load. Ollama is the right starting point; vLLM is the layer you move to when utilisation and concurrency grow.
Is Ollama fast enough for production?
For single-user or low-concurrency workloads, often yes. For sustained multi-user traffic, vLLM's continuous batching typically delivers several times the throughput on the same hardware. The measurement that decides it is tokens per second at your concurrency level, which is what the two-week blueprint instruments.
Can we run Ollama and vLLM together?
Yes, and that is the pattern we recommend: Ollama for prototyping and interactive single-user work, vLLM for production serving, both behind a LiteLLM gateway that routes by workload. The switch is a configuration change, not a migration.
Next step
Measure the throughput on your own workload
Two-week readiness blueprint at €4,900: we benchmark Ollama and vLLM against your actual prompts and concurrency, and hand back a serving recommendation with the tokens-per-second numbers attached. The plan is yours to keep whoever builds it.