LLM inference, part 3 of 3
LLM batching: what 32 concurrent users buy, and what each of them pays
LLM batching trades each user's speed for total throughput. On Mistral NeMo 12B at four bits, Pexon measured 28 tokens per second for one stream and 194 for 32 streams on one Apple M4 Pro, while each user fell from 28 to 6. Between four and eight streams, throughput doubled at almost no cost per user.
One model, one machine, six batch sizes
Batching multiplies the total and divides the share
- 6.8× total
- 28 decode tokens per second for one stream, 194 for 32 streams sharing each step.
- 4.7× slower per user
- Each of the 32 users received 6 tokens per second instead of 28. Throughput was bought with latency.
- 8 streams
- Going from four to eight streams nearly doubled the total, 70 to 134 tokens per second, while each user went from 17.4 to 16.7.
Decode speed by number of concurrent streams, measured
| Concurrent streams | Total decode tokens/s | Per stream tokens/s | Time per token for each user | What it means |
|---|---|---|---|---|
| 1 | 28.3 | 28.3 | 35 ms | Faster than anyone reads; the machine idles between weight reads |
| 4 | 69.6 | 17.4 | 57 ms | 2.5x the total for a stream 38% slower |
| 8 | 133.7 | 16.7 | 60 ms | The cheap step: total doubles, each user barely notices |
| 16 | 163.2 | 10.2 | 98 ms | Gains flatten; each user now pays for the others |
| 32 | 193.6 | 6.05 | 165 ms | Close to the compute ceiling; slower than comfortable reading for chat |
Each stream decoded 128 tokens after a 512-token prompt. 'Time per token' is the inverse of per-stream speed. The reading-speed remark in the last column is our judgement, not a measurement.
Why the first steps are almost free and the later ones are not
A decode step for one sequence reads every weight in the model to produce a single token, which is why part one of this series found it bound by memory bandwidth. Put a second sequence in the same step and the weights are read once and used twice. The bytes moved barely change; the useful arithmetic doubles. In the language of the roofline, the intensity of the step rises roughly with the number of sequences in it.
So the early steps cost almost nothing. Each new stream only adds the reading of its own KV cache and its share of the arithmetic.
The free ride ends when the arithmetic units, rather than the memory bus, become the limit. The machine tells you where that is: its prefill speed, the rate at which it processes prompt tokens in large parallel passes, is the compute ceiling. On our M4 Pro, prefill ran at about 286 tokens per second, and 32-stream decode reached 194. Past that point every additional user is paid for directly in everyone else's latency.
Our position, which a serving engineer optimising cost per token would dispute: for interactive internal tools, pick the concurrency at the knee and buy the next card before going past it. Running a card at maximum throughput looks efficient on a dashboard and feels broken to the person typing.
Our measurement
llama.cpp ships a batched benchmark that runs the same prompt and generation length at several batch sizes in one go.
# Throughput vs batch size: Apple M4 Pro, 24 GB, llama.cpp build 11429, Metal
# Model: Mistral-Nemo-Instruct-2407 Q4_K_M; 512-token prompt, 128 generated tokens per stream
llama-batched-bench -m Mistral-Nemo-Instruct-2407-Q4_K_M.gguf -c 32768 -b 2048 -ub 512 \
-npp 512 -ntg 128 -npl 1,2,4,8,16,32 -fa on
| PP | TG | B | N_KV | S_PP t/s | S_TG t/s |
|-----|-----|----|-------|----------|----------|
| 512 | 128 | 1 | 640 | 282.07 | 28.30 |
| 512 | 128 | 2 | 1280 | 275.77 | 42.14 |
| 512 | 128 | 4 | 2560 | 281.35 | 69.64 |
| 512 | 128 | 8 | 5120 | 285.83 | 133.65 |
| 512 | 128 | 16 | 10240 | 284.82 | 163.18 |
| 512 | 128 | 32 | 20480 | 286.06 | 193.55 |S_TG is total decode throughput across all streams; divide by B for what one user sees. S_PP stays flat at about 285 because prefill was already compute-bound at one stream.
Measured by Pexon on 16 October 2026 on an Apple M4 Pro (20-core GPU, 24 GB unified memory) with llama.cpp build 11429 (d81235049) from Homebrew. One run per batch size. We have not benchmarked vLLM or any NVIDIA card for this page; the shape of the curve is general, the knee position is specific to this machine and model.
Which concurrency setting fits who is waiting
The question
Who is waiting for the tokens, and how many of them at once?
- One engineer at a terminal or an agent loop for one user
- Run batch size one on a card of its own. Every concurrent job on that card slows the one person you are serving.
- A team of five to twenty sharing an internal assistant
- Use llama.cpp's server with a fixed number of slots near your measured knee and the unified KV buffer on. Add a card before raising the slot count past the knee.
- Hundreds of users or mixed-length traffic on datacenter GPUs
- Use vLLM with continuous batching and paged KV memory, and size max_num_seqs and memory from measured concurrency rather than the defaults.
- Overnight jobs where nobody watches the stream
- Maximise the batch until throughput stops rising. Latency is irrelevant when the reader is a database.
What decides the batch size you can actually run is memory, not the scheduler
Every stream in a batch carries its own KV cache. Our 32-stream run held 20,480 tokens of context, about 3.1 GiB at f16 for this model. The same 32 users with 8,000-token conversations would need about 39 GiB of cache, which no consumer card holds. Concurrency is therefore capped by memory long before the arithmetic runs out, and part two of this series has the formula.
Static batching, which waits for the slowest sequence in a group before admitting new work, wastes much of what is left. The Orca paper proposed scheduling at the granularity of a single decode iteration instead, so a finished sequence is replaced in the next step.
The vLLM authors then showed that existing systems used only 20.4% to 38.2% of their KV cache memory for actual token state, the rest lost to reservation and fragmentation. Paging that memory is how vLLM reports 2 to 4 times the throughput of FasterTransformer and Orca at the same latency. Both ideas are now standard: llama.cpp's server enables continuous batching by default, and vLLM pages its cache.
Three settings that decide your real batch size
- llama.cpp slots and the context split: with the KV buffer not unified, the context is divided by the number of sequences. Our first run used -c 16384 for up to 32 streams, which left 512 cells per stream for 640 tokens, and failed with 'failed to find a memory slot'. llama-server unifies the buffer by default when slots are automatic; set -c for the total you need.
- vLLM max_num_seqs: the engine picks a default per GPU class in its source, 256 sequences on cards under 70 GB in v0.31.0. It is a ceiling on concurrency, not a promise that the memory for that many exists.
- vLLM gpu_memory_utilization: defaults to 0.92 of the card for weights, activations and cache together. Whatever the weights leave inside that fraction is the KV pool, and the pool, not max_num_seqs, decides how many long conversations run at once.
Where we would choose differently
If you sell tokens rather than serve colleagues, cost per token is the metric and running well past the knee is correct, because nobody is watching each stream. And if the workload is a few heavy users on one card, batching tuning is a distraction from the real fix, which is more memory bandwidth or a second card. Which of these you are is a measurement of your own traffic, the first deliverable of GPU sizing for LLM inference.
Where to go next
- GPU sizing for LLM inference: buy the right number of cards — concurrency, cache and card count sized from your measured traffic
- Prefill vs decode: why token generation ignores TFLOPs — part 1, why a single stream leaves the compute units idle
- KV cache size: why 32k context runs out of memory — part 2, the memory every additional stream claims
- Ollama vs. vLLM: which serving layer fits your workload — choosing the engine that does the batching
Questions before you raise the concurrency setting
Does batching make a single request faster?
No, it makes every request in the batch slightly slower. Batching shares one read of the weights across several sequences, so the machine produces more tokens per second in total. In our run one stream decoded at 28 tokens/s alone and at 17 when it shared the step with three others.
Do I need vLLM to get continuous batching?
No, llama.cpp's server does it too, with continuous batching on by default. Set the number of slots with -np and keep the unified KV buffer so slots share the context pool. vLLM earns its place at higher concurrency, where paged KV memory and its scheduler keep many more sequences in flight on datacenter GPUs.
What batch size should I configure?
Start from the latency you can tolerate per user, not from the throughput you want. Pick the largest concurrency at which per-user speed still meets that target, and check that their KV cache fits. On our machine, eight streams kept each user near 17 tokens/s; thirty-two dropped them to 6.
Why does throughput stop rising at higher batch sizes?
Because decode reaches the compute ceiling the machine shows during prefill. Each extra stream adds work but no extra weight reads, until the arithmetic units are the bottleneck. Our 32-stream decode reached 194 tokens/s against a prefill rate of about 286 on the same hardware.
When is batching the wrong optimisation?
When one person is waiting and nobody else is. A coding assistant or an agent loop for a single engineer gains nothing from spare batch capacity, and every concurrent job on the same card makes that engineer's tokens slower. Give latency-critical users their own card or their own priority.
What does it cost to find the right concurrency for our workload?
Our fixed-price Readiness Blueprint is €4,900 over two weeks. It measures peak concurrent sessions, prompt and answer lengths and the latency your users accept, then sizes slots, KV memory and cards against them. The model and its assumptions are yours to keep, whoever builds the cluster.
Next step
Set concurrency from your users' latency, not a default
Two weeks, fixed price: peak concurrency, prompt and answer lengths and acceptable latency measured on your traffic, then slots, KV memory and cards sized against them.