LLM inference, part 1 of 3
Prefill vs decode: the same GPU, ten times slower once it starts writing
Prefill vs decode is the split behind every slow local LLM. On Mistral NeMo 12B at four bits, Pexon measured 284 tokens per second reading the prompt and 29 writing the answer on one Apple M4 Pro. Decode rereads all 7.5 GB of weights for each token, so memory bandwidth, not TFLOPs, sets its speed.
Decode is not slow because the GPU is weak. Every generated token rereads the entire model from memory, so the number on the spec sheet that sets its speed is GB/s, not TFLOPs.
Where buying compute helps, and where it does not
Five situations, what the consensus expects, and what limits them
| Situation | What the consensus expects | What actually limits it | The caveat in the same row |
|---|---|---|---|
| One user, short prompt, long answer | A faster GPU fixes it | Memory bandwidth: 7.5 GB of weights read per token | Speed scales with GB/s, so a card with twice the TFLOPs and the same bandwidth changes nothing |
| Long document in, short summary out | A faster GPU fixes it | Compute: prefill handles the whole prompt in one parallel pass | Here the consensus is right, and it is the row most chat benchmarks never test |
| Same chat, 32,768 tokens of history | Same speed as an empty chat | Bandwidth again: the KV cache is reread every step; we measured 29.6 falling to 14.4 tokens/s | Flash attention saves memory here, it does not remove the read |
| Thirty-two users at once | Thirty-two times slower for each | Batching shares one weight read: we measured 28 tokens/s total for one stream, 194 for 32 | Each user still drops from 28 to 6 tokens/s; throughput is bought with latency |
| Upgrading an RTX 4070 to an RTX 4090 | 2.8x the FP16 tensor TFLOPs, so 2.8x faster | 2.0x the bandwidth, 504 to 1,008 GB/s | Expect close to 2x on single-user decode; the rest arrives only in prefill and large batches |
Two phases, one model, two different bottlenecks
Prefill and decode run the same weights and stress different parts of the machine. Prefill takes your entire prompt, system instructions and retrieved context included, and pushes it through the network in one pass. Every weight that arrives from memory is multiplied against hundreds or thousands of tokens at once, so the arithmetic units have work for every byte they wait for.
Decode cannot do that. It produces exactly one token, appends it, and runs the network again for the next one.
Each of those steps still has to read every weight, all 7.5 GB of them for a four-bit Mistral NeMo 12B, to do one token's worth of multiplication. At roughly two floating-point operations per parameter per token, our prefill run was doing about 7.0 trillion operations a second and our decode run about 0.7 trillion on the same chip. Nothing about the hardware changed between the two lines of output. The amount of work per byte fetched did, by a factor of ten.
That is why the first token of a long answer arrives quickly and the rest trickle. The trickle is the memory bus, running near its limit, and no setting in the inference engine makes a bus wider.
- 1. Prompt: system text, retrieved context, the question
- 2. Prefill: one parallel pass over every prompt token
- 3. KV cache: keys and values for every token so far
- 4. Decode: reads all weights and the cache, emits one token
- 5. Answer: streamed to the reader token by token
- Then from Decode back to KV cache: append the token, repeat per output token
The decoding phase incurs a similar level of I/O to the prefill phase, making it constrained by the GPU's memory bandwidth.
Our measurement
The same model and the same machine, timed in both phases with llama.cpp's own benchmark tool. pp512 is prefill over a 512-token prompt; tg128 is decode of 128 tokens.
# Prefill vs decode on one machine: Apple M4 Pro, 24 GB unified memory
# Model: Mistral-Nemo-Instruct-2407, Q4_K_M GGUF (6.96 GiB, 12.25 B params)
# llama.cpp build 11429 (d81235049), Metal backend, 3 repetitions
llama-bench -m Mistral-Nemo-Instruct-2407-Q4_K_M.gguf -p 512 -n 128 -r 3
| model | size | params | backend | test | t/s |
| llama 13B Q4_K - Medium | 6.96 GiB | 12.25 B | BLAS,MTL | pp512 | 284.06 ± 1.93 |
| llama 13B Q4_K - Medium | 6.96 GiB | 12.25 B | BLAS,MTL | tg128 | 29.22 ± 0.30 |Decode came out at 29.2 tokens per second. Multiply by the 7.47 GB the weights occupy and the machine was moving 218 GB/s, which is 80% of the 273 GB/s Apple specifies for the M4 Pro. That is what a bandwidth-bound workload looks like from the outside.
Measured by Pexon on 16 October 2026 on an Apple M4 Pro (14-core CPU, 20-core GPU, 24 GB) with llama.cpp build 11429 from Homebrew, Metal backend, default flash-attention setting. llama-bench labels the architecture 'llama 13B'; the file is Mistral NeMo 12B. We have not run this benchmark on NVIDIA hardware; every NVIDIA figure below is a datasheet ceiling.
The same division, five devices
Single-user decode ceiling for this model, set by memory bandwidth alone
Apple M4 Pro, 273 GB/s (we measured 29)
37 tok/s
RTX 4070, 504 GB/s
67 tok/s
L40S, 864 GB/s
116 tok/s
RTX 4090, 1,008 GB/s
135 tok/s
H100 SXM, 3.35 TB/s
448 tok/s
Bandwidth divided by the 7.47 GB the weights occupy. Ignores the KV cache and every overhead, so real numbers land below. Our M4 Pro reached 80% of its ceiling.
Why the TFLOPs on the box are the wrong number, in one ratio
Arithmetic intensity, operations performed per byte moved, decides which side of the machine is the limit. The scaling literature puts it simply: for a matrix multiplication against a small batch, the intensity is roughly the number of tokens in the batch. Every accelerator has a crossover point where compute time and memory time are equal, peak operations per second divided by bandwidth.
For the H100 that crossover is about 295. For an RTX 4070 it is about 116 with FP32 accumulation and 231 with FP16 accumulation, from NVIDIA's own figures of 58.3 and 116.6 dense tensor TFLOPS over 504 GB/s.
Single-user decode sits at roughly one operation per byte at 16-bit weights, and about 3.3 at the 0.61 bytes per parameter of Q4_K_M. Quantisation helps, and it is still thirty to ninety times below the line. Our position is blunt: for anyone running one or a few users per card, tensor TFLOPs should be the last column you read on a datasheet, after bandwidth and after memory capacity.
The strongest objection is that serving systems batch, and batching moves decode towards the line. It does, and it is the subject of part three. But batching raises total throughput, not the speed any single user sees, and most private deployments are judged by the person waiting at the prompt.
When the consensus holds, and when it does not
Compute is the limit: buy TFLOPs
- Long prompts, short answers: document summarisation, classification, extraction over retrieved context. Time to first token is prefill.
- Heavy batching: a serving stack running dozens of concurrent streams climbs towards the compute ceiling. On our machine 32 streams reached 194 tokens/s against a prefill rate of 286.
- Embedding and reranking: these are all prefill. There is no decode loop to wait on.
Bandwidth is the limit: buy GB/s
- One person at a prompt: chat, coding assistants and agents streaming to a single user. Tokens per second track bandwidth divided by model size.
- Long conversations: the KV cache joins the weights on every step. At 32,768 tokens of history our decode fell to 14.4 tokens/s.
- Larger models on the same card: doubling the weights halves the ceiling, whatever the compute rating says.
What this costs us to say
Bandwidth-first buying is the wrong rule for a retrieval workload that sends 20,000 tokens in and gets 200 back. There, prefill is most of the wall-clock time and compute decides it, and a cheaper high-bandwidth card can lose to a compute-heavy one. Which of the two your traffic is cannot be read off a datasheet; it has to be measured, which is what GPU sizing for LLM inference starts with.
Where to go next
- GPU sizing for LLM inference: buy the right number of cards — the same bandwidth arithmetic, applied to your measured traffic
- KV cache size: why 32k context runs out of memory — part 2, the memory the decode loop has to reread
- LLM batching: 6.8x the throughput at 4.7x the latency — part 3, how serving stacks push decode towards the compute line
- LLM quantisation accuracy: what four bits actually costs — the accuracy side of shrinking the weights decode has to read
Next in the series · Part 2KV Cache Size: Why 32k Context Runs Out of Memory
What engineers ask when the tokens crawl
Why is the first token fast and every token after it slow?
Because they are two different workloads on the same weights. The first token waits for prefill, which pushes the whole prompt through the model in one parallel pass and keeps the compute units busy. Every later token is a separate decode step that reads all of the weights again to produce one token, and that step waits on memory, not on arithmetic.
Will a GPU with more TFLOPs generate tokens faster?
Only in proportion to its memory bandwidth, not its TFLOPs. Moving from an RTX 4070 to an RTX 4090 buys 2.8 times the FP16 tensor throughput but 2.0 times the bandwidth, 504 against 1,008 GB/s, so single-user generation should get roughly twice as fast. The extra compute shows up in prefill and in large batches.
How do I estimate tokens per second before buying hardware?
Divide memory bandwidth by the size of the weights. That is a ceiling, not a forecast: our M4 Pro reached 80% of it, 29.2 tokens per second against 36.5. Add the KV cache to the denominator for long conversations, because decode rereads that too, and at 32k tokens of history our measured speed halved.
Is prefill always compute-bound?
No, a short prompt can be memory-bound as well. The arithmetic intensity of a pass is roughly the number of tokens in it, and a GPU needs somewhere between about 100 and 300 tokens per pass before compute becomes the limit. A prompt of a few dozen tokens stays below that line; retrieval-augmented and document prompts are far above it.
What does it cost to size this properly for our own workload?
Our fixed-price Readiness Blueprint is €4,900 over two weeks. It measures your real prompt and answer lengths and peak concurrency, then turns them into a bandwidth, memory and card-count model you keep. A per-token ceiling from a datasheet is free; knowing which of your workloads is prefill-heavy is the part that needs the measurement.
Next step
Size the hardware against your own prompts
Two weeks, fixed price: your real prompt lengths, answer lengths and peak concurrency measured, then turned into a bandwidth, memory and card-count model you keep, whether or not we build the cluster.