KV cache size, not quadratic attention, is what fills your GPU at 32k context

One model, two context lengths

What llama.cpp allocated for Mistral NeMo 12B at 4,096 and 32,768 tokens

Read from llama.cpp's allocation log on the Metal backend. CUDA reserves its scratch buffers slightly differently, so treat the two scratch rows as an order of magnitude; the weight and cache rows are the same on any backend.

The cache is the line item that grows, and it grows in a straight line

Generating a token means comparing it with every token before it. For that comparison each earlier token contributes two vectors per layer: a key, which is what the new token is matched against, and a value, which is what it contributes once matched. Those vectors do not change once computed, so every engine stores them instead of recomputing them for each new token. That store is the KV cache.

One more token in the conversation means one more key and one more value in every layer, and nothing else. The cache therefore grows linearly with context and with the number of conversations in flight, and llama.cpp allocates it up front for the full context you configure, before the first prompt arrives.

That is why the error appears when you raise the context setting, not when someone sends a long prompt.

The formula and the allocator agree to the mebibyte. If your estimate is off, the inputs are wrong, not the arithmetic.

Where the quadratic story comes from, and why it is not your out-of-memory error

Attention compute is quadratic: every token is scored against every other, so doubling the context quadruples the scoring work. Memory is a different matter. If the full score matrix were stored, one layer of Mistral NeMo at 32,768 tokens would need 32,768 × 32,768 entries for each of 32 heads, about 64 GiB at two bytes each, and nobody would get past 8k context on any card.

Nobody stores it. FlashAttention computes the scores in tiles that fit in on-chip memory and never writes the n × n matrix out; its authors state the memory cost as linear in sequence length while the operation count stays quadratic. Without flash attention, llama.cpp still processes the prompt in 512-token chunks, so its scratch buffer scales with context times 512, which is the 2,176 MiB in the table. A 512-token chunk scored against 32,768 positions for 32 heads at four bytes is 2,048 MiB, most of that buffer.

Grouped-query attention is the other reason the numbers are survivable. Mistral NeMo has 32 query heads but only 8 key-value heads, so it stores a quarter of what full multi-head attention would. At 32k tokens that is 5 GiB instead of 20.

Our view, stated plainly: an out-of-memory error at long context is a KV cache budgeting problem in every engine we would deploy. Debugging it as an attention-algorithm problem wastes a day.

Prefill vs decode: why token generation ignores TFLOPs: where the quadratic compute cost actually shows up, as prefill time

One conversation, getting longer

The f16 KV cache for Mistral NeMo 12B as the context grows

1. 4,096 tokens 640 MiB
2. 16,384 tokens 2.5 GiB
3. 32,768 tokens 5 GiB, with 6.96 GiB of weights: past a 12 GB card
4. 131,072 tokens, the full window 20 GiB for one user, before any weights

Computed at 160 KiB per token, the figure llama.cpp allocated; one sequence, f16 cache. Every additional concurrent user adds the same again, which is why the 128k window the model advertises is not a setting a single consumer card can honour.

Reproduce it

Two commands: one prints what llama.cpp allocates for your model and context, the other computes it before you download anything.

# 1. What llama.cpp allocates (verbose log, one generated token)
llama-completion -v -m model.gguf -c 32768 -ctk q8_0 -ctv q8_0 -fa on \
  -n 1 -p "Hi" -no-cnv 2>&1 | grep -a -E "llama_kv_cache: size|compute buffer"
# llama_kv_cache: size = 2720.00 MiB ( 32768 cells, 40 layers, 1/1 seqs) ...

# 2. The same number from config.json, before downloading weights
def kv_cache_mib(layers, kv_heads, head_dim, tokens, bytes_per_value=2.0, seqs=1):
    # q8_0 = 34/32 bytes per value, q4_0 = 18/32
    return 2 * layers * kv_heads * head_dim * tokens * bytes_per_value * seqs / 2**20

kv_cache_mib(40, 8, 128, 32768)          # 5120.0  (f16)
kv_cache_mib(40, 8, 128, 32768, 34/32)   # 2720.0  (q8_0)

If the two numbers disagree, check num_key_value_heads in config.json. The widely copied formula in NVIDIA's 2023 inference-optimisation post uses hidden_size instead, which ignores grouped-query attention and gives 25 GiB for this model at 32k instead of 5.

Sources: Mistral-Nemo-Instruct-2407 config.json · FlashAttention (Dao et al., 2022) · GQA (Ainslie et al., 2023) · llama.cpp server README (flash attention and cache-type flags) · NVIDIA: Mastering LLM Techniques, Inference Optimization (2023) · ggml block_q8_0 definition. Read 16 October 2026. Vendor documentation changes; verify against the current release.

Where to go next

• GPU sizing for LLM inference: buy the right number of cards — the same memory model, fed with your measured context lengths and users
• Prefill vs decode: why token generation ignores TFLOPs — part 1, why every byte of cache also costs decode speed
• LLM batching: 6.8x the throughput at 4.7x the latency — part 3, where the cache decides how many users fit at once
• LLM quantisation accuracy: what four bits actually costs — shrinking the weights to leave room for the cache