Private AI · inference performance
KV Cache Size: Why 32k Context Runs Out of Memory
KV cache size grows linearly with context: 5 GiB at 32k tokens for Mistral NeMo 12B, measured. The n² attention matrix is not what fills a 12 GB card.
8 min read
Author · Pexon Romania
Flavius Moldovan, Pexon Romania
Flavius Moldovan writes for Pexon Romania S.R.L. on the Pexon engineering blog, starting with a three-part series on how LLM inference works: prefill and decode, KV cache size, and batching, each measured on real hardware. This page lists every published post under that byline, newest first, with its date and reading time.
3 published posts
Everything Flavius Moldovan has published
Newest first. Business areas: Private AI.
Private AI · inference performance
LLM Batching: 6.8x the Throughput at 4.7x the Latency
LLM batching measured on one 12B model: 28 tokens/s for one user, 194 for 32 together, each of them 4.7 times slower. Where the trade turns.
8 min read
Private AI · inference performance
Prefill vs Decode: Why Token Generation Ignores TFLOPs
Prefill vs decode, measured on one 12B model: 284 tokens/s reading the prompt, 29 tokens/s writing the answer. Decode is bound by memory bandwidth.
9 min read
Next step
Talk to the people who write this
Thirty minutes with an architect from the team, about your systems rather than ours.