GPU sizing for LLM inference decides how many GPUs an on-premise model actually needs, before the hardware is ordered. Pexon measures your real prompt lengths and concurrency, computes KV cache demand — Llama 3.3 70B costs 320 KiB per token at FP16 — and hands over a benchmarked vLLM serving stack with the arithmetic written down.
The starting point
Industrial companies between EUR 50m and 2bn revenue tend to arrive at this decision the same way. A vendor proposes eight H100s, finance asks whether four would do, and nobody can answer, because no one has measured prompt length, completion length or peak concurrency. So the cluster is bought oversized and idles, or bought undersized and queues by ten in the morning.
Both cost real money, and both are decidable in advance.
Why the workload decides the card
Token generation is bound by memory bandwidth, not by compute — which is why the card choice is a throughput decision rather than a price comparison. Long-document summarisation is prefill-heavy and tolerates a slower card; interactive chat at concurrency is decode-heavy and does not. NVIDIA lists 3.35 TB/s for the H100 SXM against 864 GB/s for the L40S, close to a factor of four, and the question is which of the two your traffic actually is.
That is why the workload profile comes first in the deliverable order. A sizing model built on invented prompt lengths answers the question beautifully and wrongly; the two weeks of instrumentation are what turn the arithmetic from a demonstration into a decision.
What we build
01
Two weeks of instrumentation on your actual traffic: prompt and completion length distributions, peak concurrent sessions, and the time-to-first-token your users will tolerate. Sizing without this is arithmetic performed on invented inputs, which is how the eight-GPU proposal happens.
02
Weights, KV cache and activation overhead per candidate configuration. Llama 3.3 70B publishes 80 layers, 8 key-value heads and a head dimension of 128, which works out to 320 KiB of KV cache per token at FP16 — a single 32k-token session holds 10 GiB before it generates anything. Numbers like that decide the card count, and they are computable now.
03
vLLM or TensorRT-LLM deployed on the chosen hardware with continuous batching and paged KV cache, then benchmarked against your own traffic rather than a leaderboard. The PagedAttention paper (Kwon et al., SOSP 2023) reports 2-4x throughput over FasterTransformer and Orca at equal latency; what it does on your workload is an empirical question, and the harness stays on your cluster so you can re-run it.
Llama 3.3 70B at FP16, from the model's own published dimensions.
Layers, key-value heads, head dimension
80 · 8 · 128
published by the model
KV cache per token
320 KiB
at FP16
One 32k-token session
10 GiB
held before it generates anything
€4,900
We instrument your real workload, build the memory and throughput model against three candidate hardware configurations, and hand you the numbers with every assumption written down. The document is yours to keep, whether Pexon builds the cluster or your existing integrator does.
All prices are net and exclude VAT.
The two-week Readiness Blueprint is fixed at €4,900 net. It returns the measured workload profile, a memory and throughput model for three candidate hardware configurations, and a costed build plan. Cluster build and serving work is scoped from that plan, never quoted before the measurement exists.
Not at FP16: 70 billion parameters at two bytes each is roughly 140 GB of weights, before a single token of KV cache. Four-bit quantisation brings the weights near 40 GB and does fit, but it costs accuracy on some tasks and leaves less headroom for concurrent sessions. Whether that trade is acceptable is an evaluation question, and it is measurable in days.
It depends on whether your workload is prefill-heavy or decode-heavy. Token generation is bound by memory bandwidth, and NVIDIA lists 3.35 TB/s for the H100 SXM against 864 GB/s for the L40S — close to a factor of four. For long-document summarisation the cheaper cards often hold up; for interactive chat at concurrency they usually do not.
Next step
Thirty minutes with the architect who would actually run the engagement.