GPU sizing for LLM inference is arithmetic, not a vendor quote.
GPU sizing for LLM inference decides how many GPUs an on-premise model actually needs, before the hardware is ordered. Pexon measures your real prompt lengths and concurrency, computes KV cache demand — Llama 3.3 70B costs 320 KiB per token at FP16 — and hands over a benchmarked vLLM serving stack with the arithmetic written down.
The cluster gets sized before anyone measures the workload.
Industrial companies between EUR 50m and 2bn revenue tend to arrive at this decision the same way. A vendor proposes eight H100s, finance asks whether four would do, and nobody can answer, because no one has measured prompt length, completion length or peak concurrency. So the cluster is bought oversized and idles, or bought undersized and queues by ten in the morning. Both cost real money, and both are decidable in advance.
Three deliverables, in this order
A measured workload profile
Two weeks of instrumentation on your actual traffic: prompt and completion length distributions, peak concurrent sessions, and the time-to-first-token your users will tolerate. Sizing without this is arithmetic performed on invented inputs, which is how the eight-GPU proposal happens.
A memory and throughput model you keep
Weights, KV cache and activation overhead per candidate configuration. Llama 3.3 70B publishes 80 layers, 8 key-value heads and a head dimension of 128, which works out to 320 KiB of KV cache per token at FP16 — a single 32k-token session holds 10 GiB before it generates anything. Numbers like that decide the card count, and they are computable now.
A benchmarked serving stack
vLLM or TensorRT-LLM deployed on the chosen hardware with continuous batching and paged KV cache, then benchmarked against your own traffic rather than a leaderboard. The PagedAttention paper (Kwon et al., SOSP 2023) reports 2-4x throughput over FasterTransformer and Orca at equal latency; what it does on your workload is an empirical question, and the harness stays on your cluster so you can re-run it.
What changes
- You order a GPU count you can defend to finance, with the arithmetic attached rather than a vendor's recommendation.
- Quantisation becomes a measured trade-off against your own evaluation set instead of a rumour from a public benchmark.
- The choice between an H100 SXM at 3.35 TB/s and an L40S at 864 GB/s stops being a price comparison and becomes a throughput decision.
- Power and rack planning starts from real figures: NVIDIA rates the H100 SXM at 700 W per card and the PCIe variant at 350 W, and your facilities team needs that number early.
- When the first workload doubles, the sizing model tells you what to add, instead of restarting the procurement conversation from nothing.
Fixed at €4,900. We instrument your real workload, build the memory and throughput model against three candidate hardware configurations, and hand you the numbers with every assumption written down. The document is yours to keep, whether Pexon builds the cluster or your existing integrator does. All prices are net and exclude VAT.
Sizing questions we get asked
What does a GPU sizing engagement cost?
The two-week Readiness Blueprint is fixed at €4,900 net. It returns the measured workload profile, a memory and throughput model for three candidate hardware configurations, and a costed build plan. Cluster build and serving work is scoped from that plan, never quoted before the measurement exists.
Can we run a 70B model on a single 80 GB GPU?
Not at FP16: 70 billion parameters at two bytes each is roughly 140 GB of weights, before a single token of KV cache. Four-bit quantisation brings the weights near 40 GB and does fit, but it costs accuracy on some tasks and leaves less headroom for concurrent sessions. Whether that trade is acceptable is an evaluation question, and it is measurable in days.
Do we need H100s, or will L40S cards do?
It depends on whether your workload is prefill-heavy or decode-heavy. Token generation is bound by memory bandwidth, and NVIDIA lists 3.35 TB/s for the H100 SXM against 864 GB/s for the L40S — close to a factor of four. For long-document summarisation the cheaper cards often hold up; for interactive chat at concurrency they usually do not.
Not a sales call. An architecture call.
Thirty minutes with the architect who would actually run the engagement.