The data cannot leave. So the server comes to you.
An AI server is the hardware that runs LLM inference on premises: GPUs, memory bandwidth and a serving layer. Pexon sizes the hardware against the real workload before anything is bought, from GPU selection to a vLLM or Ollama serving layer your own team can operate.
An AI server is the hardware that runs your model where your data lives.
An AI server is a machine optimised to run large language models locally: one or more GPUs, enough memory bandwidth for token generation, and a serving layer such as vLLM or Ollama that exposes the model as an API. It exists because some workloads cannot leave the building — the data classification forbids the cloud, the contract requires on-premise processing, or the token bill has crossed the line where owned hardware is cheaper.
The common mistake is deciding on the hardware before the workload. Teams pick a GPU because it is popular, buy the server, and discover in month three that utilisation is ten percent. The sizing work exists precisely to prevent that: it runs on rented capacity or existing cards, establishes the utilisation curve, and only then does procurement happen.
What an AI server actually costs, by configuration
The range is wide, and the number that matters is the one computed from your workload, not the one on a price list. These bands are the shapes we see in production.
| Configuration | Hardware | Use case |
|---|---|---|
| Entry workstation | 1× RTX 4090 (24 GB), 64 GB RAM | Single-team prototyping, fine-tuning experiments, Ollama with 7-14B models |
| Mid-range inference server | 2× RTX 5090 (32 GB each), 128 GB RAM | Production serving for 20-50 users, 32-70B models at 4-bit |
| Production rack server | 4× A100 80 GB or H100, NVLink, 512 GB RAM | 70B+ models, high concurrency, enterprise serving with vLLM |
| GPU cluster | 8× H100, InfiniBand or RoCE fabric | Frontier-class workloads, 140B+ models, multi-tenant serving |
The serving layer matters as much as the cards. LiteLLM and GPU inference sizing are the two pages that turn these bands into a decision.
Three ways to run inference, and when each one wins
The decision is not about the model. It is about the shape of the workload and where the data may be processed.
| Option | When it wins | When it loses |
|---|---|---|
| Buy an AI server | Large, steady workload; data cannot leave; contract requires on-premise | Bursty traffic, pilot phase, uncertain model roadmap |
| Rent GPU servers | Pilot, sizing phase, temporary capacity spikes | Long-running steady load — rental exceeds owned cost after months |
| Cloud model API | Small or bursty workloads, no residency constraint | Data classification forbids it, or bill scales linearly with every user |
Our rule: the two-week blueprint measures prompt lengths, concurrency and KV-cache demand before any of the three is chosen. Procurement after the arithmetic, never before.
From bare metal to a working inference API
An AI server is not a GPU in a box. The stack that makes it useful has five layers, and each one is an engineering decision.
| Layer | What it does |
|---|---|
| GPU and memory | The cards and the memory bandwidth that decide tokens per second. Sizing is arithmetic, not a vendor quote. |
| Serving layer | vLLM, TensorRT-LLM or Ollama in front of the model: continuous batching, KV-cache management, the throughput your team has to carry. |
| Model and quantisation | Which open-weight model for which workload, and at what quantisation — measured against your eval set, not a leaderboard. |
| Gateway | LiteLLM in front of everything: routing, budgets, audit logging, per-team attribution. |
| Access control | Authentication bound to your existing directory, so the AI server is governed by the same identities as everything else. |
The arithmetic before the purchase
Most AI server projects fail in the last mile, not in the benchmark: nobody can operate the serving layer after handover, the KV-cache arithmetic was wrong, or the cards were bought before the workload was measured. We deliver the runbook, the alerting and the sizing assumptions as part of the build — the same way every engagement here starts with one source taken end to end into production.
Start with the GPU inference sizing blueprint at EUR 4,900: two weeks of instrumented traffic, a memory and throughput model, and a hardware plan with every assumption written down.
Common questions
What is an AI server?
An AI server is a machine optimised to run large language models locally: one or more GPUs, enough memory bandwidth for token generation, and a serving layer such as vLLM or Ollama that exposes the model as an API. It is the alternative to sending prompts to a vendor API, and it exists because some workloads cannot leave the building.
How much does an AI server cost?
From roughly EUR 3,000 for a workstation with a single RTX 4090 up to EUR 150,000+ for a multi-GPU rack server with A100 or H100 cards. The honest answer depends on the workload: prompt length, concurrency and context window decide the GPU count. Pexon's two-week blueprint measures the workload first and returns a costed hardware plan before any procurement.
Should we buy an AI server or rent GPUs?
Buy if the workload is large and steady, if the data classification forbids the cloud, or if a contract requires processing inside your own estate. Rent if the traffic is bursty or the project is still a pilot. Buying before sizing is the most common and most expensive mistake in on-premise AI.
Not a sales call. An architecture call.
Thirty minutes with the architect who would actually run the engagement.