The data cannot leave. So the model comes to it.
Private AI means running open-weight models on hardware you control instead of sending prompts to a vendor API. Pexon covers the stack that decision creates: GPU sizing, model selection and quantisation, a vLLM or TensorRT-LLM serving layer, and access control bound to your existing directory. Built for industrial companies between EUR 50m and 2bn revenue.
Open-weight models on hardware you own.
Private AI means running open-weight models on hardware you control instead of sending prompts to a vendor API. The decision is not about the model — open-weight models have narrowed the capability gap far enough that whether they can do the work is rarely the interesting question. It is about where the data is processed, who operates the infrastructure, and what the platform team can carry afterwards.
For an industrial company, the trigger is usually one of three: the data classification forbids the API, the workload is large and steady enough that per-token pricing stops being cheaper than owned hardware, or a customer contract requires processing inside your own estate. Bursty, low-volume workloads are usually better served by an API — and we say so instead of selling you a GPU server you do not need.
The model is rarely the constraint.
What is left after the model choice is engineering: memory bandwidth, KV-cache management, quantisation, continuous batching — and an operations story your platform team can actually carry after handover. These projects fail in that last mile, not in the model benchmark, which is why the engagement is structured around the serving layer and the runbook, not around the demo.
A private AI deployment that nobody can operate is a liability with a nice dashboard. The runbook, the alerting and the upgrade path are part of the deliverable, and the sizing assumptions behind them are written down so the next capacity decision does not restart the project.
Four stages, and the order is not optional
A private AI engagement moves through four stages. Each one produces an artefact the next stage consumes — and skipping one means re-doing the work at triple the cost when it fails in production.
| Stage | What it produces |
|---|---|
| GPU sizing | How many cards, which generation, rented first. Sizing runs on rented capacity or existing hardware before anything is bought — the arithmetic comes before the procurement, not after it. |
| Model selection and quantisation | Which open-weight model for which workload, and at what quantisation. The choice is measured against your eval set, not against leaderboard position. |
| Serving layer | vLLM or TensorRT-LLM in front of the model: continuous batching, KV-cache management, and the throughput the platform team has to carry after handover. |
| Access control | Authentication and authorisation bound to your existing directory, so the private AI estate is governed by the same identities as everything else. |
The sizing stage is covered in depth on the GPU inference sizing page. Hosting the checkpoint — Nemotron, Qwen, GLM or DeepSeek behind a gateway — is the open weight model offer. The gateway questions — routing, caching, cost control across models — are covered in the LiteLLM proxy enterprise guide.
When private AI beats a hosted API — and when it does not
The honest answer has three criteria. First, the data classification: if the data cannot leave the estate, no price comparison matters. Second, the shape of the workload: large and steady traffic makes owned hardware cheaper than per-token pricing; bursty traffic makes it a waste. Third, the contract: some customers and some sectors require processing inside your own estate, full stop.
The common failure is deciding on the model before the workload. Teams pick an open-weight model because it is free to download, buy a GPU server for it, and discover in month three that the utilisation is ten percent. The sizing work exists precisely to prevent that — it runs on rented capacity or existing cards, establishes the utilisation curve, and only then does procurement happen.
No GPUs before the arithmetic.
Buying hardware before sizing is the most expensive mistake in private AI, and it is also the most common one, because it feels like progress. The sizing engagement runs on rented capacity — or on whatever cards you already have — and produces the numbers: throughput at what concurrency, memory for which context length, quantisation at which quality cost. Procurement happens after the arithmetic, and the arithmetic is the part that is cheap.
Where a private AI engagement starts
GPU Sizing for LLM Inference — Buy the Right Number
How many GPUs an on-premise LLM actually needs: KV cache arithmetic, quantisation trade-offs, and a vLLM serving stack benchmarked on your own traffic.
Open-Weight Model Hosting on Hardware You Control
Open-weight model hosting on your GPUs: Nemotron, Qwen, GLM and DeepSeek behind a LiteLLM gateway in the EU. Start with a two-week blueprint.
Common questions
When does private AI beat a hosted model API?
When the data classification forbids the API, when the workload is large and steady enough that per-token pricing stops being cheaper than owned hardware, or when a customer contract requires processing inside your own estate. Bursty, low-volume workloads are usually better served by an API, and we will say so.
Do we need to buy GPUs before we start?
No, and buying first is the common mistake. The sizing work runs on rented capacity or on whatever cards you already have, because the point is to establish how many you need. Procurement happens after the arithmetic, not before it.
Not a sales call. An architecture call.
Thirty minutes with the architect who would actually run the engagement.