The model runs where your data already lives.
Field notes on running open-weight models on hardware you operate yourself, not a vendor comparison. This hub collects the work that decision creates: choosing a model, what four-bit quantisation costs in accuracy, sizing GPUs against a real token budget, and the serving and identity layer around it. For teams whose data cannot leave the building.
Self-hosted inference, without the ideology.
This area is about running models on infrastructure you control: open-weight model selection, quantisation, GPU sizing, serving stacks, and the identity layer that decides who is allowed to ask what. It is written for the engineers and architects who will be paged when it breaks, not for the person signing the contract.
The position we take here is unpopular in both directions. Private AI is not a cost-saving measure — a self-hosted stack with real availability and a real on-call rotation is usually more expensive per token than an API for the first year, and anyone who tells you otherwise is comparing a GPU rental price against a list price. It is a control measure. You do it because a contract, a works council agreement or a customer NDA makes the alternative impossible, and then you make it cheap by being deliberate about which workloads need it at all.
Most companies end up mixed: an open-weight model behind their own gateway for anything touching contracts, drawings or personnel data, and a frontier API for the rest. The interesting engineering is the routing rule between them, and that is what these posts spend their time on.
Common questions
Is private AI only relevant for defence and regulated industries?
No. The most common driver we see stated is contractual rather than regulatory: customer agreements, works council agreements or supplier NDAs that forbid sending certain material to a third-party processor. That constraint appears in manufacturing long before any regulator does.
Do open-weight models close the gap to the frontier models?
For narrow, well-specified tasks against your own data, the gap is much smaller than benchmark tables suggest, because retrieval quality and prompt design dominate. For open-ended reasoning it is still real. Deciding per use case beats deciding once for the whole company.
Not a sales call. An architecture call.
Thirty minutes with the architect who would actually run the engagement.