Field notes on running open-weight models on hardware you operate yourself, not a vendor comparison. This hub collects the work that decision creates: choosing a model, what four-bit quantisation costs in accuracy, sizing GPUs against a real token budget, and the serving and identity layer around it. For teams whose data cannot leave the building.
23 posts, newest first
inference performance
KV cache size grows linearly with context: 5 GiB at 32k tokens for Mistral NeMo 12B, measured. The n² attention matrix is not what fills a 12 GB card.
8 min read · Flavius Moldovan
inference performance
LLM batching measured on one 12B model: 28 tokens/s for one user, 194 for 32 together, each of them 4.7 times slower. Where the trade turns.
8 min read · Flavius Moldovan
inference performance
Prefill vs decode, measured on one 12B model: 284 tokens/s reading the prompt, 29 tokens/s writing the answer. Decode is bound by memory bandwidth.
9 min read · Flavius Moldovan
assistants
A ChatGPT alternative by constraint: Claude for long work, Perplexity for citations, Mistral for EU hosting, or self-hosting when prompts must stay in-network.
13 min read · Phillip Pham
workflow automation
n8n vs LangGraph for AI workflows: visual automation builder against agentic graph framework. When drag-and-drop wins and when code-first pays.
4 min read · Noel Dinger
engineering tooling
AI tools for engineers that earn their place in 2026: coding agents, model gateways, observability and evaluation. A practical stack, honestly assessed.
5 min read · Noel Dinger
open source interface
Open WebUI self-hosted: a private ChatGPT-style interface for your own models, with RAG, multi-user access and no data leaving your estate.
5 min read · Noel Dinger
inference serving
Model Hardware Standard: Anthropic's 27 August 2026 research preview for agents on lab devices. MCP is one access path. Open source is the named successor.
5 min read · Noel Dinger · updated 14 Sept 2026
inference serving
llm-d is the Kubernetes-native stack above vLLM: cache-aware routing, optional prefill/decode split, founded by Red Hat, Google, IBM, CoreWeave and NVIDIA.
5 min read · Phillip Pham
open weight serving
Nemotron is NVIDIA's open-weight family. Nemotron 3 Ultra is 550B/55B MoE, OpenMDW 1.1, self-hostable when prompts must stay on your GPUs.
5 min read · Phillip Pham
model routing
GLM 5.3 Flash: open-weight, 1M context, and an audit of 1,000+ PRs for 12 cents where a frontier model costs over $100. Routing just got cheaper.
6 min read · Phillip Pham
agent orchestration
LangGraph structures agentic AI as a directed graph of nodes, edges and state — control, visibility and human-in-the-loop for production agents.
6 min read · Phillip Pham
inference serving
LLM self hosting on Kubernetes is production reality: 650B tokens a month, 30 tenants, shared GPUs. The four optimizations that make it economical.
6 min read · Phillip Pham
agent orchestration
The same model can shine in one harness and stall in another. Most recent gains came from the harness — tools, memory and the agentic loop.
6 min read · Phillip Pham
open source platform
The open-source AI platform is the harness, not the model. How open harnesses, model routing and a 90/10 cost split keep your architecture independent.
7 min read · Phillip Pham
llm gateway
What is LiteLLM? The open-source gateway that puts one API in front of every LLM provider — virtual keys, team budgets, usage tracking, no lock-in.
8 min read · Phillip Pham
inference platform
Azure AI Foundry vs Kubernetes for LLM workloads: managed model serving against self-operated inference on AKS. Cost, control, GPU scaling and when each wins.
6 min read · Noel Dinger
llm gateway
LiteLLM vs OpenRouter: self-hosted gateway with budgets and audit logging against hosted routing with one API key.
5 min read · Noel Dinger
llm observability
MLflow vs Langfuse for LLM observability: experiment tracking and model registry against tracing, evals and production monitoring.
4 min read · Noel Dinger
inference serving
Ollama vs vLLM for LLM serving: developer ergonomics against production throughput. When Ollama is enough, when vLLM pays, and how to run both behind one gateway.
4 min read · Noel Dinger
open weight serving
Qwen hosting on your own hardware: model selection, vLLM vs Ollama setup, GPU sizing and EU data residency.
4 min read · Noel Dinger
llm gateway
LiteLLM proxy: Docker first, virtual keys not the master key, budgets per team. Self-hosted for control, not cheaper tokens. Monitor before the first incident.
9 min read · Phillip Pham
What four-bit and three-bit quantisation cost in accuracy, the KV cache and power arithmetic behind the decision, and how offline retrieval stays permission-aware.
10 min read · Noel Dinger
This area is about running models on infrastructure you control: open-weight model selection, quantisation, GPU sizing, serving stacks, and the identity layer that decides who is allowed to ask what. It is written for the engineers and architects who will be paged when it breaks, not for the person signing the contract.
The position we take here is unpopular in both directions. Private AI is not a cost-saving measure — a self-hosted stack with real availability and a real on-call rotation is usually more expensive per token than an API for the first year, and anyone who tells you otherwise is comparing a GPU rental price against a list price. It is a control measure. You do it because a contract, a works council agreement or a customer NDA makes the alternative impossible, and then you make it cheap by being deliberate about which workloads need it at all.
Most companies end up mixed: an open-weight model behind their own gateway for anything touching contracts, drawings or personnel data, and a frontier API for the rest. The interesting engineering is the routing rule between them, and that is what these posts spend their time on.
No. The most common driver we see stated is contractual rather than regulatory: customer agreements, works council agreements or supplier NDAs that forbid sending certain material to a third-party processor. That constraint appears in manufacturing long before any regulator does.
For narrow, well-specified tasks against your own data, the gap is much smaller than benchmark tables suggest, because retrieval quality and prompt design dominate. For open-ended reasoning it is still real. Deciding per use case beats deciding once for the whole company.
Thirty minutes with the architect who would actually run the engagement.