Open WebUI — the ChatGPT-style front end that never sends a prompt anywhere

Your team wants a ChatGPT. Your data says it cannot leave the building.

The tension is real: employees are comfortable with a chat interface, and every vendor sells them one — hosted, with prompts travelling to someone else's infrastructure. Open WebUI is the answer that resolves the tension: an open-source, self-hosted chat interface that gives your team the familiar experience while the models, the prompts and the data stay inside your estate.

It connects to whatever you run — Ollama for a quick start, vLLM for production throughput, or any OpenAI-compatible endpoint — and adds the layer the raw serving layer does not have: conversations, document upload, retrieval and per-user accounts.

The interface is the easy half. The deployment — SSO, the serving layer behind it, the audit trail — is the part that decides whether the workspace survives a security review.

What the deployment actually looks like

Open WebUI runs in Docker and connects to the model layer behind it. The production shape is three layers: the serving layer (Ollama or vLLM) runs the models, Open WebUI provides the interface, and the LiteLLM gateway in front logs every request against a user identity.

The gateway is what makes the deployment governable: per-team budgets, request logs with retention, and the ability to route one workload to a self-hosted model and another to a frontier API from the same configuration.

Open WebUI behind the gateway

Docker compose for Open WebUI, pointed at your serving layer, with the gateway as the single egress point.

# docker-compose.yml — Open WebUI + your model layer
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports: ["3000:8080"]
    environment:
      OPENAI_API_BASE_URL: http://litellm:4000/v1   # gateway in front
      OPENAI_API_KEY: os.env/WEBUI_KEY
      ENABLE_RAG_WEB_SEARCH: "false"
      DEFAULT_USER_ROLE: "user"
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - litellm

Three decisions that shape the workspace

Each is answered in the first two weeks, and each decides how the workspace holds up in production.

• Where the model layer runs — Ollama for a quick start, vLLM for throughput; the choice follows the concurrency profile, not a preference.
• Who may access what — SSO with role-based access, so the workspace is governed by the same directory as everything else.
• What the gateway logs — every prompt against a user identity, with retention set to what your works council agreed.

A chat interface your team trusts, on hardware you control, with every prompt on the record. That is the whole product.

Build on the private AI stack

• Ollama vs. vLLM — Which serving layer to put behind the Open WebUI interface.
• Qwen hosting — A strong open-weight model family to serve through the workspace.
• GPU inference sizing — The arithmetic behind how many GPUs the deployment needs.