The LLM quantisation accuracy trade-off: what four bits actually costs

Four bits is cheap. Three bits is not.

The LLM quantisation accuracy trade-off is smaller than most procurement conversations assume, right up to the point where it suddenly is not. Dropping to five or four bits costs almost nothing you can measure. Dropping to three costs you arithmetic. Uygar Kurt's January 2026 evaluation ran thirteen llama.cpp quantisation schemes over Llama-3.1-8B-Instruct against GSM8K, HellaSwag, IFEval, MMLU, TruthfulQA and WikiText-2 perplexity, and the shape of the result in the table below is stable enough to plan against.

Q4_K_M gives up 0.32 average points against the F16 baseline for a file 69% smaller. Q3_K_S gives up 3.98, and the interesting part is where that loss lands: 9.32 points on GSM8K, while HellaSwag moves 0.64. Commonsense multiple choice barely notices three-bit weights. Multi-step arithmetic falls off a cliff.

That asymmetry maps onto workloads directly, and it is the whole basis of the rule we apply. Retrieval, classification, extraction and summarisation are commonsense-shaped tasks, and they tolerate four-bit weights without an argument. Anything that chains arithmetic breaks first, and it breaks quietly, returning a confident wrong number instead of an error. So a model doing arithmetic on production data does not run below five bits, and the eval harness carries at least one arithmetic benchmark built from your own domain before anybody signs off on a quantisation level.

Perplexity on its own will not catch this. Q4_K_M and Q3_K_M sit 0.40 apart on WikiText-2 and 4.25 apart on GSM8K.

One hardware fact narrows the choice before accuracy ever enters it. vLLM documents a 2x reduction in model memory and up to a 1.6x throughput improvement from FP8, but supports FP8 W8A8 only on NVIDIA GPUs with compute capability 8.9 or higher, which means Ada Lovelace and Hopper. The A100 is compute capability 8.0. If the fleet you already own is A100s, FP8 is off the table whatever the accuracy data says, and the route runs through INT8, AWQ or GPTQ instead.

The same 32B model at five sizing points

The table below puts one model on five hardware configurations, computed from published specifications rather than measured on a bench. vLLM's --gpu-memory-utilization default of 0.92 governs how much of each card the engine may claim for weights, activations, graph capture and cache combined.

Two assumptions are stated rather than buried: 2 GB per card for activations and CUDA graph capture, and a 4,096-token working session costing 1 GiB of cache. Both move under real traffic. Treat the session counts as an order of magnitude, not a service level.

None of this is a benchmark. It is division.

Read the last two rows against each other, because that comparison is the argument of this whole post. Two H100 SXM cards at bfloat16 carry roughly 72 sessions and draw up to 1,400 W. One H100 NVL running FP8 weights carries roughly 48 and draws 350 to 400 W. Quantising bought two thirds of the concurrency at a quarter of the power, and it did it purely by handing the cache the memory the weights had been sitting in.

# One H100 NVL, FP8 weights quantised at load time, 8k context.
# vLLM logs the number of KV cache blocks it allocated at startup —
# that line is the only sizing number on this page that is not an estimate.
vllm serve Qwen/Qwen3-32B \
  --quantization fp8 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype auto

Retrieval offline is easy. Access control offline is not.

Running retrieval without a cloud is a solved engineering problem: the embedding model shares the same GPUs, the vector store runs on ordinary servers, and no document leaves the site. Access control is where offline deployments actually fail, and they fail in a specific, repeatable way.

The failure mode is post-filtering. A naive pipeline fetches the top 50 chunks by vector similarity and then strips the ones the user is not cleared for. Qdrant's own write-up on filtering states the cost directly: apply a low-cardinality filter after the search, and you often end up discarding a large portion of the results the vector search returned. What a search engineer reads as a throughput problem is, on a permissioned corpus, a correctness problem. The user gets a thin, apparently broken answer, and the pressure to loosen the filter starts the same week.

The position we hold here is unpopular with anyone who likes batch jobs. The permission decision belongs inside the query, evaluated against live directory group membership, never against a nightly copy of an access-control table. A drawing reclassified at 09:00 has to be unreachable at 09:01, and in a plant the documents that get reclassified mid-morning are precisely the ones worth protecting. That costs a directory lookup per query. Pay it.

Three behaviours belong in the manuals you read before building that layer. All three are counterintuitive.

• Elasticsearch combines document-level security queries across a user's roles with OR. Its documentation is explicit that if one role grants access to an index without document level security and another grants access with it, DLS is not applied and that user sees every document in the index.
• Relevance scores are computed without the role query. Elastic states that DLS does not affect the global index statistics relevancy scoring uses, and that a restricted user could still learn field names and terms existing only in documents they cannot read.
• Qdrant uses a filterable vector index that builds additional links between the remaining points, so a permission filter there costs throughput rather than silently costing recall.

{
  "indices": [
    {
      "names": [ "plant-documents" ],
      "privileges": [ "read" ],
      "query": {
        "template": {
          "source": "{ \"terms\": { \"acl_groups\": {{#toJson}}_user.metadata.groups{{/toJson}} }}"
        }
      }
    }
  ]
}

Where to go next

• GPU sizing for LLM inference: buy the right number of cards
• Sovereign private AI: open weights deployed in your own data centre
• Enterprise RAG platform with permission-aware retrieval
• Permission-aware retrieval: four ways the entitlement model breaks