Build a RAG system in the order that makes it answer well

The claim: retrieval quality dominates model choice

A RAG system (retrieval-augmented generation) connects a vector search to a language model, so your assistants answer from your own documents instead of from the model's training data. Before the model writes a word, the system finds the relevant passages, hands them over with the question, and the model answers from that context. The architecture is settled; the engineering is in the retrieval layer.

This guide is the complete build, in the order the work actually runs: data preparation, chunking, embeddings, the vector database, retrieval, evaluation, and the production rollout with permissions and monitoring. Each section names the decision, what the choice changes, and the failure mode you are buying if you skip it.

Our position, stated plainly so it can be argued with: on a well-built retrieval pipeline, a mid-size open-weight model beats a frontier model bolted onto bad retrieval. The failures users notice in enterprise RAG — the assistant answering from the wrong document, or refusing to answer from the right one — are retrieval failures wearing a model failure's clothes, and the fix is almost never a bigger model.

The pipeline

Six steps, and the order is the method

1. Prepare the data before you chunk a single document. The retrieval layer inherits every defect of the corpus. Deduplicate, normalise formats, decide what belongs in the index at all, and record the provenance of each document. Garbage in, plausible-sounding garbage out.
2. Chunk by meaning, not by character count. The chunk is the unit the model reads, so its boundaries decide what the search can find. Fixed-size chunks are the floor, recursive and semantic strategies the next step up — the comparison table below scores them against document types.
3. Embed with a model matched to your language and your domain. The embedding model turns chunks into vectors, and the distance between vectors is what search ranks. Match the model to your language and your document domain; a general model on technical documents retrieves differently from one tuned on them.
4. Choose the vector store on operations, not on benchmarks. The database holds the vectors and answers the similarity search. The decision that matters is managed versus self-hosted, because it decides where your data lives and who operates the search layer — then hybrid search for the retrieval itself.
5. Evaluate before you celebrate. A RAG system without a test set is a demo. Build a set of real questions and golden answers, measure faithfulness, context recall and context relevance, and record the numbers. The metrics table below is the minimum set.
6. Roll out with permissions, monitoring and an owner. Permission-aware retrieval — the index knows who may read what — is what turns the assistant into something a compliance officer signs off. Monitoring and an owner come with it; an unowned assistant is a liability with a chat window.

The vector store and the embeddings are one decision, not two

The vector database and the embedding model are usually chosen separately and regretted together. The store decides scaling, hybrid search and the operations burden; the embedding model decides what the search means. The pairing matters because hybrid search — keyword and vector together — is where most production systems land, and it constrains both choices.

The managed-versus-self-hosted question is the one to decide first, because it is the data-sovereignty question. A self-hosted vector store on your own infrastructure keeps the corpus inside the estate, which is the difference between a defensible DSGVO story and a subprocessor list. The cost is operations: a vector database is infrastructure with an owner, backup and a recovery drill, not a library you import.

On the embedding side, match the model to your language and your domain. The sibling site's own comparison of German-language embedding models found the gap between a tuned model and a generic one is real and measurable on domain documents — the same logic applies to any industry vocabulary. Start with the strongest model for your language, then evaluate whether a cheaper one holds the retrieval quality.

Executable artefact

The minimal pipeline, in the shape a prototype should take

Versions and providers are placeholders. The shape is the point: documents in, chunks out, vectors into the store, and the retrieval call that answers a question.

# Minimal RAG pipeline sketch (conceptual, not a stack recommendation)
# 1. Load and prepare
documents = load_corpus("documents/")          # dedupe, normalise, provenance
# 2. Chunk
chunks = semantic_chunk(documents)             # or recursive; measure, then pick
# 3. Embed
vectors = embed(chunks, model="your-language-model")
# 4. Store
collection = vector_store.add(vectors)         # hybrid search on, permissions tagged
# 5. Retrieve
hits = collection.search(query, top_k=8)       # permission-filtered at query time
# 6. Generate
answer = llm(context=hits, question=query)
# 7. Evaluate — before you call it done
metrics = ragas.evaluate(testset, answers)     # faithfulness, context recall, relevance

Seven numbered lines, one per pipeline step, because the evaluation line is the one most prototypes are missing.

Evaluation is a test set, three metrics and a recorded baseline

The evaluation question — is this system good enough — is answered by measurements on your own documents, and the minimum set has three metrics. Faithfulness asks whether the answer follows the retrieved context rather than inventing beyond it. Context recall asks whether retrieval found what was needed to answer. Context relevance asks whether what it found was on topic rather than merely similar.

The test set is the unglamorous half of the work. It needs real questions from the people who will use the system, golden answers you agree on, and a spread across the document types and difficulty levels the deployment will actually see. Twenty questions from the pilot users are worth more than two hundred generated from a template, because the generated set encodes the assumption that you already know what will be asked.

The baseline is what makes the metrics useful. Record the numbers before you tune anything, then change one variable at a time — chunk size, embedding model, retrieval strategy — and compare. The whole discipline collapses without the baseline, because the improvement you cannot measure is the improvement you will re-argue at the next review.

RAG metrics

The three numbers that tell you whether retrieval works

The three form a chain: recall says whether the knowledge is findable, relevance says whether the search is precise, faithfulness says whether the model stays honest with what it got. Fix them in that order.

Sources: RAGAS documentation — faithfulness, context recall, context relevance · Microsoft Learn — advanced retrieval-augmented generation patterns. Read 2026-09-02. Vendor documentation changes; verify against the current release.

Keep reading

• Permission-aware retrieval: the index that knows who may read what
• Enterprise RAG platform: retrieval your security team approves
• Service desk automation: retrieval in front of a support queue
• Company Brain hub: more retrieval field notes