Enterprise RAG: a practical implementation guide

Most enterprise RAG projects do not fail on the model. They fail because the retrieval layer returns the wrong passages, because nobody agreed what a correct answer looks like, or because the system quietly exposes documents a user was never meant to see. The demo works on twenty curated PDFs; the pilot collapses on forty thousand messy ones.

A RAG system that is done properly looks fairly unglamorous. The corpus is inventoried and permissioned before a single embedding is created. Chunking is chosen deliberately rather than left at a default. Retrieval is measured on its own, separately from generation, against a graded test set that someone in the business actually wrote. Access control is enforced at retrieval time, not by hoping the prompt behaves. Every answer is traceable back to its sources, and every change is regression-tested before it reaches users.

None of that is exotic. It is ordinary engineering discipline applied to a component that happens to be probabilistic — which is exactly the discipline that gets skipped when a proof of concept is mistaken for a product. Our LLM development and infrastructure work is largely this: turning a promising prototype into something that can be operated, audited and changed safely.

RAG grounds a language model's answers in your own content by retrieving relevant passages at query time and passing them to the model as context. It is the right choice when the knowledge changes often, when answers must cite a source, and when access to the underlying content differs by user.

Choose RAG when

• The answer lives in documents that change weekly — policies, contracts, tickets, product documentation, standard operating procedures.
• Users need to see where an answer came from before they act on it.
• Different people are entitled to see different subsets of the same corpus.

Choose fine-tuning when

• You need to change how the model behaves — tone, output format, a domain-specific classification task — rather than what it knows.
• The knowledge is stable and small, and repeating it in context on every request is wasteful.

Choose plain search when

• Users mostly want the document itself, not a synthesised answer. A good search interface with filters and previews is cheaper, faster and easier to trust.
• The cost of a plausible-but-wrong sentence is higher than the cost of one more click.

These are not exclusive. Many production systems fine-tune a small model for routing or formatting, retrieve for knowledge, and fall back to search results when confidence is low.

Ingestion

Ingestion is where most of the effort really goes. You need connectors to the systems that hold the content, reliable extraction for each file type, and a way to handle updates and deletions so the index does not drift away from reality. Treat ingestion as a scheduled data pipeline with monitoring and replay, not a one-time script — the same expectations you would place on any other production data flow.

Chunking

Chunk boundaries decide what retrieval can possibly return. Split on document structure — headings, clauses, sections — before falling back to fixed token windows, keep a modest overlap so a sentence spanning a boundary is not lost, and carry metadata (source, section path, effective date, owning department, permission labels) on every chunk. That metadata is what later makes filtering, citation and access control possible.

Embeddings and the vector store

Pick an embedding model that matches your languages and your domain vocabulary, and pin the version — changing it silently invalidates every vector you have stored. The vector store should support metadata filtering and hybrid search, because pure semantic similarity is poor at exact identifiers, product codes and clause numbers, which is precisely what enterprise users search for.

Retrieval and re-ranking

Combine keyword and vector retrieval, fetch a generous candidate set, then re-rank with a cross-encoder or a similar scoring stage before selecting the handful of passages that will reach the model. Re-ranking is usually the single highest-return improvement available once a baseline exists, and it is far cheaper than swapping the generation model.

Generation

Constrain the model: answer only from the supplied context, cite sources, and say plainly when the context does not contain the answer. An honest “not found in the available documents” is a feature, and users trust the system more once they have seen it decline to guess.

The defining security question in enterprise RAG is simple: can the retrieval layer return a passage the requesting user is not entitled to read? If permissions are checked only in the application, or only in the prompt, the answer is yes — and a single well-phrased question can surface a salary sheet or an unsigned contract.

• Enforce at retrieval. Carry the source system's access labels onto every chunk and apply them as a hard filter on the query, executed with the calling user's identity rather than a service account.
• Re-check on change. Permissions move. Re-synchronise access metadata on the same schedule as content, and remove chunks when their source document is deleted or restricted.
• Segregate the genuinely sensitive. Some corpora — personal data, board material, regulated records — belong in a separate index with their own approval path, or out of scope entirely for the first release.
• Log for audit. Record the query, the retrieved chunk identifiers, the model and prompt version, and the response, with retention that matches your existing policy. Without this you cannot investigate a complaint or demonstrate control to an auditor.
• Decide where inference runs. Data residency, contractual restrictions on training use, and sector rules will narrow the hosting options. Settle this before you build, not during procurement review.

Prompt injection deserves specific attention: retrieved content is untrusted input, and a document can contain instructions aimed at the model. Keep tool use and write actions behind explicit authorisation rather than relying on the model to refuse.

A single “is the answer good?” score tells you nothing actionable. When an answer is wrong, you need to know whether the right passage was never retrieved or whether it was retrieved and the model ignored it — those are entirely different fixes.

Build a graded test set

Collect real questions from the people who will use the system. For each, have a subject expert record the acceptable answer and the documents that support it. A few hundred well-chosen items are worth more than thousands of synthetic ones. Keep a held-out slice that nobody tunes against, and include the awkward cases: ambiguous phrasing, questions spanning several documents, and questions the corpus genuinely cannot answer.

Retrieval metrics

• Recall at k — was a supporting passage anywhere in the retrieved set?
• Precision and rank position — how much noise did the model have to read past?
• Coverage by document type and department, to expose blind spots in ingestion.

Generation metrics

• Faithfulness — is every claim supported by the retrieved context?
• Answer completeness against the expert's reference answer.
• Correct abstention — does it decline when the context is genuinely insufficient?
• Citation accuracy — do the cited sources actually contain the claim?

Automate the run, gate every change behind it, and keep the results in version control next to the code. Model versions, prompts, chunking parameters and embedding models all change behaviour; treating an evaluation run as a build step is what makes those changes safe. It is the same reflex as any other DevOps pipeline — the artefact under test just happens to be probabilistic.

1. Data readiness. Inventory the corpus: systems, volumes, formats, owners, update frequency and permission model. Identify what is out of date, duplicated or unowned, and decide what is excluded from scope.
2. Pilot scope. Pick one user group and one question domain narrow enough that a subject expert can judge every answer. Write down what success means in their words before building.
3. Baseline first. Stand up plain hybrid search over the same corpus. If that already solves most of the need, you have saved a great deal of work and gained a benchmark either way.
4. Evaluation gate. Build the graded test set and agree the thresholds that must be met before any real user is admitted.
5. Security review. Confirm retrieval-layer access control, audit logging, data residency, retention and the handling of excluded categories. Get sign-off from whoever owns information security.
6. Human-in-the-loop launch. Release to the pilot group with citations visible and clear feedback controls, and read the feedback weekly. Early users are your best evaluation dataset.
7. Monitoring. Track latency, cost per query, retrieval failure rate, abstention rate, thumbs-down rate and volume by question type. Alert on drift, not just on errors.
8. Rollback plan. Version prompts, indexes and model choices so any one of them can be reverted independently, and define who can switch the assistant off and under what conditions.
9. Operating owner. Name the team responsible for the corpus, the evaluation set and the release cadence after launch. A RAG system without an owner degrades quietly as its content ages.

Worked through in order, this is a few weeks of deliberate effort rather than a multi-quarter programme — and it is the difference between an assistant people rely on and one they abandon after the second wrong answer. If it would help to pressure-test your own plan against it, our AI practice does exactly that kind of review.