Beyond the chatbot demo: running an internal AI assistant in production

The question that brings people to this article is usually: we built an internal assistant in a fortnight and it was impressive — why has it taken six months to get it into real use and why does nobody trust it? The answer is that a demo and a production system are measured on different things, and almost none of the difficult work is visible in the demo.

A demo is judged on whether a handful of chosen questions produce good answers in front of an audience. Production is judged on whether an arbitrary question from an arbitrary employee, about a document that changed last Tuesday, that they may or may not be permitted to see, produces an answer they can act on — and whether they can tell when it cannot.

Five things separate the two: retrieval quality on the long tail, permissions, evaluation and monitoring, cost at real volume, and adoption. Each is ordinary engineering and organisational work. None of it is glamorous, and skipping any one of them is what produces a well-liked pilot that nobody uses.

Users blame the model when the answer is wrong. In our experience the model is rarely the problem — it faithfully answered using the passages it was given, and the passages were the wrong ones. The assistant is a retrieval system with a language interface on top, and most quality work is retrieval work.

Where retrieval breaks in real corpora

• Stale and superseded documents. Three versions of the same policy exist and nothing marks which is current. The system confidently cites the 2023 one.
• Naive chunking. Fixed-size splits cut tables in half and separate a clause from the heading that gives it meaning.
• Vocabulary mismatch. Staff ask in the words they use; documents are written in formal or legacy terminology. Pure semantic search handles some of this and misses product codes, contract numbers and acronyms — which is why hybrid keyword-plus-vector retrieval is usually necessary rather than optional.
• Questions spanning documents. The answer requires combining a policy, a rate card and an exception note. Single-pass retrieval returns one of the three.
• Unanswerable questions. The corpus simply does not contain the answer, and the system has no path to say so.

That last one deserves emphasis. An assistant that answers everything is less useful than one that answers eighty per cent and reliably declines the rest, because the second can be trusted. Design the refusal path — a threshold on retrieval confidence, a stated “this is not covered by the documents I have”, and a route to a human — as a feature, not an error state. The enterprise RAG implementation guide covers the pipeline mechanics in more depth.

This is the issue that most often stops a pilot at the security review, and it is structural rather than incidental. A demo indexes a shared drive. Production must respect the fact that not everyone may see everything on that drive — and a retrieval system flattens exactly the access boundaries that file permissions were maintaining.

Worse, the leak is subtle. The assistant does not hand over the restricted document; it summarises its contents in a fluent paragraph with no filename attached. Conventional access logging sees nothing unusual.

What a defensible design requires

1. Permissions captured at ingestion. Every chunk carries the access control metadata of its source document, synchronised as permissions change rather than snapshotted once.
2. Filtering at query time, before generation. The candidate set is restricted to what this user may see. Filtering after retrieval, or instructing the model not to mention restricted content, is not a control.
3. The user's identity, not a service account. If the assistant queries the index with broad system credentials, every user effectively has administrator reach.
4. Deletion and revocation that propagate. When a document is deleted or access is withdrawn, the embeddings must follow within a defined window.
5. Audit at the answer level. Log which sources informed which answer for which user, so that a question about exposure has a factual answer.

The pragmatic route for a first release is to scope the corpus deliberately: index only material that is genuinely readable by the whole intended audience, ship that, and add permission-scoped sources as a second phase with the machinery above in place. That is a much better trade than delaying indefinitely — or shipping something that quietly redistributes salary letters. In finance, healthcare and public sector settings, treat it as a hard gate.

Assistants degrade quietly. The corpus drifts, a supplier updates a model, someone edits a prompt to fix one complaint and breaks four other behaviours. Without measurement, the first signal is people gradually stopping using it — which arrives months late and cannot be diagnosed.

The offline set

Maintain 100 to 200 real questions with expected sources and acceptable answers, built with the people who do the work. Include unanswerable questions where declining is the correct behaviour. Score three things separately, because they fail for different reasons: did retrieval return the right source, is the answer factually supported by what was retrieved, and did it decline appropriately when it should have.

Run it before every change to a prompt, model, chunking strategy or retrieval parameter. This is a regression suite and should be treated with the same discipline as any other — which in practice means it lives in your pipeline, not in someone's notebook.

The live signals

• Thumbs up and down with an optional reason — low response rates still surface the worst failures.
• Questions that returned no confident source, clustered by topic. This is your content gap backlog and often the most valuable output of the whole system.
• Repeat and rephrase rate — a user asking three variants did not get their answer.
• Abandonment: sessions opened and left without a follow-up action.
• Latency at the ninety-fifth percentile, and cost per conversation, tracked together.

Assign someone to read a sample of real conversations weekly. Fifteen minutes of reading actual transcripts consistently reveals more than a dashboard, particularly in the first quarter.

Pilot economics mislead in both directions. Twenty enthusiasts asking careful questions produce a bill nobody notices. Two thousand employees, some of whom paste entire documents in, produce a different picture — and the driver is usually architecture rather than per-token price.

• Context size. Retrieving twenty passages when four would do multiplies the cost of every request. Better retrieval is a cost programme as much as a quality one.
• Conversation history. Naively re-sending the full transcript makes turn ten several times more expensive than turn one. Summarise or window it.
• Re-embedding. Changing the embedding model means reprocessing the whole corpus. Budget for it as a periodic event, not a surprise.
• Model tiering. Routing, classification and simple lookups do not need the frontier model that hard synthesis does.
• Caching. A large fixed system prompt sent on every call is a good candidate for provider-side caching where available; identical repeat questions can often be served from your own cache.

Model the cost per conversation early and put it on the same dashboard as usage. A system whose unit cost is understood can be scaled deliberately; one whose bill is a monthly surprise gets throttled in a panic, usually just as adoption is building.

A technically sound assistant with no adopted workflow is a cost centre. Adoption is not a launch email; it is a deliberate programme with an owner, and it usually needs more calendar time than the build did.

1. Start with one team and one job. “Ask anything” gives people no idea where to start and guarantees an unflattering first impression. Pick a specific recurring task with a measurable current cost.
2. Set the boundary explicitly. Tell people what it does know, what it does not, and how confidently to treat it. Expectations set honestly at launch buy enormous patience later.
3. Show sources always. Citations that open the underlying document turn a black box into a faster search. This single feature does more for trust than any accuracy improvement.
4. Meet people where they work. An assistant in the tool staff already have open is used; one behind a separate login and a bookmark is not.
5. Close the feedback loop visibly. When someone reports a bad answer and a fix follows with a note, reporting continues. When feedback disappears, it stops within weeks.
6. Name an owner for the corpus. Retrieval quality is content quality. Someone must own removing superseded documents and filling the gaps that the no-answer log reveals.

A hypothetical illustration of the sequencing, offered as an example rather than a case study: a support team is the first cohort, using the assistant only for questions about a documented product policy set. Answers cite the clause. The team lead reads a sample of transcripts weekly and files gaps. After a month the no-answer log has driven a dozen documentation fixes, and only then does a second team join. Slower on paper, and considerably faster to real use than a company-wide launch.

Before an internal assistant goes beyond a pilot cohort, we would want honest answers to all of these. Anything unanswered is a scoping decision, not a detail to settle later.

1. Is every indexed document one that every user of the assistant may lawfully read?
2. If not, are permissions applied at query time using the requesting user's identity?
3. Do deletions and access revocations propagate to the index within a defined window?
4. Does every answer show the sources it used, linked and openable?
5. Can the system decline, and does it, when retrieval confidence is low?
6. Is there an offline evaluation set, and does it gate changes before release?
7. Are no-answer questions logged, clustered and routed to someone who owns the content?
8. Is cost per conversation measured and visible next to usage?
9. Is the model version pinned, with a tested route for provider updates?
10. Is there a named owner for the corpus and a named owner for the system?
11. Is a human review requirement defined for the categories of answer where it matters?
12. Does the rollout name a first cohort, a first job to be done, and a measure of whether it worked?

Most stalled assistants we are asked to look at fail on the same three: no evaluation set, no permissions model, and no owner for the content. Those are answerable in weeks — but only once they are named as the actual problem rather than treated as a model-quality complaint. It is the ongoing operational side, closer to managed services than to a build project, that decides whether the thing is still in use a year on.