A RAG AI assistant for company knowledge answers employees’ questions from your own documents, with a citation for every answer. RAG (retrieval-augmented generation) works in two steps: the system first searches your policies, manuals, contracts and tickets for the passages relevant to the question, then a language model writes an answer using only those passages. Done well, it turns twenty years of shared drives into something people can ask.
This guide explains how a production RAG assistant works, the decisions that make it accurate or useless, how to keep it within your permissions and Canadian privacy obligations, and what a realistic project looks like. It is written for leaders deciding whether an internal assistant is worth building, and for the IT teams who will have to run it.
What is a RAG AI assistant, and why not just use ChatGPT?
A general chatbot answers from what its model learned during training. It doesn’t know your travel policy, your product’s warranty terms or the procedure your plant follows for a lockout. Pasting documents into a chat window works for one-off questions but doesn’t scale, doesn’t respect permissions and leaves no record.
A RAG assistant connects the model to your knowledge in a controlled way:
- Current. When a policy changes, the index is updated and the next answer reflects it. No retraining.
- Cited. Every answer links to the passages it used, so people can check and trust it.
- Permission-aware. Users only get answers built from documents they could already open.
- Private. It runs in your cloud account, on private model endpoints that do not train on your data.
How does retrieval-augmented generation work?
A production assistant has two pipelines: one that prepares your knowledge, and one that answers questions.
The ingestion pipeline
- Connect sources. SharePoint, Google Drive, Confluence, a document management system, a ticketing tool, a CRM knowledge base. Each connector also reads permissions and change history.
- Clean and parse. Convert PDFs, Word files, slides and scans into clean text, keeping headings, tables and page numbers. Drop duplicates and superseded versions.
- Chunk. Split documents into passages that make sense on their own, usually by section rather than a fixed number of words, and attach metadata: title, date, owner, language, permissions.
- Index. Store each chunk with a vector embedding (for meaning) and keywords (for exact terms like part numbers and policy codes).
The question pipeline
- Understand the question. Rewrite follow-up questions into standalone ones and detect language.
- Retrieve. Run hybrid search (semantic plus keyword), filtered by the user’s permissions and by metadata such as region or product line.
- Rerank. A second model reorders the top results by relevance, so the best passages reach the model.
- Generate. The model answers from the retrieved passages only, cites them and says plainly when the answer isn’t in the sources.
- Log. The question, the passages retrieved, the answer and any user feedback are stored for evaluation and audit.
RAG, fine-tuning or long context: which should you use?
| Approach | Good for | Weak at | Use for company knowledge? |
|---|---|---|---|
| RAG | Large, changing document sets; citations; per-user permissions | Questions needing whole-corpus analysis | Yes, the default |
| Fine-tuning | Style, tone, output formats, narrow classification | Storing facts, updating, citing, permissions | Rarely, and alongside RAG |
| Long context (paste everything) | A few long documents, one-off analysis | Cost per question, permissions, scale | For small, static sets only |
Large context windows have made it tempting to skip retrieval. For a handful of documents that’s fine. For thousands of documents with different owners and audiences, retrieval remains cheaper per question, faster and the only practical way to enforce permissions.
What makes a RAG assistant accurate?
In our experience, when a RAG assistant gives a bad answer, the model is rarely the cause. The cause is usually upstream.
Source quality
If three versions of the expense policy are on the shared drive, the assistant will find all three. Before indexing, decide which sources are authoritative, archive superseded versions and assign an owner per collection. This is unglamorous work and the most valuable thing you can do for accuracy.
Chunking and metadata
Chunks that split a table in half, or separate a rule from its exceptions, produce confident wrong answers. Chunk by document structure, keep tables whole and attach metadata so retrieval can filter by date, region, language or product.
Retrieval quality
Measure retrieval on its own: for each test question, did the top results include the passage that contains the answer? If not, no prompt will save the answer. Improve with hybrid search, reranking, better chunking or query rewriting before touching the generation step.
Faithful generation
Instruct the model to answer only from the provided passages, to cite each claim and to say “I couldn’t find this in the documents” when appropriate. Then check it: a faithfulness score on a test set tells you how often answers stay within the sources.
How do you keep a company knowledge assistant secure?
An assistant that can read everything is a data leak waiting for a curious employee. Design for security from the first sprint:
- Permission sync. Read access lists from each source and filter results at query time, so HR files stay with HR and board minutes stay with the board.
- Canadian hosting. The index, logs and application run in your cloud account in a Canadian region, and models are called through private endpoints that do not retain or train on your data. Our article on Canadian data residency for AI explains the options.
- Personal information. Identify sources containing personal information early. Under PIPEDA and Québec’s Law 25 you need to know where it flows, and a privacy impact assessment is often required.
- Audit logging. Every question, retrieved passage and answer is logged with the user’s identity.
- Prompt injection defences. Treat document content as data, not instructions, and limit what the assistant can do beyond answering.
Our security approach lists the defaults we apply on every assistant.
How do you evaluate a RAG assistant?
Build a test set before you build the assistant. Ask the people who field questions today (HR advisors, support leads, plant supervisors) for 100 to 300 real questions with the correct answer and the source document. Then measure:
| Metric | Question it answers |
|---|---|
| Retrieval recall | Did the right passage appear in the top results? |
| Answer correctness | Does the answer match what an expert would say? |
| Faithfulness | Is every claim supported by a cited passage? |
| Refusal accuracy | Does it say “not found” when the answer isn’t in the sources? |
| Latency and cost | How long and how much per answer? |
Run the suite on every change to prompts, chunking, models or sources. It turns “it seems better” into a number and stops regressions before users see them.
Where do RAG assistants pay off first?
- HR and policy questions. Benefits, leave, expenses, travel, in English and French.
- Operating procedures. Field technicians and plant staff asking about equipment, safety steps and maintenance procedures.
- Sales and product knowledge. Specs, pricing rules, compatibility and contract terms for sales and support teams.
- IT and internal support. Answers drawn from the knowledge base and past resolved tickets.
- Customer-facing support. Once internal accuracy is proven, the same pattern powers customer answers; see our guide to AI chatbots for customer service.
The best assistants live where people already work: Microsoft Teams, Slack, the intranet, the helpdesk or a CRM sidebar. Connecting to those tools is part of our systems integration work and usually matters more for adoption than any interface we could design.
A realistic timeline for a RAG assistant
| Phase | Weeks | What happens |
|---|---|---|
| Discover | 1–2 | Choose one knowledge domain, inventory sources and permissions, collect real questions with answers |
| Prototype | 3–5 | Ingest the sources, build hybrid retrieval, measure against the test set |
| Build | 6–11 | Permission sync, Teams or intranet integration, logging, security review, pilot users |
| Launch and improve | 12+ | Rollout, feedback loop, monthly quality report, source owners keep content current |
Common mistakes to avoid
- Indexing everything at once. Start with one domain and expand once quality is proven.
- No content owner. Without someone responsible for keeping sources current, accuracy decays within months.
- Hiding the sources. Users trust answers they can check. Always show citations.
- Skipping French. In bilingual organizations, test both languages from the start, including questions in one language about documents in the other.
- Measuring only satisfaction. Thumbs-up ratings are useful, but they don’t replace a test set with known answers.
Getting started
If your team spends hours answering the same questions from documents that already exist, a RAG assistant is one of the most reliable first AI projects you can choose. Our AI assistants service covers how we build them on your knowledge, in your cloud.
Describe your knowledge problem in our free AI blueprint and get a solution outline, the systems involved and a realistic timeline in about two minutes. Or book a call to talk through your sources with a senior engineer.
