Why most "chat with your docs" bots fail

A business AI agent is only as good as the documents it can find when a question arrives. A general-purpose model has never seen your HR policy, price list or incident reports, so it fills the gaps with plausible guesses.

Picture an employee asking the internal assistant: "How many vacation days can I carry over into next year?" A bare model replies "typically five days". The answer sounds confident and reasonable, and it is wrong if your policy says ten.

In support, sales or legal work, answers like that cost money and trust. Retrieval-augmented generation (RAG) fixes the problem by fetching the relevant passages first and making the model answer from them. This article walks the full path: indexing, retrieval, generation, the step from pipeline to agent, and how to prove the result works.

What RAG is, in plain terms

RAG splits the work into two pipelines: an offline one that indexes your documents and an online one that answers questions from that index. The idea comes from a 2020 paper by Lewis et al. at Facebook AI Research. It paired a neural retriever over Wikipedia with a text generator and beat models that relied on their weights alone on knowledge-intensive tasks (Lewis et al., 2020).

Offline pipeline (runs whenever a document changes):

  1. Your documents → parse and clean
  2. Split into chunks with metadata
  3. Compute embeddings and write them to the index (vectors + full text)

Online pipeline (runs for every user question):

  1. User question → hybrid search over the index, filtered by access rights
  2. Rerank the top 20 candidates
  3. Top 5–8 chunks → the LLM writes an answer with citations

The top lane runs whenever a document changes. The bottom lane runs for every question and reads only from the index, filtered by the user's access rights.

The business case is practical rather than academic. Your knowledge changes every week, and re-indexing a document takes seconds, while fine-tuning a model takes days and still cannot point to a source. RAG also lets you enforce access rights at retrieval time, so the model never sees a passage the user is not allowed to read.

Step 1: Indexing — parse, enrich, chunk

Retrieval quality is decided mostly at indexing time, long before anyone types a question. Three things matter: clean text, useful metadata and sensible chunks.

Parsing. PDFs, slide decks and scanned contracts lose their structure when flattened into plain text. Keep headings, list numbering and tables, because "Section 4.2" is often exactly what makes a passage findable. Convert tables to Markdown or to one sentence per row, or the numbers end up separated from their column names.

Metadata. Every chunk should carry its source, section path, last-updated date, owner and access group. Metadata powers filters ("only policies valid in 2026"), citations and permission checks.

Chunking. Split by structure first (headings, paragraphs), then by size, with a small overlap so a sentence cut at the border survives in both neighbours. A few hundred tokens per chunk with 10–15% overlap is a reasonable starting point; tune it against your evaluation set rather than trusting any default.

from dataclasses import dataclass
 
@dataclass
class Chunk:
    text: str
    source: str
    section: str
    updated: str
    acl_group: str
 
def chunk_document(doc, max_chars=2000, overlap=200):
    chunks = []
    for section in doc.sections:          # split by headings first
        text, start = section.text, 0
        while start < len(text):
            end = min(start + max_chars, len(text))
            chunks.append(Chunk(
                # prepend the breadcrumb so the chunk makes sense alone
                text=f"{doc.title} > {section.path}\n\n{text[start:end]}",
                source=doc.url,
                section=section.path,
                updated=doc.updated_at,
                acl_group=doc.acl_group,
            ))
            if end == len(text):
                break
            start = end - overlap
    return chunks

Note the breadcrumb prepended to each chunk. A passage that says "the limit is 30 days" is useless on its own; prefixed with "Refund Policy > EU customers", it answers a real question.

Step 2: Retrieval — hybrid search, reranking, context

Pure vector search is not enough for business documents. Embeddings capture meaning well but blur exact tokens: SKU codes, error IDs, clause numbers and product names. Keyword ranking in the BM25 family catches those exactly (Robertson & Zaragoza, 2009), so production systems run both searches and merge the results.

The standard merge is Reciprocal Rank Fusion (RRF). Each document scores the sum of 1 / (k + rank) across the result lists, with k = 60 in the original paper (Cormack et al., 2009). RRF needs no score calibration between the two engines, which makes it a safe default.

Here is the whole thing in PostgreSQL with the pgvector extension. Postgres full-text ranking stands in for BM25 here; search engines such as Elasticsearch or OpenSearch use BM25 natively.

-- $1 = query embedding, $2 = query text, $3 = user's access groups
WITH semantic AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rnk
  FROM chunks
  WHERE acl_group = ANY($3)
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, ROW_NUMBER() OVER (
           ORDER BY ts_rank(tsv, plainto_tsquery('english', $2)) DESC) AS rnk
  FROM chunks
  WHERE tsv @@ plainto_tsquery('english', $2)
    AND acl_group = ANY($3)
  ORDER BY ts_rank(tsv, plainto_tsquery('english', $2)) DESC
  LIMIT 50
)
SELECT c.id, c.text, c.source,
       COALESCE(1.0 / (60 + s.rnk), 0)
     + COALESCE(1.0 / (60 + k.rnk), 0) AS rrf_score
FROM semantic s
FULL OUTER JOIN keyword k ON s.id = k.id
JOIN chunks c ON c.id = COALESCE(s.id, k.id)
ORDER BY rrf_score DESC
LIMIT 20;

Notice that the access filter sits inside both subqueries. Permissions are enforced by the database, not by asking the model to keep secrets.

The top 20 candidates then go to a reranker: a cross-encoder that reads the question and each passage together and reorders them by relevance (Nogueira & Cho, 2019). Only the best five to eight passages reach the language model.

One more cheap upgrade is contextual retrieval. Before indexing, an LLM writes a one- or two-sentence summary of where each chunk sits in its document, and that summary is prepended to the chunk for both embedding and keyword search. In Anthropic's tests this cut the top-20 retrieval failure rate by 35% with contextual embeddings alone, by 49% with contextual BM25 added and by 67% with reranking on top (Anthropic, 2024).

Step 3: Generation — answer from sources or say "I don't know"

The generation prompt has one job: make the model treat the retrieved passages as the only source of truth. That takes three rules: numbered sources, a citation for every claim and explicit permission to refuse.

You are the internal assistant for Acme Corp.
Answer ONLY from the sources below. Cite every claim as [n].
If the sources do not contain the answer, reply
"I couldn't find this in our documentation" and suggest who to ask.
If sources conflict, prefer the most recently updated one
and mention the conflict.
 
<sources>
[1] HR Policy > Leave > Carry-over (updated 2026-03-01)
    Employees may carry over up to 10 unused vacation days...
[2] HR FAQ > Leave (updated 2024-11-15)
    Up to 5 days can be carried over...
</sources>
 
Question: How many vacation days can I carry over?

With this prompt, the model answers "up to 10 days [1]" and flags that the older FAQ [2] still says five. That second sentence is valuable: it tells the document owner what to fix.

The order of passages matters as well. Liu et al. showed that models use information at the start and end of a long context far better than in the middle (Liu et al., 2023). Put the strongest passages first, and resist stuffing 40 chunks in "just in case": more context often means worse answers, not better ones.

Citations double as a product feature. When users can click through to the exact paragraph, they trust correct answers and catch wrong ones themselves.

From pipeline to agent

A RAG pipeline answers one question with one search; an agent decides what to search, where, and whether it already has enough. In practice the step up means giving the model retrieval as a tool it can call repeatedly, alongside a few others.

A typical business agent needs only a handful of tools:

  • search_docs(query, filters) — the hybrid retrieval above, with metadata filters the model fills in itself.
  • get_document(id) — the full document, for when a chunk is not enough context.
  • query_crm(customer_id) — structured facts that should never be guessed from text.
  • create_ticket(...) — an action with side effects, executed only after the user confirms.

Here is how the loop looks on a real support question:

User:   Can customer #4471 still get a refund on the March order?
 
Agent -> query_crm(4471)
         EU customer, order placed 2026-03-12, Pro plan
Agent -> search_docs("refund window EU customers", {doc_type: "policy"})
         [1] Refund Policy > EU: refunds within 30 days of purchase
Agent -> search_docs("refund exceptions Pro plan")
         no relevant passages
 
Agent:  No. The 30-day refund window for EU customers closed
        on April 11 [1]. I found no exception for the Pro plan;
        the billing team can approve one manually.

The agent combined structured data with policy text and ran a second search when the first one left a gap. It also admitted what the documents do not cover, instead of inventing an exception.

Research points in the same direction. Self-RAG trains models to decide when to retrieve and to critique their own drafts (Asai et al., 2023). For questions that span a whole corpus, such as "what themes recur across all incident reports?", GraphRAG builds a knowledge graph with community summaries instead of fetching isolated chunks (Edge et al., 2024). To connect tools across systems without custom glue code, the Model Context Protocol provides an open standard.

One rule never changes: the agent acts with the user's permissions, not its own. Every tool call carries the user's identity, and every filter is applied in code.

Step 4: Measure before you ship

RAG fails silently: a wrong answer with a citation looks exactly like a right one. So before launch, build a golden set of 100–200 real questions, each with the expected answer and the document that contains it. Pull them from support tickets, chat threads and search logs, not from imagination.

Then measure the two layers separately, because a bad answer can come from either one:

Layer Metric Question it answers
Retrieval Recall@k Is the right chunk among the top k results?
Retrieval MRR How high does the first right chunk rank?
Generation Faithfulness Is every claim supported by the retrieved sources?
Generation Answer relevance Does the answer address the actual question?
End to end Refusal accuracy Does it say "I don't know" when the docs are silent?

Frameworks such as RAGAS automate faithfulness and relevance scoring with an LLM as the judge (Es et al., 2023). Spot-check the judge against a few dozen human labels before trusting it. Rerun the full set on every change to chunking, embeddings, the reranker or the prompt, and keep the results so you can compare versions.

Common mistakes and a launch checklist

Most failed RAG projects fail on data and process, not on the model. The usual suspects are stale or duplicate documents that contradict each other, tables flattened into noise and access rights checked in the prompt instead of the query. The most expensive one is having no golden set, so every change is judged by gut feeling.

Before you open the agent to users, check that:

  • Every document has an owner and an update date, and outdated versions are removed from the index
  • Every chunk carries its source, section path and access group
  • Hybrid search with reranking has been tested against vector-only search on your own questions
  • The prompt requires citations and explicitly allows "I don't know"
  • A golden set of 100+ real questions runs on every change
  • Questions, retrieved chunks and user feedback are logged for review

Start small: one department, one document set, one golden set. A narrow agent that cites its sources and admits its gaps earns more trust than a broad one that guesses, and that trust is what lets you expand it later.

Once the pipeline works, three topics decide whether it survives production: how you measure quality (How to evaluate LLM applications), how you connect the agent to business systems (MCP for business) and what it costs to run (LLM cost optimization). Retrieved documents are also untrusted input; see AI agent security.

To go deeper into the retrieval layer, see embeddings and chunking and vector databases compared; for the decision between retrieval and training, see fine-tuning vs RAG vs prompting.

Sources

  1. Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
  2. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
  3. Cormack, Clarke & Büttcher (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009.
  4. Nogueira & Cho (2019). Passage Re-ranking with BERT.
  5. Anthropic (2024). Introducing Contextual Retrieval.
  6. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts.
  7. Asai et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.
  8. Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization.
  9. Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation.
  10. pgvector — open-source vector similarity search for PostgreSQL.
  11. Model Context Protocol — open standard for connecting AI agents to tools and data.