Designing a knowledge base RAG pipeline that actually works
RAG (retrieval-augmented generation) is the right approach for grounding AI answers in your content. The gap between "RAG works in a demo" and "RAG works in production" is mostly about chunking, reranking, and faithfulness scoring.
Why RAG matters for customer support
A general-purpose LLM does not know your product. It will hallucinate product-specific answers confidently. The fix: retrieve relevant content from your knowledge base before generating a response, and constrain the model to only use what it retrieved.
This is retrieval-augmented generation. It sounds straightforward. The production failure modes are subtle.
Chunking
How you split documents before embedding them determines retrieval quality more than the choice of embedding model.
The naive approach — split every 512 tokens — breaks mid-sentence and loses context. Better: split on semantic boundaries (paragraphs, numbered steps, headings). For FAQ documents, each Q&A pair is its own chunk.
We also add a small context prefix to each chunk: the document title and the heading it came from. This gives the embedding more signal about what the chunk is about, without bloating the chunk itself.
Reranking
A two-stage retrieval pipeline: vector search retrieves the top-20 candidates, a cross-encoder reranks them to the top-5. The reranker sees the query and each candidate together — much more accurate than similarity alone. We use a small cross-encoder model that runs in under 50ms.
Faithfulness scoring
After generating a response, a judge model checks each sentence against the retrieved sources. Sentences with no source attribution are flagged. The score is logged and surfaced to tenants in their analytics dashboard — they can see, per AI response, how grounded it was.
When faithfulness drops below a threshold, the response is withheld and a human escalation is triggered. The AI does not guess; it escalates.
What actually breaks in production
Short queries. "Track my order" retrieves the wrong chunk because it has low semantic specificity. We handle this with query expansion: the model rewrites the query into a more specific version before retrieval.
Multilingual content. A knowledge base in English does not retrieve well for Arabic queries, even with a multilingual embedding model. We embed bilingual documents and index Arabic content separately. Queries are language-detected and routed to the right index.
Stale content. Tenants update their pricing pages and FAQs. We run a re-embedding job nightly and on every knowledge base update. Staleness is surfaced in the dashboard as a "knowledge base last updated" timestamp.
The practical rule: measure retrieval recall, measure faithfulness, and treat them as product metrics — not as one-time evaluations you run at launch.