How we evaluate AI chatbot response quality at scale
Automated evals are the only way to catch quality regressions when you deploy changes to an AI pipeline. Here is the lightweight eval framework we built and what it actually catches.
The problem with vibes-based evaluation
The most common way teams evaluate AI chatbot quality is: someone reads a sample of conversations and says "it looks good." This works at 10 conversations. It fails at 10,000.
When you update a prompt, swap a model, or change a retrieval strategy, you need to know — with data, not intuition — whether quality went up or down.
What we measure
We track four metrics per conversation:
Faithfulness. Does the AI response contain only information that can be traced to the knowledge base or the retrieved context? A high faithfulness score means no hallucinations. We run a lightweight judge model that checks each response against its sources.
Relevance. Does the response actually address what the user asked? We score this on a 0–1 scale using an embedding-based similarity between the user query and the response.
Tone match. Does the response match the tenant's configured tone (professional, friendly, direct)? We prompt a classifier model with the tenant's tone spec and the response.
Resolution rate. Did the conversation end with the user's issue resolved, or did it escalate to human or close unresolved? This is the lagging indicator — the others are leading.
Our eval loop
Every night, a batch job runs the previous day's conversations through the eval pipeline. Metrics are written to a conversation_evals table. A dashboard shows the rolling 7-day average per metric per tenant.
When we deploy a prompt change, we run a shadow eval against 1,000 historical conversations before promoting. If faithfulness drops more than 2 percentage points, the change is blocked.
What it actually catches
Since shipping evals: one faithfulness regression from a prompt change that made the model too confident about order statuses it couldn't verify. One relevance drop from a retrieval change that was deprioritizing short FAQ answers. Both would have shipped silently without automated evals.
The system is not perfect — a bad judge model introduces its own errors. But an imperfect metric you check regularly beats a perfect metric you never run.