You know how tweaking chatbot replies can feel like an endless loop of “Did I fix it yet?” At Instacart, they gave that grind a major boost by using large language models (LLMs) to automatically score and improve customer-support chatbot responses. Here’s the scoop:

The Pain Point

  • Manual testing is slow, expensive, and hard to scale
  • Developers spend hours iterating on tone, accuracy, and usefulness
  • Hard to catch subtle issues (like sounding too robotic or missing a key detail)

The AI-Powered Fix

  1. Define a clear rubric you care about
    • Relevance: Does the answer address the user’s question?
    • Correctness: Is the info factually accurate?
    • Brand Voice: Is it friendly, concise, and on-brand?
  2. Craft a few “example evaluations” as prompts
    • Show the LLM what a 5-star vs. 1-star answer looks like
  3. Call the LLM in CI/CD to score every new chatbot reply
    • Scores come back in seconds, not days
    • Locked in as a quality gate before changes hit production
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import openai

eval_prompt = f"""
Rate the following chatbot response from 1 (poor) to 5 (excellent)
on relevance, correctness, and brand voice. Explain each score.
User question: "{user_query}"
Chatbot response: "{bot_reply}"
"""

result = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  prompt=eval_prompt,
  temperature=0
)
print(result.choices[0].message.content)

Why It Matters

  • 5× faster feedback loops for developers
  • Human evaluators now focus on edge cases, not routine checks
  • Consistent quality and tone, even as the bot learns new skills
  • Early data shows LLM scores correlate strongly with human ratings

Real-World Wins

  • Rolled out a tricky multi-step refund flow in days instead of weeks
  • Spotted a subtle policy mismatch before it reached real customers
  • Cut external QA budget by nearly 40%

Challenges & Trade-Offs

  • LLMs can still hallucinate or misapply rubrics
  • API costs add up—optimize prompts and batch requests
  • Need guardrails to catch weird edge-case behaviors

So What Does This Mean for You?

  • If you’re building a chatbot (or any text-generator), think beyond just generating replies.
  • Use AI to evaluate AI—that quick “sanity check” can save your sanity.
  • Start small: pick one metric, build a simple rubric, and watch how fast your team iterates.

Resources & References