Skip to main content
The Conversation Quality metric evaluates user satisfaction across an entire chatbot session by analyzing tone, engagement, and sentiment. It classifies each conversation as GOOD or BAD depending on whether the user’s overall experience reflects positive engagement or frustration directed at the bot. The metric focuses on conversational flow rather than task success, emphasizing how naturally and politely the user and bot interact. It excludes non-textual or purely action-based agent outputs (e.g., button clicks). This is a boolean metric that indicates whether the conversation quality is good.

​
Conversation Quality at a glance

​
When to use this metric

​
How to improve Conversation Quality scores

Some techniques to improve Conversation Quality scores are:
  • Ensure bots provide clear, empathetic, and concise responses
  • Detect and mitigate repeated clarification loops
  • Train models to de-escalate external frustration effectively
  • Log complete sessions to allow accurate tone assessment
Common issues that can cause low scores are:
  • Mislabeling external frustration as bot-directed
  • Incomplete logs
  • Abrupt session truncation

​
Performance Benchmarks

We evaluated Conversation Quality against human expert labels on an internal dataset of agentic conversation samples using top frontier models.

​
GPT-4.1 Classification Report

Benchmarks based on internal evaluation dataset. Performance may vary by use case.
If you would like to dive deeper or start implementing Conversation Quality, check out the following resources:

​
Examples

  • Conversation Quality Examples - Log in and explore the “Conversation Quality” Log Stream in the “Preset Metric Examples” Project to see this metric in action.

​
How-to guides