Evaluations Guide

​

Overview

The Evaluations system enables you to assess, track, and improve the quality of AI-generated responses. By creating evaluations, you can build datasets for fine-tuning models, monitor performance trends, and ensure consistent high-quality customer interactions.

​

Evaluation Status Types

Each evaluation is assigned a status that reflects the quality of the response:

​

Creating Evaluations

​

Manual Evaluation Creation

To create a new evaluation manually:

1. Navigate to the Evaluations Dashboard
2. Click “Create New Evaluation”
3. Fill in the evaluation details:

{
  "eval_name": "Customer Refund Request Handling",
  "eval_type": "Customer Service",
  "user_message": "I want to return my order #12345. It arrived damaged.",
  "preferred_output": "I'm sorry to hear your order arrived damaged. I'll help you with the return right away. I've initiated a return for order #12345 and you'll receive a prepaid shipping label via email within 24 hours. Once we receive the item, your refund will be processed within 3-5 business days.",
  "non_preferred_output": "You need to go to our website and fill out the return form.",
  "eval_status": "Outstanding",
  "description": "Exemplary handling of damaged product return with empathy and clear next steps"
}

​

Evaluation Types

Choose the appropriate type for your evaluation:

• Customer Service: General customer inquiries and support
• Technical Support: Technical issues and troubleshooting
• Sales: Sales-related interactions and inquiries
• Product Support: Product-specific questions and guidance
• General: Other types of interactions

​

Building Effective Test Sets

To create robust evaluations, build test sets that represent real-world scenarios:

• Golden Test Sets: Curate 50-100 high-quality examples with expected responses.
• Diverse Scenarios: Include in-scope queries, out-of-scope small talk, and adversarial questions.
• Synthetic Data: Generate variations using LLMs to expand coverage, including edge cases and bias tests.
• Iteration: Continuously update based on production data and user feedback.

Use these test sets to ensure comprehensive evaluation coverage.

​

Managing Evaluations

​

Dashboard Features

The Evaluations Dashboard provides comprehensive tools for management:

​

Types of Evaluation Methods

Enhance your evaluation process using different methods:

Combine these methods for comprehensive quality assurance.

​

Performance Statistics

Monitor your evaluation metrics through dashboard cards:

• Total Evaluations: Overall count of all evaluations
• Outstanding: Count of exceptional responses
• Needs Review: Evaluations requiring further assessment
• This Week: Recent evaluation activity

Each metric includes trend indicators showing performance changes over time.

​

Exporting for Model Training

​

JSONL Export Format

Evaluations can be exported in JSONL format for model fine-tuning:

{
  "input": {
    "messages": [
      {
        "role": "user",
        "content": "I want to return my order #12345. It arrived damaged."
      }
    ],
    "tools": [],
    "parallel_tool_calls": true
  },
  "preferred_output": [
    {
      "role": "assistant",
      "content": "I'm sorry to hear your order arrived damaged..."
    }
  ],
  "non_preferred_output": [
    {
      "role": "assistant",
      "content": "You need to go to our website and fill out the return form."
    }
  ]
}

​

Export Methods

​

Advanced Export Options

For evaluations involving tool use or function calling:

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "process_return",
        "description": "Process a product return request",
        "parameters": {
          "type": "object",
          "properties": {
            "order_id": {
              "type": "string",
              "description": "The order ID to process return for"
            },
            "reason": {
              "type": "string",
              "description": "Reason for the return"
            }
          },
          "required": ["order_id", "reason"]
        }
      }
    }
  ]
}

​

Best Practices

​

Creating High-Quality Evaluations

✅ "Damaged Product Return - Empathetic Response"
❌ "Eval 1"

​

Evaluation Criteria

When assessing responses, consider:

​

Advanced Evaluation Metrics

To further refine your evaluations, consider these advanced metrics tailored for AI-generated customer service responses:

Incorporate these metrics into your evaluation rubrics for more comprehensive assessments.

​

Use Cases

​

Model Fine-Tuning

Export evaluations to create training datasets:

1. Collect Examples: Build a corpus of high-quality evaluations
2. Export to JSONL: Use the bulk export feature
3. Prepare Dataset: Format according to your model’s requirements
4. Fine-Tune: Use the dataset to improve model performance

​

Quality Monitoring

Track response quality over time:

• Monitor status distribution (Outstanding vs. Unsatisfactory)
• Identify patterns in problematic responses
• Track improvement after training or process changes

​

Team Training

Use evaluations for human agent training:

• Share examples of outstanding responses
• Highlight common mistakes to avoid
• Create training materials from real scenarios

​

Troubleshooting

​

Next Steps