> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations Guide

> Create and manage evaluations to improve AI response quality and train custom models

## Overview

The Evaluations system enables you to assess, track, and improve the quality of AI-generated responses. By creating evaluations, you can build datasets for fine-tuning models, monitor performance trends, and ensure consistent high-quality customer interactions.

<Card title="Key Benefits" icon="sparkles">
  * **Quality Assurance**: Monitor and improve response quality
  * **Model Training**: Export evaluations as JSONL for fine-tuning
  * **Performance Tracking**: Analyze trends and identify areas for improvement
  * **Team Insights**: Understand response patterns across different support types
</Card>

## Evaluation Status Types

Each evaluation is assigned a status that reflects the quality of the response:

<CardGroup cols={2}>
  <Card title="Outstanding" icon="badge-check" color="#10b981">
    Exceptional responses that exceed expectations. These serve as gold-standard examples for training.
  </Card>

  <Card title="Satisfactory" icon="check-circle" color="#3b82f6">
    Good responses that meet quality standards and properly address customer needs.
  </Card>

  <Card title="Needs Further Review" icon="clock" color="#f59e0b">
    Responses requiring additional assessment or minor improvements before final classification.
  </Card>

  <Card title="Unsatisfactory" icon="exclamation" color="#f97316">
    Responses with significant issues that don't meet quality standards.
  </Card>

  <Card title="Code Red" icon="exclamation-triangle" color="#ef4444">
    Critical failures requiring immediate attention and remediation.
  </Card>
</CardGroup>

## Creating Evaluations

### Manual Evaluation Creation

To create a new evaluation manually:

1. Navigate to the Evaluations Dashboard
2. Click "Create New Evaluation"
3. Fill in the evaluation details:

```json theme={null}
{
  "eval_name": "Customer Refund Request Handling",
  "eval_type": "Customer Service",
  "user_message": "I want to return my order #12345. It arrived damaged.",
  "preferred_output": "I'm sorry to hear your order arrived damaged. I'll help you with the return right away. I've initiated a return for order #12345 and you'll receive a prepaid shipping label via email within 24 hours. Once we receive the item, your refund will be processed within 3-5 business days.",
  "non_preferred_output": "You need to go to our website and fill out the return form.",
  "eval_status": "Outstanding",
  "description": "Exemplary handling of damaged product return with empathy and clear next steps"
}
```

### Evaluation Types

Choose the appropriate type for your evaluation:

* **Customer Service**: General customer inquiries and support
* **Technical Support**: Technical issues and troubleshooting
* **Sales**: Sales-related interactions and inquiries
* **Product Support**: Product-specific questions and guidance
* **General**: Other types of interactions

### Building Effective Test Sets

To create robust evaluations, build test sets that represent real-world scenarios:

* **Golden Test Sets**: Curate 50-100 high-quality examples with expected responses.
* **Diverse Scenarios**: Include in-scope queries, out-of-scope small talk, and adversarial questions.
* **Synthetic Data**: Generate variations using LLMs to expand coverage, including edge cases and bias tests.
* **Iteration**: Continuously update based on production data and user feedback.

Use these test sets to ensure comprehensive evaluation coverage.

## Managing Evaluations

### Dashboard Features

The Evaluations Dashboard provides comprehensive tools for management:

<AccordionGroup>
  <Accordion title="Search and Filter">
    * **Smart Search**: Search by name, type, ticket ID, or description
    * **Type Filter**: Filter by evaluation type (Customer Service, Technical Support, etc.)
    * **Status Filter**: View evaluations by their quality status
    * **Date Range**: Filter by creation date (Last 7/30/90 days or all time)
    * **Sorting**: Sort by date, name, status, or type in ascending/descending order
  </Accordion>

  <Accordion title="Bulk Operations">
    * **Select Multiple**: Use checkboxes to select multiple evaluations
    * **Bulk Export**: Export selected evaluations to JSONL format
    * **Bulk Delete**: Remove multiple evaluations at once
    * **Select All**: Quickly select all visible evaluations
  </Accordion>

  <Accordion title="View Modes">
    * **Table View**: Traditional table layout for efficient scanning
    * **Card View**: Visual card layout for detailed preview
    * **Analytics View**: Coming soon - detailed insights and trends
  </Accordion>
</AccordionGroup>

### Types of Evaluation Methods

Enhance your evaluation process using different methods:

<AccordionGroup>
  <Accordion title="Human Evaluations">
    * Direct feedback from users or experts
    * Best for nuanced quality assessment
    * Example: Rate response empathy on a 1-5 scale
  </Accordion>

  <Accordion title="LLM-as-Judge">
    * Use another AI to evaluate outputs
    * Scalable for large datasets
    * Example: Prompt an LLM to score factual accuracy
  </Accordion>

  <Accordion title="Code-based Evaluations">
    * Programmatic checks for specific criteria
    * Ideal for objective metrics
    * Example: Verify response format or keyword presence
  </Accordion>
</AccordionGroup>

Combine these methods for comprehensive quality assurance.

### Performance Statistics

Monitor your evaluation metrics through dashboard cards:

* **Total Evaluations**: Overall count of all evaluations
* **Outstanding**: Count of exceptional responses
* **Needs Review**: Evaluations requiring further assessment
* **This Week**: Recent evaluation activity

Each metric includes trend indicators showing performance changes over time.

## Exporting for Model Training

### JSONL Export Format

Evaluations can be exported in JSONL format for model fine-tuning:

```json theme={null}
{
  "input": {
    "messages": [
      {
        "role": "user",
        "content": "I want to return my order #12345. It arrived damaged."
      }
    ],
    "tools": [],
    "parallel_tool_calls": true
  },
  "preferred_output": [
    {
      "role": "assistant",
      "content": "I'm sorry to hear your order arrived damaged..."
    }
  ],
  "non_preferred_output": [
    {
      "role": "assistant",
      "content": "You need to go to our website and fill out the return form."
    }
  ]
}
```

### Export Methods

<Tabs>
  <Tab title="Manual Export">
    1. Navigate to the Export tab in the dashboard
    2. Select "Create New" mode
    3. Enter the evaluation details:
       * User Message
       * Preferred Output
       * Non-Preferred Output
       * Tools (optional, for function calling)
    4. Click "Export to JSONL"
  </Tab>

  <Tab title="Bulk Export">
    1. Navigate to the Export tab
    2. Select "From Existing Evals" mode
    3. Choose evaluations to export:
       * Use checkboxes to select specific evaluations
       * Click "Select All" to export all evaluations
    4. Click "Export X Evaluations"
  </Tab>
</Tabs>

### Advanced Export Options

For evaluations involving tool use or function calling:

```json theme={null}
{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "process_return",
        "description": "Process a product return request",
        "parameters": {
          "type": "object",
          "properties": {
            "order_id": {
              "type": "string",
              "description": "The order ID to process return for"
            },
            "reason": {
              "type": "string",
              "description": "Reason for the return"
            }
          },
          "required": ["order_id", "reason"]
        }
      }
    }
  ]
}
```

## Best Practices

### Creating High-Quality Evaluations

<Steps>
  <Step title="Clear Naming">
    Use descriptive names that indicate the scenario being evaluated

    ```
    ✅ "Damaged Product Return - Empathetic Response"
    ❌ "Eval 1"
    ```
  </Step>

  <Step title="Realistic Scenarios">
    Base evaluations on actual customer interactions and common use cases
  </Step>

  <Step title="Comprehensive Coverage">
    Include both ideal responses and common mistakes to avoid
  </Step>

  <Step title="Consistent Standards">
    Apply evaluation criteria consistently across similar interaction types
  </Step>

  <Step title="Regular Review">
    Periodically review and update evaluations to maintain relevance
  </Step>
</Steps>

### Evaluation Criteria

When assessing responses, consider:

<CardGroup cols={2}>
  <Card title="Accuracy" icon="bullseye">
    * Correct information provided
    * Proper understanding of the issue
    * Appropriate solution offered
  </Card>

  <Card title="Tone & Empathy" icon="heart">
    * Professional and friendly tone
    * Empathy for customer situation
    * Appropriate level of formality
  </Card>

  <Card title="Completeness" icon="list-check">
    * All questions answered
    * Clear next steps provided
    * No missing information
  </Card>

  <Card title="Efficiency" icon="rocket">
    * Concise yet comprehensive
    * Direct problem resolution
    * Minimal back-and-forth needed
  </Card>
</CardGroup>

## Advanced Evaluation Metrics

To further refine your evaluations, consider these advanced metrics tailored for AI-generated customer service responses:

<CardGroup cols={2}>
  <Card title="Relevance" icon="bullseye-arrow">
    * How well the response addresses the specific query
    * Avoidance of unnecessary information
    * Alignment with customer needs
  </Card>

  <Card title="Coherence" icon="arrows-collapse-horizontal">
    * Logical flow of information
    * Consistent language and structure
    * Easy to follow reasoning
  </Card>

  <Card title="Helpfulness" icon="hand-holding-heart">
    * Enables customer action
    * Provides value beyond basic information
    * Anticipates follow-up needs
  </Card>

  <Card title="Safety & Bias" icon="shield-check">
    * Absence of harmful content
    * Fair and unbiased responses
    * Compliance with ethical guidelines
  </Card>
</CardGroup>

Incorporate these metrics into your evaluation rubrics for more comprehensive assessments.

## Use Cases

### Model Fine-Tuning

Export evaluations to create training datasets:

1. **Collect Examples**: Build a corpus of high-quality evaluations
2. **Export to JSONL**: Use the bulk export feature
3. **Prepare Dataset**: Format according to your model's requirements
4. **Fine-Tune**: Use the dataset to improve model performance

### Quality Monitoring

Track response quality over time:

* Monitor status distribution (Outstanding vs. Unsatisfactory)
* Identify patterns in problematic responses
* Track improvement after training or process changes

### Team Training

Use evaluations for human agent training:

* Share examples of outstanding responses
* Highlight common mistakes to avoid
* Create training materials from real scenarios

## Troubleshooting

<AccordionGroup>
  <Accordion title="Evaluations not appearing">
    * Ensure you're logged into the correct organization
    * Check filters aren't hiding evaluations
    * Refresh the dashboard
  </Accordion>

  <Accordion title="Export failing">
    * Verify all required fields are filled
    * Check for valid JSON in tools field
    * Ensure evaluations have required data
  </Accordion>

  <Accordion title="Performance issues">
    * Use filters to reduce displayed evaluations
    * Export in smaller batches
    * Clear browser cache if needed
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Create Your First Evaluation" icon="plus" href="https://response.cx/evals">
    Start building your evaluation dataset
  </Card>

  <Card title="Best Practices" icon="lightbulb" href="https://response.cx">
    Deep dive into evaluation strategies
  </Card>
</CardGroup>