curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"limit": 10,
"status": "pending"
}'
{
"object": "eval.run_summary",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"total": 102,
"passed": 1,
"failed": 1,
"ungraded": 1,
"has_more": true,
"note": "Two-person tent, green — replacement for damaged pole set.",
"runs": [
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
]
}
Run an agent's eval suite (regression check)
Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has_more reports when the suite was longer than the cap.
POST
/
api
/
v1
/
agents
/
{id}
/
evals
/
run
curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"limit": 10,
"status": "pending"
}'
{
"object": "eval.run_summary",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"total": 102,
"passed": 1,
"failed": 1,
"ungraded": 1,
"has_more": true,
"note": "Two-person tent, green — replacement for damaged pole set.",
"runs": [
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
]
}
Each eval costs a real generation plus a judge call, so runs are sequential and capped per call; has_more reports when the suite was longer than the cap.
Path parameters
string (uuid)
required
The agent id.
Request body
object
integer
How many of the agent’s evals to run (newest first). Capped at 20 per call. Default:
10. Minimum: 1. Maximum: 20.string
Only run evals with this
status. Maximum length: 50.Response
object
string
required
Allowed values:
'eval.run_summary'.string (uuid)
required
integer
required
Number of evals run (
runs.length).integer
required
Runs with
passed: true.integer
required
Runs with
passed: false.integer
required
Runs with
passed: null.boolean
required
True when more evals exist for the agent than were run — the suite stopped at
limit.string
Present only when
has_more is true: states that only the first N were run.object[]
required
Show runs
Show runs
string
required
Allowed values:
'eval.run'.string (uuid)
required
string (uuid)
required
The agent the eval was replayed against (the request
agent_id, else the eval’s own).string,null
The replayed customer message. Absent when the run itself failed (
reason: 'run_failed').string,null
The eval’s
preferred_output. Absent when the run itself failed.string,null
The agent’s reply on this run; null when the agent returned no reply. Absent when the run itself failed.
boolean,null
required
True when
verdict is pass, false when it is partial or fail, and null when the reply was not graded — ‘the judge was unavailable’ and ‘the agent got it wrong’ are kept distinct.boolean
required
Whether the judge produced a verdict. When false,
reason says why and the verdict fields are absent.string
Present only when
graded is false. Allowed values: 'judge_unavailable', 'judge_failed', 'judge_empty', 'judge_invalid_json', 'judge_invalid_verdict', 'agent_returned_no_reply', 'run_failed'.string
Present only when
reason is run_failed (a run inside a suite threw); the public error message.integer
Generation plus judge time. Absent when the run itself failed.
string
Present only when
graded is true. Judged on substance, not wording. Allowed values: 'pass', 'partial', 'fail'.number,null
Present only when
graded is true. 0–5, how closely the substance matches (5 = equivalent); one decimal. Minimum: 0. Maximum: 5.boolean,null
Present only when
graded is true.string[]
Present only when
graded is true. Essential points from the reference that the reply is missing. Maximum items: 10.string,null
Present only when
graded is true. Maximum length: 2000.string,null
Present only when
graded is true. The judge’s reasoning, so a caller can disagree with the grade. Maximum length: 2000.Status codes
| Code | Meaning |
|---|---|
200 | The suite result. Evals run sequentially; a run that throws is recorded as an ungraded eval.run with reason: 'run_failed' rather than failing the whole suite. |
400 | Invalid request. issues names the offending field(s). |
401 | Missing, invalid, or expired API key. |
403 | The key lacks a required scope. |
404 | Not found, or not in the organization that owns the key. |
429 | Rate limit exceeded. |
Access
| Required scope | responses:write |
| Rate limit | 30 requests per minute |
Using this contract
Read the source OpenAPI document for declared schemas and alternatives. This page also includes documented corrections from the spec overlays. Example IDs and values are illustrative; replace them with records from your workspace.curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/agents/{id}/evals/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"limit": 10,
"status": "pending"
}'
{
"object": "eval.run_summary",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"total": 102,
"passed": 1,
"failed": 1,
"ungraded": 1,
"has_more": true,
"note": "Two-person tent, green — replacement for damaged pole set.",
"runs": [
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
]
}
Last modified on September 21, 2026