curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Replay one eval and grade it against the recorded answer
Runs the real generation path, then grades the reply against preferred_output on substance rather than wording…
POST
/
api
/
v1
/
evals
/
{id}
/
run
curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Runs the real generation path, then grades the reply against preferred_output on substance rather than wording.
passed is null — never false — when grading could not run, so an unavailable judge is not mistaken for a failing agent.
Path parameters
string (uuid)
required
The eval id.
Request body
object
string (uuid)
Agent to replay against. Defaults to the eval’s own
agent_id; 400 if neither is set.Response
object
string
required
Allowed values:
'eval.run'.string (uuid)
required
string (uuid)
required
The agent the eval was replayed against (the request
agent_id, else the eval’s own).string,null
The replayed customer message. Absent when the run itself failed (
reason: 'run_failed').string,null
The eval’s
preferred_output. Absent when the run itself failed.string,null
The agent’s reply on this run; null when the agent returned no reply. Absent when the run itself failed.
boolean,null
required
True when
verdict is pass, false when it is partial or fail, and null when the reply was not graded — ‘the judge was unavailable’ and ‘the agent got it wrong’ are kept distinct.boolean
required
Whether the judge produced a verdict. When false,
reason says why and the verdict fields are absent.string
Present only when
graded is false. Allowed values: 'judge_unavailable', 'judge_failed', 'judge_empty', 'judge_invalid_json', 'judge_invalid_verdict', 'agent_returned_no_reply', 'run_failed'.string
Present only when
reason is run_failed (a run inside a suite threw); the public error message.integer
Generation plus judge time. Absent when the run itself failed.
string
Present only when
graded is true. Judged on substance, not wording. Allowed values: 'pass', 'partial', 'fail'.number,null
Present only when
graded is true. 0–5, how closely the substance matches (5 = equivalent); one decimal. Minimum: 0. Maximum: 5.boolean,null
Present only when
graded is true.string[]
Present only when
graded is true. Essential points from the reference that the reply is missing. Maximum items: 10.string,null
Present only when
graded is true. Maximum length: 2000.string,null
Present only when
graded is true. The judge’s reasoning, so a caller can disagree with the grade. Maximum length: 2000.Status codes
| Code | Meaning |
|---|---|
200 | The replay result: the agent’s reply on this run, graded against the eval’s preferred_output by a reference-based judge. passed is null (not false) when grading did not happen. |
400 | Invalid request. issues names the offending field(s). |
401 | Missing, invalid, or expired API key. |
403 | The key lacks a required scope. |
404 | Not found, or not in the organization that owns the key. |
429 | Rate limit exceeded. |
Access
| Required scope | responses:write |
| Rate limit | 30 requests per minute |
Using this contract
Read the source OpenAPI document for declared schemas and alternatives. This page also includes documented corrections from the spec overlays. Example IDs and values are illustrative; replace them with records from your workspace.curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Last modified on September 21, 2026