curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Replay one eval and grade it against the recorded answer
Runs the real generation path, then grades the reply against preferred_output on substance rather than wording…
POST
/
api
/
v1
/
evals
/
{id}
/
run
curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Runs the real generation path, then grades the reply against preferred_output on substance rather than wording.
passed is null — never false — when grading could not run, so an unavailable judge is not mistaken for a failing agent.
Path parameters
Request body
object
Response
object
The agent the eval was replayed against (the request
agent_id, else the eval’s own).The agent’s reply on this run; null when the agent returned no reply. Absent when the run itself failed.
True when
verdict is pass, false when it is partial or fail, and null when the reply was not graded — ‘the judge was unavailable’ and ‘the agent got it wrong’ are kept distinct.Whether the judge produced a verdict. When false,
reason says why and the verdict fields are absent.Present only when
graded is false. Allowed values: 'judge_unavailable', 'judge_failed', 'judge_empty', 'judge_invalid_json', 'judge_invalid_verdict', 'agent_returned_no_reply', 'run_failed'.Present only when
reason is run_failed (a run inside a suite threw); the public error message.Present only when
graded is true. Judged on substance, not wording. Allowed values: 'pass', 'partial', 'fail'.Present only when
graded is true. 0–5, how closely the substance matches (5 = equivalent); one decimal. Minimum: 0. Maximum: 5.Present only when
graded is true. Essential points from the reference that the reply is missing. Maximum items: 10.Present only when
graded is true. The judge’s reasoning, so a caller can disagree with the grade. Maximum length: 2000.Status codes
| Code | Meaning |
|---|---|
200 | The replay result: the agent’s reply on this run, graded against the eval’s preferred_output by a reference-based judge. passed is null (not false) when grading did not happen. |
400 | Invalid request. issues names the offending field(s). |
401 | Missing, invalid, or expired API key. |
403 | The key lacks a required scope. |
404 | Not found, or not in the organization that owns the key. |
429 | Rate limit exceeded. |
Access
| Required scope | responses:write |
| Rate limit | 30 requests per minute |
Using this contract
Read the source OpenAPI document for declared schemas and alternatives. This page also includes documented corrections from the spec overlays. Example IDs and values are illustrative; replace them with records from your workspace.curl --silent --show-error --fail-with-body --request POST \
--url 'https://response.stateset.com/api/v1/evals/{id}/run' \
--header "Authorization: Bearer $RESPONSECX_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}'
{
"object": "eval.run",
"eval_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"agent_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"user_message": "string",
"expected": "string",
"actual": "string",
"passed": true,
"graded": true,
"reason": "judge_unavailable",
"error": "string",
"duration_ms": 8,
"verdict": "pass",
"equivalence": 1.5,
"contradicts": true,
"omits": [
"string"
],
"differences": "string",
"explanation": "string"
}
Last modified on September 21, 2026