> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Voice Operations

> Deployment, health probes, metrics, alerting, and scaling for the voice platform.

# Voice Operations

## Deployment

The repo ships Docker Compose for local and single-host use, and Kubernetes manifests for
production — namespace, deployment, configmap, ingress, a separate **admin ingress**, and an HPA.

<Note>
  Admin routes have their own ingress. Keep it internal — it exposes tenant cache controls,
  diagnostics, and session data, gated only by `ADMIN_API_KEY`.
</Note>

## Health probes

| Endpoint            | Use                                  |
| ------------------- | ------------------------------------ |
| `GET /health`       | Health info, active sessions, uptime |
| `GET /health/live`  | Kubernetes liveness                  |
| `GET /health/ready` | Kubernetes readiness                 |

Kafka is probed and reported in readiness but only **fails** it when
`HEALTH_READY_REQUIRE_KAFKA=true` — so by default a broker outage cannot pull call-serving pods
out of rotation.

## Metrics

Prometheus, with pre-configured Grafana dashboards, plus Alertmanager in the Compose stack.

| Port                     | Auth                   |
| ------------------------ | ---------------------- |
| Main port `/metrics`     | Requires `x-admin-key` |
| Dedicated `METRICS_PORT` | **Unauthenticated**    |

<Warning>
  The dedicated metrics port is unauthenticated by design and must be network-restricted. Do not
  expose it through a public ingress.
</Warning>

Metrics worth alerting on:

* `escalation_bridge_failed_total` — a background escalation failed and the caller got a recovery
  prompt instead of a human.
* Provider outcome metrics for ElevenLabs — these reveal circuit-breaker trips before callers
  report bad audio.

## Auth transport rollout

`GET /admin/alerts/auth-transport` evaluates rollout state as `ok` / `warn` / `error` with
actionable alert codes, and `GET /admin/auth/transport` shows the legacy fallback toggle and
stream-secret readiness.

Use these while migrating off legacy query auth — they tell you whether any tenant still depends
on it before you turn it off.

## Tenant cache

| Route                                       | Use                                                |
| ------------------------------------------- | -------------------------------------------------- |
| `GET /admin/tenants/cache-metrics`          | Entries, hits, misses, reloads, evictions          |
| `GET /admin/tenants/cache-metrics/by-token` | Top token-level counters                           |
| `POST /admin/tenants/reload-cached`         | Reload while preserving live session and log state |
| `POST /admin/tenants/evict-idle`            | Evict idle entries (`max_idle_secs`)               |

Idle entries also expire automatically per `TENANT_CACHE_IDLE_TTL_SECS` (default 3600s).

## Twilio webhook diagnostics

When inbound calls fail to resolve to the right tenant:

| Route                                                   | Use                                                                          |
| ------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `GET /admin/twilio/webhook-diagnostics`                 | Hint-cache size and limits, resolution counters, success and cache-hit rates |
| `POST /admin/twilio/webhook-diagnostics/cache/clear`    | Clear hint-cache entries during incident recovery                            |
| `POST /admin/twilio/webhook-diagnostics/counters/reset` | Reset counters                                                               |

## Resilience

* **Retries** with exponential backoff on external API calls.
* **Circuit breaker** on ElevenLabs, so transport and 5xx/429 failures don't cascade.
* **Graceful shutdown** with resource cleanup.
* **Rate limiting** per IP and per tenant, with automatic bucket cleanup.

## Load testing

A `load-test/` harness ships with the repo for validating capacity before a launch.

## Related

* [Configuration](/stateset-voice/configuration)
* [Voice API](/stateset-voice/api) — the full admin surface