> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbooks

> Operational procedures — dispatcher pressure, DLQ remediation, worker scaling, secret rotation, migrations, and worker rollback.

# Runbooks

Operational procedures for running the engine.

## Dispatcher pressure

**Symptoms:** `dispatcher_errors_total` rising, `dispatcher_events_dispatched_total` stalled, or
events accumulating in `dispatch_outbox`.

### Diagnose

```bash theme={null}
kubectl logs -l app=dispatcher --tail=100

psql -c "SELECT status, COUNT(*) FROM dispatch_outbox GROUP BY status"
psql -c "SELECT COUNT(*) FROM dispatch_dlq WHERE status = 'pending'"
```

### What normal looks like

| Metric                              | Healthy                                                 |
| ----------------------------------- | ------------------------------------------------------- |
| `dispatcher_iterations_total`       | Increasing steadily                                     |
| `dispatcher_batch_duration_seconds` | p99 under 5s                                            |
| `dispatcher_events_failed_total`    | Occasional failures normal; **sustained growth is not** |

The dispatcher backs off exponentially on consecutive errors — `poll_ms * 2^n`, capped at 30
seconds — and resets to `poll_ms` on the first successful iteration.

### DLQ remediation

```bash theme={null}
# List
curl -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq"

# Retry one
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq/$DLQ_ID/retry"

# Resolve (discard) one
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq/$DLQ_ID/resolve"
```

### Tuning

| Variable                              | Default | Purpose                     |
| ------------------------------------- | ------- | --------------------------- |
| `CONTROL_PLANE_DISPATCH_POLL_MS`      | 1000    | Base polling interval       |
| `CONTROL_PLANE_DISPATCH_BATCH_SIZE`   | 25      | Events per batch            |
| `CONTROL_PLANE_DISPATCH_MAX_ATTEMPTS` | 20      | Retries before DLQ          |
| `CONTROL_PLANE_DISPATCH_JITTER_MS`    | 100     | Random jitter per iteration |

## Worker scaling

Health server on `WORKER_HEALTH_PORT` (default 9090):

| Endpoint       | Meaning                                                                                        |
| -------------- | ---------------------------------------------------------------------------------------------- |
| `GET /healthz` | Liveness — 200 whenever the process is running                                                 |
| `GET /readyz`  | Readiness — 200 only once accepting work **and** the latest Temporal namespace probe succeeded |
| `GET /metrics` | Prometheus metrics                                                                             |

Check task queue depth:

```bash theme={null}
temporal task-queue describe --task-queue stateset-response-automation-v2
```

If pending tasks grow while workers are healthy, scale up replicas.

<Warning>
  **RLS session variables.** When the worker queries the database directly, the control plane sets
  `SET app.tenant_id = '<uuid>'` for row-level security. If RLS is enabled and the session variable
  is not set, **queries return empty results rather than erroring** — which reads like missing
  data, not a misconfiguration. Confirm `CONTROL_PLANE_AUTH_MODE=required` in production.
</Warning>

## Worker rollback

<Warning>
  **Rolling the worker image back is not unconditionally safe.** The worker has no build-id
  versioning, so old and new workflow code cannot serve histories side by side. CI never rolls the
  worker back automatically for this reason.
</Warning>

### Decision tree

**1. Has the new worker taken any workflow tasks?**

```bash theme={null}
kubectl logs deploy/next-temporal-v2-worker -n stateset-temporal \
  --since=30m | grep -c "run workflow activation"
```

* **No** → `kubectl rollout undo` is safe. Done.
* **Yes** → continue.

**2. Did the new code add or remove `ctx.patched()` markers, or change any workflow command
sequence?** Check the deploy diff for `_PATCH_ID`, `ctx.patched`, `start_activity`, `ctx.timer`.

* **No — activity-internals only** → undo is safe. Histories recorded under the new image replay
  identically on the old one.
* **Yes** → any in-flight execution that advanced past a new marker will **fail replay** on the
  old code. Choose one:

| Option                         | Approach                                                                                                  |
| ------------------------------ | --------------------------------------------------------------------------------------------------------- |
| **Roll forward** *(preferred)* | Ship a fix on top of the new code                                                                         |
| **Pause, then undo**           | Pause the task queue so nothing else advances, undo, then handle stranded runs                            |
| **Accept failures**            | Undo without pausing — affected runs surface as workflow task failures and retry forever against old code |

<Warning>
  Do not pick "accept failures" silently. Every stuck run pins a ticket.
</Warning>

For the pause-then-undo path, find affected runs:

```bash theme={null}
temporal workflow list \
  --query 'ExecutionStatus="Running" AND WorkflowType="ResponseAutomationV2Workflow"'
```

Runs that fail with `NonDeterminismError` after the undo must be terminated and re-submitted
from the API. They are idempotent to re-run up to their last committed mutation, thanks to the
ledger.

**3. Afterwards, always:** confirm `kubectl rollout status` is healthy, check worker
`:9090/readyz`, and watch that `temporal workflow count --query 'ExecutionStatus="Running"'`
returns to baseline.

<Note>
  The proper fix is build-id pinning via `WorkerVersioningStrategy`, which would make undo safe by
  construction. It is not yet implemented.
</Note>

## Migrations

Migrations run via the `migrate` binary with baseline detection.

**Follow expand/contract for rolling deploys** — add the new column or table first, deploy code
that writes both, backfill, then remove the old shape in a later migration. A migration that
drops or renames in one step will break the pods still running the previous image.

Concurrent index migrations are marked `-- migrate:no-transaction`, since `CREATE INDEX
CONCURRENTLY` cannot run inside a transaction. A failed concurrent-index migration leaves an
invalid index that must be dropped before retrying.

## Other procedures

The repo's runbooks also cover **brand secret rotation** (including rotation without restart and
verifying secret resolvability), **rate limiting** (global and per-tenant, with monitoring and
adjustment), **circuit breaker and connector recovery** (health checks, failure states, recovery
steps), and **brand onboarding** (validate a manifest, onboard into shadow, activate after
validation).

## Related

* [Operations](/next-temporal/operations) — deployment topology and metrics
* [Control plane](/next-temporal/control-plane) — outbox, DLQ, migration modes