Runbooks

Operational procedures for running the engine.

Dispatcher pressure

Symptoms: dispatcher_errors_total rising, dispatcher_events_dispatched_total stalled, or events accumulating in dispatch_outbox.

Diagnose

kubectl logs -l app=dispatcher --tail=100

psql -c "SELECT status, COUNT(*) FROM dispatch_outbox GROUP BY status"
psql -c "SELECT COUNT(*) FROM dispatch_dlq WHERE status = 'pending'"

What normal looks like

Metric	Healthy
`dispatcher_iterations_total`	Increasing steadily
`dispatcher_batch_duration_seconds`	p99 under 5s
`dispatcher_events_failed_total`	Occasional failures normal; sustained growth is not

The dispatcher backs off exponentially on consecutive errors — poll_ms * 2^n, capped at 30 seconds — and resets to poll_ms on the first successful iteration.

DLQ remediation

# List
curl -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq"

# Retry one
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq/$DLQ_ID/retry"

# Resolve (discard) one
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "$API_URL/v1/brands/$BRAND_ID/dispatch-dlq/$DLQ_ID/resolve"

Tuning

Variable	Default	Purpose
`CONTROL_PLANE_DISPATCH_POLL_MS`	1000	Base polling interval
`CONTROL_PLANE_DISPATCH_BATCH_SIZE`	25	Events per batch
`CONTROL_PLANE_DISPATCH_MAX_ATTEMPTS`	20	Retries before DLQ
`CONTROL_PLANE_DISPATCH_JITTER_MS`	100	Random jitter per iteration

Worker scaling

Health server on WORKER_HEALTH_PORT (default 9090):

Endpoint	Meaning
`GET /healthz`	Liveness — 200 whenever the process is running
`GET /readyz`	Readiness — 200 only once accepting work and the latest Temporal namespace probe succeeded
`GET /metrics`	Prometheus metrics

Check task queue depth:

temporal task-queue describe --task-queue stateset-response-automation-v2

If pending tasks grow while workers are healthy, scale up replicas.

RLS session variables. When the worker queries the database directly, the control plane sets SET app.tenant_id = '<uuid>' for row-level security. If RLS is enabled and the session variable is not set, queries return empty results rather than erroring — which reads like missing data, not a misconfiguration. Confirm CONTROL_PLANE_AUTH_MODE=required in production.

Worker rollback

Rolling the worker image back is not unconditionally safe. The worker has no build-id versioning, so old and new workflow code cannot serve histories side by side. CI never rolls the worker back automatically for this reason.

Decision tree

1. Has the new worker taken any workflow tasks?

kubectl logs deploy/next-temporal-v2-worker -n stateset-temporal \
  --since=30m | grep -c "run workflow activation"

No → kubectl rollout undo is safe. Done.
Yes → continue.

2. Did the new code add or remove ctx.patched() markers, or change any workflow command sequence? Check the deploy diff for _PATCH_ID, ctx.patched, start_activity, ctx.timer.

No — activity-internals only → undo is safe. Histories recorded under the new image replay identically on the old one.
Yes → any in-flight execution that advanced past a new marker will fail replay on the old code. Choose one:

Option	Approach
Roll forward (preferred)	Ship a fix on top of the new code
Pause, then undo	Pause the task queue so nothing else advances, undo, then handle stranded runs
Accept failures	Undo without pausing — affected runs surface as workflow task failures and retry forever against old code

Do not pick “accept failures” silently. Every stuck run pins a ticket.

For the pause-then-undo path, find affected runs:

temporal workflow list \
  --query 'ExecutionStatus="Running" AND WorkflowType="ResponseAutomationV2Workflow"'

Runs that fail with NonDeterminismError after the undo must be terminated and re-submitted from the API. They are idempotent to re-run up to their last committed mutation, thanks to the ledger. 3. Afterwards, always: confirm kubectl rollout status is healthy, check worker :9090/readyz, and watch that temporal workflow count --query 'ExecutionStatus="Running"' returns to baseline.

The proper fix is build-id pinning via WorkerVersioningStrategy, which would make undo safe by construction. It is not yet implemented.

Migrations

Migrations run via the migrate binary with baseline detection. Follow expand/contract for rolling deploys — add the new column or table first, deploy code that writes both, backfill, then remove the old shape in a later migration. A migration that drops or renames in one step will break the pods still running the previous image. Concurrent index migrations are marked -- migrate:no-transaction, since CREATE INDEX CONCURRENTLY cannot run inside a transaction. A failed concurrent-index migration leaves an invalid index that must be dropped before retrying.

Other procedures

The repo’s runbooks also cover brand secret rotation (including rotation without restart and verifying secret resolvability), rate limiting (global and per-tenant, with monitoring and adjustment), circuit breaker and connector recovery (health checks, failure states, recovery steps), and brand onboarding (validate a manifest, onboard into shadow, activate after validation).

Operations — deployment topology and metrics
Control plane — outbox, DLQ, migration modes

Kubernetes Deployment iCommerce Quickstart

Get started

Platform

StateSet One

StateSet Response

StateSet ResponseCX

Temporal Engine

StateSet iCommerce

StateSet Commerce

StateSet EDI

StateSet Mail

StateSet Voice

StateSet NSR

AI Agents

StateSet Sandbox

StateSet ACP

StateSet UCP

StateSet ICP

StateSet Sequencer

StateSet Set L2

StateSet Cloud

Tools

Runbooks

Runbooks

Dispatcher pressure

Diagnose

What normal looks like

DLQ remediation

Tuning

Worker scaling

Worker rollback

Decision tree

Migrations

Other procedures

​Runbooks

​Dispatcher pressure

​Diagnose

​What normal looks like

​DLQ remediation

​Tuning

​Worker scaling

​Worker rollback

​Decision tree

​Migrations

​Other procedures

​Related

Runbooks

Dispatcher pressure

Diagnose

What normal looks like

DLQ remediation

Tuning

Worker scaling

Worker rollback

Decision tree

Migrations

Other procedures

Related