> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations

> Deployment topology, pod security, health probes, metrics, and DLQ handling for the Temporal engine.

# Operations

## Deployment topology

```
  ingress ──▶ engine-api    2 replicas, anti-affinity      :8080
              engine-worker 1 replica                      :9090
              dispatcher    1 replica, PDB                 :9091

  worker + api mount /secrets (projected volume, brand credentials)
  api ──▶ Temporal, PostgreSQL
  worker ──▶ Temporal, external connectors
  dispatcher ──▶ PostgreSQL, Temporal
```

## Pod security

* All pods run **non-root (uid 1000)** with `readOnlyRootFilesystem` and no privilege
  escalation.
* **API**: preferred pod anti-affinity across nodes, `PodDisruptionBudget` with
  `maxUnavailable: 1`.
* **Worker**: `PodDisruptionBudget` with `maxUnavailable: 1`.
* **Dispatcher**: `PodDisruptionBudget` with `maxUnavailable: 0` — singleton protection.
* **NetworkPolicy**: restricts ingress to service ports (8080, 9090, 9091); egress is open.
* **Brand secrets**: projected volume at `/secrets` on worker and API pods.
* **Config validation**: `AppConfig::validate()` runs at startup, failing fast on invalid
  config.

<Note>
  The dispatcher's `maxUnavailable: 0` is deliberate — it is the single writer draining the
  outbox. Evicting it during a node drain would stall dispatch, so the budget refuses.
</Note>

## Health probes

| Binary          | Port | Liveness   | Readiness | Metrics    |
| --------------- | ---- | ---------- | --------- | ---------- |
| `engine-api`    | 8080 | `/health`  | `/health` | `/metrics` |
| `engine-worker` | 9090 | `/healthz` | `/readyz` | `/metrics` |
| `dispatcher`    | 9091 | `/healthz` | `/readyz` | `/metrics` |

`engine-api`'s `/health` checks both the database and Temporal.

## Metrics

All three binaries expose Prometheus metrics with purpose-tuned histogram buckets:

| Prefix        | Buckets (seconds)                           | Measures                     |
| ------------- | ------------------------------------------- | ---------------------------- |
| `connector_`  | 0.01 … 60                                   | External API calls           |
| `dispatcher_` | 0.01 … 60                                   | Dispatch batch latency       |
| `llm_`        | 0.01 … 60                                   | LLM API calls                |
| `workflow_`   | 1, 5, 10, 30, 60, 120, 300, 600, 1800, 3600 | End-to-end workflow duration |

### Key series

**API** — `http_requests_total`, `http_request_duration_seconds`, `workflows_started_total` (by
workflow type), `workflow_signals_total` (by type and signal), `rate_limiter_rejections_total`.

**Worker / activities** — `connector_requests_total` and `connector_request_duration_seconds`
(by connector and status), `connector_errors_total` (by context, status, and whether the error
was retryable).

**Dispatcher** — `dispatcher_iterations_total`, `dispatcher_batch_duration_seconds`,
`dispatcher_events_claimed_total`, `dispatcher_events_dispatched_total`,
`dispatcher_events_failed_total`, `dispatcher_events_swept_total`, `dispatcher_errors_total`.

<Tip>
  `connector_errors_total` splits on **retryable**. A rising non-retryable count means an
  integration is misconfigured or a credential has expired — retries won't clear it, so alert on
  that series separately from the retryable one.
</Tip>

## Dead letter queue

An event that fails `max_attempts` times (default 20) moves to `workflow_dispatch_dlq`.
Operators can list, retry, or resolve DLQ items through the control-plane API.

Watch `dispatcher_events_failed_total` alongside DLQ depth: a growing DLQ with a flat failure
rate means a burst that already drained, while both climbing together means an ongoing outage.

## Running locally

```bash theme={null}
# migrations first
cargo run -p engine-control-plane --bin migrate

# then the processes
cargo run -p engine-api
cargo run -p engine-worker
cargo run -p engine-control-plane --bin dispatcher
```

Onboarding a brand:

```bash theme={null}
cargo run -p engine-control-plane --bin import_brand_manifests -- --tenant-id <uuid>
cargo run -p engine-control-plane --bin onboard_brand -- --tenant-id <uuid> --brand <slug>
```

## Related

* [Control plane](/next-temporal/control-plane) — outbox, DLQ, migration modes
* [Configuration](/next-temporal/configuration) — the environment reference