> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes Deployment

> First-rollout checklist for the Rust Temporal engine — prerequisites, ordered phases, and known manifest gaps.

# Kubernetes Deployment

The deployment checklist for the Rust workflow stack, and the gaps in the current manifests you
need to know about before a production rollout.

## Goal

A complete deployment means:

* Temporal cluster available and healthy.
* Temporal server pods isolated behind a Cloud SQL proxy sidecar.
* Control-plane database migrations applied **before** app traffic.
* `engine-api`, `engine-worker`, and `dispatcher` rolled out **in a controlled order**.
* Per-brand connector secrets sourced from Vault and resolving correctly.
* One brand onboarded in `shadow` for validation before live traffic.

Runtime components are the Temporal Helm release, `engine-api`, `engine-worker`, `dispatcher`,
and the `migrate` job, in the `stateset-temporal` namespace.

## Known gaps

<Warning>
  These are current limitations of the shipped manifests, not steps to perform. Read them before
  planning a rollout — several change what you must do out of band.
</Warning>

### Temporal schema is managed outside Helm

Helm manages the Temporal server Deployments, and the release uses a Cloud SQL proxy sidecar.
Consequently the **Helm chart schema jobs are disabled**: the Temporal default and visibility
databases must already exist, and schema setup and updates are handled outside the Helm install
path.

### Vault prerequisites must exist before brand onboarding

Required in `stateset-temporal` before onboarding any brand:

* `SecretStore/vault-backend`
* The `external-secrets-vault-token` Secret
* One `ExternalSecret` per brand
* `next-temporal-v2` ServiceAccount permission to `get` namespace Secrets

### Monitoring coverage is partial

The current overlay scrapes **the API only**. Worker and dispatcher metrics Services and
ServiceMonitors are missing — so the dispatcher and connector metrics described in
[Operations](/next-temporal/operations) are exposed by the pods but not collected.

### Image versioning

Production should use immutable semver tags, release tags, or image digests. Combined with the
[worker rollback constraints](/next-temporal/runbooks#worker-rollback), pinning matters more
here than in a typical stateless service.

### No environment overlays yet

The manifest layout is a single v2 stack — `dev`, `staging`, and `prod` overlays do not exist.
Replica counts and some deployment assumptions are documented in more than one place; the
manifests should become the source of truth once overlays land.

## Phase 0 — cluster prerequisites

Confirm all of the following before starting:

* Kubernetes cluster access and namespace creation permissions
* Helm access
* Ingress controller installed
* `cert-manager` installed
* `external-dns` installed, if hostname automation is required
* External Secrets Operator installed
* Prometheus Operator installed, if using `ServiceMonitor` and `PrometheusRule`
* Managed PostgreSQL endpoints provisioned
* Cloud SQL proxy credential secret available in `stateset-temporal`
* Vault token secret available in `stateset-temporal`
* Container registry pull access configured

## Phase 1 — build and publish the image

Build the release image, push an **immutable** tag, record the digest, and update the manifests
or overlay values to that exact tag or digest.

For GitHub Actions, configure Workload Identity Federation secrets:
`GCP_WORKLOAD_IDENTITY_PROVIDER` and `GCP_ARTIFACT_REGISTRY_SERVICE_ACCOUNT`.

## Phase 2 — prepare secrets

**Shared app secrets:** `CONTROL_PLANE_DATABASE_URL`, `CONTROL_PLANE_API_KEYS_JSON`.

**Temporal Helm variables:** `TEMPORAL_CLOUDSQL_INSTANCE`,
`TEMPORAL_CLOUDSQL_CREDENTIALS_SECRET`, `TEMPORAL_DB_NAME`, `TEMPORAL_DB_VISIBILITY_DB_NAME`,
`TEMPORAL_DB_USER`, `TEMPORAL_DB_PASSWORD`.

**Vault-backed brand connector secrets** for the first rollout brand — Shopify, Gorgias, the LLM
provider, the KB provider, and any other connector enabled in the brand manifest.

## Phase 3 — install Temporal

Apply the namespace, ensure the Cloud SQL proxy credential secret exists in it, run the schema
bootstrap script for a first install, then install or upgrade the Helm release. Wait for
frontend, history, matching, and worker services to be healthy.

**Validate:**

```bash theme={null}
temporal task-queue describe --task-queue stateset-response-automation-v2
```

Confirm the default and visibility schemas were provisioned out of band.

## Phase 4 — run the control-plane migration

Apply config, service account, and Vault SecretStore bootstrap resources, then apply **only the
migration job** and wait for completion.

**Validate:** control-plane tables exist, `schema_migrations` is current, and no failed
migration job pods remain.

## Phase 5 — roll out the app stack

In order:

1. API Deployment, Service, PDB, Ingress
2. Worker Deployment and PDB
3. Dispatcher Deployment and PDB

**Validate:** `/readyz` on all three, the API can reach Temporal, the worker is polling the
configured task queue, and the dispatcher health dashboard is healthy.

## Phase 6 — monitoring

Apply ServiceMonitors and PrometheusRules. Confirm targets are scraped and alert rules load
without errors.

## Phase 7 — onboard the first brand in shadow

Import or validate the brand manifest, create the brand Vault entry, apply the brand
`ExternalSecret`, then onboard with `--activate`.

**Validate:** the brand's routing mode is `shadow`, and connector secret resolution works.

## Phase 8 — end-to-end smoke test

Start a workflow through the API and verify it appears in Temporal, the worker executes
activities, dispatcher metrics are healthy, and there is **no DLQ growth**.

## Phase 9 — promote to controlled traffic

```
shadow ──▶ canary ──▶ live
```

Move from `shadow` to `canary`, watch the parity dashboard, and verify the mismatch rate stays
within threshold. **Only then** consider `live`.

<Note>
  Phases 7 through 9 are the reason this is a staged rollout rather than a deploy. The app stack
  being healthy (Phase 5) tells you the engine runs; only the parity dashboard tells you it
  behaves the same as the system it replaces.
</Note>

## Related

* [Operations](/next-temporal/operations) — topology, probes, metrics
* [Runbooks](/next-temporal/runbooks) — rollback, DLQ, migrations
* [Control plane](/next-temporal/control-plane) — migration modes and the parity dashboard