Skip to main content

​
Operations

​
Deployment topology

​
Pod security

  • All pods run non-root (uid 1000) with readOnlyRootFilesystem and no privilege escalation.
  • API: preferred pod anti-affinity across nodes, PodDisruptionBudget with maxUnavailable: 1.
  • Worker: PodDisruptionBudget with maxUnavailable: 1.
  • Dispatcher: PodDisruptionBudget with maxUnavailable: 0 — singleton protection.
  • NetworkPolicy: restricts ingress to service ports (8080, 9090, 9091); egress is open.
  • Brand secrets: projected volume at /secrets on worker and API pods.
  • Config validation: AppConfig::validate() runs at startup, failing fast on invalid config.
The dispatcher’s maxUnavailable: 0 is deliberate — it is the single writer draining the outbox. Evicting it during a node drain would stall dispatch, so the budget refuses.

​
Health probes

engine-api’s /health checks both the database and Temporal.

​
Metrics

All three binaries expose Prometheus metrics with purpose-tuned histogram buckets:

​
Key series

API — http_requests_total, http_request_duration_seconds, workflows_started_total (by workflow type), workflow_signals_total (by type and signal), rate_limiter_rejections_total. Worker / activities — connector_requests_total and connector_request_duration_seconds (by connector and status), connector_errors_total (by context, status, and whether the error was retryable). Dispatcher — dispatcher_iterations_total, dispatcher_batch_duration_seconds, dispatcher_events_claimed_total, dispatcher_events_dispatched_total, dispatcher_events_failed_total, dispatcher_events_swept_total, dispatcher_errors_total.
connector_errors_total splits on retryable. A rising non-retryable count means an integration is misconfigured or a credential has expired — retries won’t clear it, so alert on that series separately from the retryable one.

​
Dead letter queue

An event that fails max_attempts times (default 20) moves to workflow_dispatch_dlq. Operators can list, retry, or resolve DLQ items through the control-plane API. Watch dispatcher_events_failed_total alongside DLQ depth: a growing DLQ with a flat failure rate means a burst that already drained, while both climbing together means an ongoing outage.

​
Running locally

Onboarding a brand: