Azure Deployment Guide (AKS)

Complete guide for deploying StateSet Sandbox on Azure Kubernetes Service (AKS).

Prerequisites

Azure CLI configured with appropriate permissions
kubectl installed
Helm 3.x installed
Domain name for API endpoint
Azure subscription with required resource providers enabled

Quick Start (Kustomize Overlay)

Use the Azure-specific overlay to apply the correct ingress/controller defaults:

kubectl apply -k k8s/overlays/azure
# or
./scripts/deploy-k8s.sh azure
# or (CLI)
stateset deploy azure
stateset deploy azure --preflight --wait --smoke

For smoke checks, export credentials before running deploy:

export STATESET_API_KEY="your_api_key"
export STATESET_API_URL="https://api.sandbox.stateset.com/api/v1"

Before applying the overlay, provision the required controller auth secret:

read -rsp "Admin password: " ADMIN_PASSWORD; echo
bash scripts/create-controller-secret.sh \
  --admin-username admin \
  --admin-password "$ADMIN_PASSWORD"
unset ADMIN_PASSWORD

Before applying, update the Azure SAS token secret:

# Edit the placeholder SAS token
vi k8s/overlays/azure/secret.yaml

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              Microsoft Azure                                     │
│  ┌───────────────────────────────────────────────────────────────────────────┐  │
│  │                          VNet (10.0.0.0/16)                                │  │
│  │  ┌────────────────────┐  ┌────────────────────┐  ┌────────────────────┐   │  │
│  │  │  Subnet: AKS       │  │  Subnet: Database  │  │  Subnet: Redis     │   │  │
│  │  │  10.0.0.0/20       │  │  10.0.16.0/24      │  │  10.0.17.0/24      │   │  │
│  │  │  ┌──────────────┐  │  │  ┌──────────────┐  │  │  ┌──────────────┐  │   │  │
│  │  │  │ AKS Cluster  │  │  │  │ Azure DB for │  │  │  │ Azure Cache  │  │   │  │
│  │  │  │ - System Pool│  │  │  │ PostgreSQL   │  │  │  │ for Redis    │  │   │  │
│  │  │  │ - Sandbox    │  │  │  │ Flex Server  │  │  │  │              │  │   │  │
│  │  │  │   Pool       │  │  │  └──────────────┘  │  │  └──────────────┘  │   │  │
│  │  │  └──────────────┘  │  │                    │  │                    │   │  │
│  │  └────────────────────┘  └────────────────────┘  └────────────────────┘   │  │
│  └───────────────────────────────────────────────────────────────────────────┘  │
│                                                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐ │
│  │ Azure Blob   │  │ Key Vault    │  │ Application  │  │  Container Registry  │ │
│  │ Storage      │  │ Secrets      │  │ Gateway      │  │  (ACR)               │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘

Step 1: Set Up Azure Environment

# Login to Azure
az login

# Set subscription
export SUBSCRIPTION_ID="your-subscription-id"
az account set --subscription $SUBSCRIPTION_ID

# Set variables
export RESOURCE_GROUP="stateset-sandbox-rg"
export LOCATION="eastus"
export AKS_CLUSTER="stateset-sandbox-aks"

# Register required providers
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.DBforPostgreSQL
az provider register --namespace Microsoft.Cache
az provider register --namespace Microsoft.KeyVault
az provider register --namespace Microsoft.Storage
az provider register --namespace Microsoft.Network

# Create resource group
az group create --name $RESOURCE_GROUP --location $LOCATION

Step 2: Create Virtual Network

# Create VNet
az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-vnet \
  --address-prefix 10.0.0.0/16

# Create subnet for AKS
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name stateset-sandbox-vnet \
  --name aks-subnet \
  --address-prefix 10.0.0.0/20

# Create subnet for PostgreSQL
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name stateset-sandbox-vnet \
  --name db-subnet \
  --address-prefix 10.0.16.0/24 \
  --delegations Microsoft.DBforPostgreSQL/flexibleServers

# Create subnet for Redis
az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name stateset-sandbox-vnet \
  --name redis-subnet \
  --address-prefix 10.0.17.0/24

# Get subnet IDs
AKS_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RESOURCE_GROUP \
  --vnet-name stateset-sandbox-vnet \
  --name aks-subnet \
  --query id -o tsv)

DB_SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RESOURCE_GROUP \
  --vnet-name stateset-sandbox-vnet \
  --name db-subnet \
  --query id -o tsv)

Step 3: Create Azure Container Registry

# Create ACR
az acr create \
  --resource-group $RESOURCE_GROUP \
  --name statesetsandboxacr \
  --sku Premium \
  --admin-enabled false

# Get ACR ID
ACR_ID=$(az acr show --name statesetsandboxacr --query id -o tsv)

Step 4: Create AKS Cluster

# Create managed identity for AKS
az identity create \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-identity

IDENTITY_ID=$(az identity show \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-identity \
  --query id -o tsv)

IDENTITY_CLIENT_ID=$(az identity show \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-identity \
  --query clientId -o tsv)

# Create AKS cluster
az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --location $LOCATION \
  --kubernetes-version 1.29 \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --network-plugin azure \
  --network-policy azure \
  --vnet-subnet-id $AKS_SUBNET_ID \
  --service-cidr 10.1.0.0/16 \
  --dns-service-ip 10.1.0.10 \
  --enable-managed-identity \
  --assign-identity $IDENTITY_ID \
  --enable-workload-identity \
  --enable-oidc-issuer \
  --enable-addons monitoring \
  --attach-acr statesetsandboxacr \
  --nodepool-name systempool \
  --nodepool-labels node-type=controller \
  --zones 1 2 3 \
  --tier standard

# Add sandbox node pool with Kata Containers
az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $AKS_CLUSTER \
  --name sandboxpool \
  --node-count 2 \
  --node-vm-size Standard_D8s_v3 \
  --node-osdisk-size 100 \
  --workload-runtime KataMshvVmIsolation \
  --os-sku AzureLinux \
  --labels node-type=sandbox isolation=kata \
  --node-taints stateset.com/sandbox=true:NoSchedule \
  --min-count 0 \
  --max-count 20 \
  --enable-cluster-autoscaler \
  --zones 1 2 3

# Get credentials
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER

Alternative: gVisor Node Pool

If you prefer gVisor over Kata:

# Add gVisor node pool (requires custom VM image)
az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $AKS_CLUSTER \
  --name gvisorpool \
  --node-count 2 \
  --node-vm-size Standard_D8s_v3 \
  --labels node-type=sandbox isolation=gvisor \
  --node-taints stateset.com/sandbox=true:NoSchedule \
  --min-count 0 \
  --max-count 20 \
  --enable-cluster-autoscaler

Step 5: Create Azure Database for PostgreSQL

# Create private DNS zone
az network private-dns zone create \
  --resource-group $RESOURCE_GROUP \
  --name privatelink.postgres.database.azure.com

# Link DNS zone to VNet
az network private-dns link vnet create \
  --resource-group $RESOURCE_GROUP \
  --zone-name privatelink.postgres.database.azure.com \
  --name stateset-sandbox-link \
  --virtual-network stateset-sandbox-vnet \
  --registration-enabled false

# Create PostgreSQL Flexible Server
az postgres flexible-server create \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-pg \
  --location $LOCATION \
  --admin-user pgadmin \
  --admin-password "YOUR_STRONG_PASSWORD" \
  --sku-name Standard_D4s_v3 \
  --tier GeneralPurpose \
  --version 15 \
  --storage-size 128 \
  --subnet $DB_SUBNET_ID \
  --private-dns-zone privatelink.postgres.database.azure.com \
  --high-availability ZoneRedundant \
  --backup-retention 7

# Create database
az postgres flexible-server db create \
  --resource-group $RESOURCE_GROUP \
  --server-name stateset-sandbox-pg \
  --database-name sandbox

# Get connection string
PG_HOST=$(az postgres flexible-server show \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-pg \
  --query fullyQualifiedDomainName -o tsv)

Step 6: Create Azure Cache for Redis

# Create Redis cache
az redis create \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-redis \
  --location $LOCATION \
  --sku Premium \
  --vm-size P1 \
  --subnet-id /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/stateset-sandbox-vnet/subnets/redis-subnet \
  --enable-non-ssl-port false \
  --minimum-tls-version 1.2 \
  --zones 1 2 3

# Get Redis connection info
REDIS_HOST=$(az redis show \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-redis \
  --query hostName -o tsv)

REDIS_KEY=$(az redis list-keys \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-redis \
  --query primaryKey -o tsv)

Step 7: Create Azure Blob Storage

# Create storage account
az storage account create \
  --resource-group $RESOURCE_GROUP \
  --name statesetsandboxstorage \
  --location $LOCATION \
  --sku Standard_ZRS \
  --kind StorageV2 \
  --min-tls-version TLS1_2 \
  --allow-blob-public-access false \
  --https-only true

# Create container for artifacts
az storage container create \
  --account-name statesetsandboxstorage \
  --name artifacts \
  --auth-mode login

# Enable versioning
az storage account blob-service-properties update \
  --resource-group $RESOURCE_GROUP \
  --account-name statesetsandboxstorage \
  --enable-versioning true

# Set lifecycle management
az storage account management-policy create \
  --resource-group $RESOURCE_GROUP \
  --account-name statesetsandboxstorage \
  --policy '{
    "rules": [{
      "name": "deleteOldArtifacts",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {"blobTypes": ["blockBlob"], "prefixMatch": ["artifacts/"]},
        "actions": {
          "baseBlob": {"delete": {"daysAfterModificationGreaterThan": 90}},
          "version": {"delete": {"daysAfterCreationGreaterThan": 30}}
        }
      }
    }]
  }'

Step 8: Create Azure Key Vault

# Create Key Vault
az keyvault create \
  --resource-group $RESOURCE_GROUP \
  --name stateset-sandbox-kv \
  --location $LOCATION \
  --sku premium \
  --enable-rbac-authorization

# Get Key Vault ID
KEYVAULT_ID=$(az keyvault show --name stateset-sandbox-kv --query id -o tsv)

# Add secrets
az keyvault secret set \
  --vault-name stateset-sandbox-kv \
  --name jwt-secret \
  --value "$(openssl rand -base64 32)"

az keyvault secret set \
  --vault-name stateset-sandbox-kv \
  --name database-encryption-key \
  --value "$(openssl rand -hex 32)"

az keyvault secret set \
  --vault-name stateset-sandbox-kv \
  --name stripe-secret-key \
  --value "your-stripe-secret-key"

az keyvault secret set \
  --vault-name stateset-sandbox-kv \
  --name redis-password \
  --value "$REDIS_KEY"

# Grant AKS identity access to Key Vault
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee $IDENTITY_CLIENT_ID \
  --scope $KEYVAULT_ID

Step 9: Configure Workload Identity

# Get OIDC issuer URL
OIDC_ISSUER=$(az aks show \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --query oidcIssuerProfile.issuerUrl -o tsv)

# Create user-assigned identity for the controller
az identity create \
  --resource-group $RESOURCE_GROUP \
  --name sandbox-controller-identity

CONTROLLER_IDENTITY_CLIENT_ID=$(az identity show \
  --resource-group $RESOURCE_GROUP \
  --name sandbox-controller-identity \
  --query clientId -o tsv)

# Create federated credential
az identity federated-credential create \
  --name sandbox-controller-fed \
  --identity-name sandbox-controller-identity \
  --resource-group $RESOURCE_GROUP \
  --issuer $OIDC_ISSUER \
  --subject system:serviceaccount:stateset-sandbox:sandbox-controller

# Grant permissions
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee $CONTROLLER_IDENTITY_CLIENT_ID \
  --scope $KEYVAULT_ID

az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee $CONTROLLER_IDENTITY_CLIENT_ID \
  --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/statesetsandboxstorage

Step 10: Deploy StateSet Sandbox

Create Namespace

kubectl apply -f k8s/namespace.yaml

Configure Service Account with Workload Identity

# azure-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sandbox-controller
  namespace: stateset-sandbox
  annotations:
    azure.workload.identity/client-id: "YOUR_CONTROLLER_IDENTITY_CLIENT_ID"
  labels:
    azure.workload.identity/use: "true"

Install CSI Secrets Store Driver

# Install Secrets Store CSI driver
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure \
  --namespace kube-system

Create SecretProviderClass

# azure-secret-provider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-secrets
  namespace: stateset-sandbox
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "YOUR_CONTROLLER_IDENTITY_CLIENT_ID"
    keyvaultName: "stateset-sandbox-kv"
    tenantId: "YOUR_TENANT_ID"
    objects: |
      array:
        - |
          objectName: jwt-secret
          objectType: secret
        - |
          objectName: database-encryption-key
          objectType: secret
        - |
          objectName: stripe-secret-key
          objectType: secret
        - |
          objectName: redis-password
          objectType: secret
  secretObjects:
    - secretName: sandbox-controller-auth
      type: Opaque
      data:
        - objectName: jwt-secret
          key: jwt-secret
        - objectName: database-encryption-key
          key: database-encryption-key
        - objectName: stripe-secret-key
          key: stripe-secret-key
        - objectName: redis-password
          key: redis-password

Create ConfigMap

# azure-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: sandbox-controller-config
  namespace: stateset-sandbox
data:
  # Azure-specific configuration
  SANDBOX_IMAGE: "statesetsandboxacr.azurecr.io/stateset-sandbox:latest"

  # Database (Azure PostgreSQL)
  DATABASE_URL: "postgres://pgadmin:PASSWORD@stateset-sandbox-pg.postgres.database.azure.com:5432/sandbox?sslmode=require"

  # Redis (Azure Cache)
  REDIS_HOST: "stateset-sandbox-redis.redis.cache.windows.net"
  REDIS_PORT: "6380"
  REDIS_TLS: "true"

  # Storage (Azure Blob)
  STORAGE_PROVIDER: "azure"
  STORAGE_BUCKET: "artifacts"
  AZURE_STORAGE_ACCOUNT: "statesetsandboxstorage"
  # AZURE_STORAGE_SAS_TOKEN is injected via secret in the Azure kustomize overlay.
  # AZURE_STORAGE_ENDPOINT: "https://statesetsandboxstorage.blob.core.windows.net"

  # Isolation (Kata Containers on AKS)
  DEFAULT_ISOLATION: "kata"
  SANDBOX_RUNTIME_CLASS: "kata-mshv-vm-isolation"

  # Warm pool
  WARM_POOL_ENABLED: "true"
  WARM_POOL_SIZE: "5"

  # CORS
  CORS_ORIGIN: "https://sandbox.yourdomain.com"
  APP_URL: "https://api.sandbox.yourdomain.com"

  # Logging
  LOG_LEVEL: "info"
  NODE_ENV: "production"

Update Deployment for Azure

# azure-deployment-patch.yaml
spec:
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: sandbox-controller
      containers:
        - name: sandbox-controller
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: azure-keyvault-secrets

Deploy Components

kubectl apply -f azure-service-account.yaml
kubectl apply -f azure-secret-provider.yaml
kubectl apply -f azure-configmap.yaml
kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/network-policy.yaml
kubectl apply -f k8s/resource-quota.yaml

Create Kata RuntimeClass

# kata-runtimeclass.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-mshv-vm-isolation
handler: kata-mshv-vm-isolation
scheduling:
  nodeSelector:
    kubernetes.azure.com/kata-mshv-vm-isolation: "true"
  tolerations:
    - key: stateset.com/sandbox
      operator: Exists
      effect: NoSchedule

Step 11: Configure Application Gateway Ingress

Option A: Application Gateway Ingress Controller (AGIC)

# Enable AGIC addon
az aks addon update \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --addon ingress-appgw \
  --appgw-name stateset-sandbox-agw \
  --appgw-subnet-cidr 10.0.20.0/24

# azure-ingress-agic.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sandbox-controller
  namespace: stateset-sandbox
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    appgw.ingress.kubernetes.io/backend-protocol: "http"
    appgw.ingress.kubernetes.io/health-probe-path: "/health"
    appgw.ingress.kubernetes.io/request-timeout: "300"
    # WAF policy (optional)
    # appgw.ingress.kubernetes.io/waf-policy-for-path: "/subscriptions/.../applicationGatewayWebApplicationFirewallPolicies/sandbox-waf"
spec:
  tls:
    - hosts:
        - api.sandbox.yourdomain.com
      secretName: sandbox-tls-secret
  rules:
    - host: api.sandbox.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sandbox-controller
                port:
                  number: 80

Option B: NGINX Ingress Controller

# Install NGINX ingress controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

Configure TLS Certificate

# Install cert-manager
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

# Create ClusterIssuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - http01:
          ingress:
            class: nginx
EOF

Step 12: Set Up Monitoring

Azure Monitor for Containers

Already enabled during cluster creation. Configure additional monitoring:

# Enable Container Insights with Prometheus metrics
az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --enable-azure-monitor-metrics

Log Analytics Queries

// Query sandbox creation events
ContainerLog
| where LogEntry contains "sandbox.created"
| project TimeGenerated, LogEntry
| order by TimeGenerated desc
| take 100

// Query error rates
ContainerLog
| where LogEntry contains "error"
| summarize ErrorCount = count() by bin(TimeGenerated, 5m)
| render timechart

// Query sandbox metrics
InsightsMetrics
| where Name == "stateset_sandbox_active_sandboxes"
| summarize avg(Val) by bin(TimeGenerated, 1m)
| render timechart

Create Azure Alerts

# Create action group
az monitor action-group create \
  --resource-group $RESOURCE_GROUP \
  --name sandbox-alerts \
  --short-name sandbox \
  --email-receiver name=admin email=admin@example.com

# Create alert for high error rate
az monitor metrics alert create \
  --resource-group $RESOURCE_GROUP \
  --name "high-error-rate" \
  --scopes "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.ContainerService/managedClusters/$AKS_CLUSTER" \
  --condition "avg percentage>=5" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action sandbox-alerts \
  --description "High error rate in StateSet Sandbox"

Prometheus & Grafana (Optional)

# Install Prometheus stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

# Apply ServiceMonitor
kubectl apply -f k8s/prometheus/service-monitor.yaml
kubectl apply -f k8s/prometheus/alerts.yaml

Step 13: Configure Autoscaling

Cluster Autoscaler

Already enabled on sandbox node pool. Configure behavior:

# Update autoscaler profile
az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --cluster-autoscaler-profile \
    scale-down-delay-after-add=10m \
    scale-down-unneeded-time=10m \
    max-graceful-termination-sec=600

Horizontal Pod Autoscaler

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sandbox-controller
  namespace: stateset-sandbox
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sandbox-controller
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

KEDA (Event-Driven Autoscaling)

# Install KEDA
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

# keda-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sandbox-controller-scaler
  namespace: stateset-sandbox
spec:
  scaleTargetRef:
    name: sandbox-controller
  minReplicaCount: 3
  maxReplicaCount: 20
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-server.monitoring.svc.cluster.local
        metricName: stateset_sandbox_active_sandboxes
        query: stateset_sandbox_active_sandboxes
        threshold: "50"

Security Hardening

Azure Policy for AKS

# Enable Azure Policy addon
az aks enable-addons \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --addons azure-policy

# Assign built-in policies
az policy assignment create \
  --name "deny-privileged-containers" \
  --policy "95edb821-ddaf-4404-9732-666045e056b4" \
  --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.ContainerService/managedClusters/$AKS_CLUSTER"

Microsoft Defender for Containers

# Enable Defender for Containers
az security pricing create \
  --name Containers \
  --tier Standard

# Enable Defender for AKS cluster
az aks update \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --enable-defender

Private Endpoints

# Create private endpoint for PostgreSQL
az network private-endpoint create \
  --resource-group $RESOURCE_GROUP \
  --name pg-private-endpoint \
  --vnet-name stateset-sandbox-vnet \
  --subnet aks-subnet \
  --private-connection-resource-id $(az postgres flexible-server show --resource-group $RESOURCE_GROUP --name stateset-sandbox-pg --query id -o tsv) \
  --group-id postgresqlServer \
  --connection-name pg-connection

# Create private endpoint for Storage
az network private-endpoint create \
  --resource-group $RESOURCE_GROUP \
  --name storage-private-endpoint \
  --vnet-name stateset-sandbox-vnet \
  --subnet aks-subnet \
  --private-connection-resource-id $(az storage account show --resource-group $RESOURCE_GROUP --name statesetsandboxstorage --query id -o tsv) \
  --group-id blob \
  --connection-name storage-connection

Confidential Computing (Optional)

For maximum isolation with Confidential VMs:

# Add confidential computing node pool
az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $AKS_CLUSTER \
  --name confpool \
  --node-count 2 \
  --node-vm-size Standard_DC8s_v3 \
  --os-sku AzureLinux \
  --labels node-type=sandbox isolation=confidential \
  --node-taints stateset.com/sandbox=true:NoSchedule \
  --min-count 0 \
  --max-count 10 \
  --enable-cluster-autoscaler \
  --enable-sgxquotehelper

Cost Optimization

Azure Reserved Instances

Purchase reserved capacity for predictable workloads:

az reservations reservation-order purchase \
  --reservation-order-id "00000000-0000-0000-0000-000000000000" \
  --sku Standard_D4s_v3 \
  --location eastus \
  --billing-scope-id "/subscriptions/$SUBSCRIPTION_ID" \
  --term P1Y \
  --billing-plan Monthly \
  --quantity 3

Spot VMs for Sandbox Nodes

# Add spot node pool
az aks nodepool add \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $AKS_CLUSTER \
  --name spotpool \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --node-vm-size Standard_D8s_v3 \
  --labels node-type=sandbox lifecycle=spot \
  --node-taints stateset.com/sandbox=true:NoSchedule,kubernetes.azure.com/scalesetpriority=spot:NoSchedule \
  --min-count 0 \
  --max-count 20 \
  --enable-cluster-autoscaler

Estimated Monthly Costs

Resource	Configuration	Est. Cost
AKS Management	Free tier	$0
System Node Pool	3x Standard_D4s_v3	~$350
Sandbox Node Pool	2-10x Standard_D8s_v3 (Spot)	~ $150-$ 750
Azure PostgreSQL	Standard_D4s_v3 HA	~$450
Azure Cache Redis	Premium P1	~$300
Application Gateway	WAF v2	~$280
Blob Storage	100GB + transactions	~$25
Egress	~500GB/month	~$45
Key Vault	Operations	~$5
Total		~ $1,600-$ 2,200/month

Troubleshooting

Common Issues

Pod identity not working

# Check workload identity configuration
kubectl describe serviceaccount sandbox-controller -n stateset-sandbox

# Verify federated credential
az identity federated-credential show \
  --name sandbox-controller-fed \
  --identity-name sandbox-controller-identity \
  --resource-group $RESOURCE_GROUP

Database connection issues

# Test connection from a pod
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
  psql "postgres://pgadmin:pass@stateset-sandbox-pg.postgres.database.azure.com:5432/sandbox?sslmode=require"

Kata containers not working

# Check node pool configuration
az aks nodepool show \
  --resource-group $RESOURCE_GROUP \
  --cluster-name $AKS_CLUSTER \
  --name sandboxpool

# Check RuntimeClass
kubectl get runtimeclass kata-mshv-vm-isolation -o yaml

Useful Commands

# View AKS diagnostics
az aks show \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --query "agentPoolProfiles[].{name:name, count:count, vmSize:vmSize}"

# Stream logs
kubectl logs -f -l app=sandbox-controller -n stateset-sandbox

# Check node status
kubectl get nodes -o wide

# View resource usage
kubectl top nodes
kubectl top pods -n stateset-sandbox

​Azure Deployment Guide (AKS)

​Prerequisites

​Quick Start (Kustomize Overlay)

​Architecture Overview

​Step 1: Set Up Azure Environment

​Step 2: Create Virtual Network

​Step 3: Create Azure Container Registry

​Step 4: Create AKS Cluster

​Alternative: gVisor Node Pool

​Step 5: Create Azure Database for PostgreSQL

​Step 6: Create Azure Cache for Redis

​Step 7: Create Azure Blob Storage

​Step 8: Create Azure Key Vault

​Step 9: Configure Workload Identity

​Step 10: Deploy StateSet Sandbox

​Create Namespace

​Configure Service Account with Workload Identity

​Install CSI Secrets Store Driver

​Create SecretProviderClass

​Create ConfigMap

​Update Deployment for Azure

​Deploy Components

​Create Kata RuntimeClass

​Step 11: Configure Application Gateway Ingress

​Option A: Application Gateway Ingress Controller (AGIC)

​Option B: NGINX Ingress Controller

​Configure TLS Certificate

​Step 12: Set Up Monitoring

​Azure Monitor for Containers

​Log Analytics Queries

​Create Azure Alerts

​Prometheus & Grafana (Optional)

​Step 13: Configure Autoscaling

​Cluster Autoscaler

​Horizontal Pod Autoscaler

​KEDA (Event-Driven Autoscaling)

​Security Hardening

​Azure Policy for AKS

​Microsoft Defender for Containers

​Private Endpoints

​Confidential Computing (Optional)

​Cost Optimization

​Azure Reserved Instances

​Spot VMs for Sandbox Nodes

​Estimated Monthly Costs

​Troubleshooting

​Common Issues

​Useful Commands

​Support Resources