GCP Deployment Guide (GKE)
Complete guide for deploying StateSet Sandbox on Google Kubernetes Engine (GKE).Prerequisites
- gcloud CLI configured with appropriate permissions
- kubectl installed
- Helm 3.x installed
- Domain name for API endpoint
- GCP Project with billing enabled
Quick Start (Kustomize Overlay)
Use the GCP-specific overlay to apply the correct ingress/controller defaults:kubectl apply -k k8s/overlays/gcp
# or
./scripts/deploy-k8s.sh gcp
# or (CLI)
stateset deploy gcp
stateset deploy gcp --preflight --wait --smoke
export STATESET_API_KEY="your_api_key"
export STATESET_API_URL="https://api.sandbox.stateset.com/api/v1"
read -rsp "Admin password: " ADMIN_PASSWORD; echo
bash scripts/create-controller-secret.sh \
--admin-username admin \
--admin-password "$ADMIN_PASSWORD"
unset ADMIN_PASSWORD
Architecture Overview
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Google Cloud │
│ ┌───────────────────────────────────────────────────────────────────────────┐ │
│ │ VPC (10.0.0.0/16) │ │
│ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ GKE Autopilot / Standard │ │ │
│ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │
│ │ │ │ Node Pool: │ │ Node Pool: │ │ Node Pool: │ │ │ │
│ │ │ │ Controller │ │ gVisor │ │ Confidential │ │ │ │
│ │ │ │ n2-standard-4 │ │ c2-standard-8 │ │ c2d-standard-8 │ │ │ │
│ │ │ │ (3 nodes) │ │ (auto-scale) │ │ (optional) │ │ │ │
│ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │ Cloud SQL │ │ Memorystore │ │ Cloud Storage│ │ Secret Manager │ │
│ │ PostgreSQL │ │ Redis │ │ Artifacts │ │ API Keys / Secrets │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Cloud Load Balancer (HTTPS) │ │
│ │ api.sandbox.yourdomain.com │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
Step 1: Set Up GCP Project
# Set project
export PROJECT_ID="your-project-id"
export REGION="us-central1"
export ZONE="us-central1-a"
gcloud config set project $PROJECT_ID
gcloud config set compute/region $REGION
gcloud config set compute/zone $ZONE
# Enable required APIs
gcloud services enable \
container.googleapis.com \
sqladmin.googleapis.com \
redis.googleapis.com \
secretmanager.googleapis.com \
certificatemanager.googleapis.com \
compute.googleapis.com \
artifactregistry.googleapis.com \
cloudkms.googleapis.com
Step 2: Create VPC Network
# Create VPC
gcloud compute networks create stateset-sandbox-vpc \
--subnet-mode=custom
# Create subnet for GKE nodes
gcloud compute networks subnets create stateset-sandbox-subnet \
--network=stateset-sandbox-vpc \
--region=$REGION \
--range=10.0.0.0/20 \
--secondary-range=pods=10.4.0.0/14,services=10.8.0.0/20
# Create Cloud NAT for egress
gcloud compute routers create stateset-sandbox-router \
--network=stateset-sandbox-vpc \
--region=$REGION
gcloud compute routers nats create stateset-sandbox-nat \
--router=stateset-sandbox-router \
--region=$REGION \
--nat-all-subnet-ip-ranges \
--auto-allocate-nat-external-ips
Step 3: Create GKE Cluster
Option A: GKE Standard with gVisor
# Create cluster
gcloud container clusters create stateset-sandbox \
--region=$REGION \
--num-nodes=1 \
--machine-type=n2-standard-4 \
--disk-size=50GB \
--network=stateset-sandbox-vpc \
--subnetwork=stateset-sandbox-subnet \
--cluster-secondary-range-name=pods \
--services-secondary-range-name=services \
--enable-ip-alias \
--enable-network-policy \
--workload-pool=${PROJECT_ID}.svc.id.goog \
--enable-shielded-nodes \
--shielded-secure-boot \
--shielded-integrity-monitoring \
--release-channel=regular \
--node-labels=node-type=controller \
--addons=HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver
# Create gVisor-enabled node pool for sandboxes
gcloud container node-pools create sandbox-gvisor \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=c2-standard-8 \
--disk-size=100GB \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=20 \
--enable-autoscaling \
--sandbox=type=gvisor \
--node-labels=node-type=sandbox,isolation=gvisor \
--node-taints=stateset.com/sandbox=true:NoSchedule \
--shielded-secure-boot \
--shielded-integrity-monitoring
# Get credentials
gcloud container clusters get-credentials stateset-sandbox --region=$REGION
Option B: GKE Autopilot
# Create Autopilot cluster (automatically manages nodes)
gcloud container clusters create-auto stateset-sandbox \
--region=$REGION \
--network=stateset-sandbox-vpc \
--subnetwork=stateset-sandbox-subnet \
--cluster-secondary-range-name=pods \
--services-secondary-range-name=services \
--workload-pool=${PROJECT_ID}.svc.id.goog \
--release-channel=regular
# Get credentials
gcloud container clusters get-credentials stateset-sandbox --region=$REGION
Step 4: Create Cloud SQL (PostgreSQL)
# Create Cloud SQL instance
gcloud sql instances create stateset-sandbox \
--database-version=POSTGRES_15 \
--tier=db-custom-4-15360 \
--region=$REGION \
--network=stateset-sandbox-vpc \
--no-assign-ip \
--enable-google-private-path \
--storage-size=100GB \
--storage-type=SSD \
--storage-auto-increase \
--backup-start-time=03:00 \
--availability-type=REGIONAL \
--deletion-protection
# Create database
gcloud sql databases create sandbox --instance=stateset-sandbox
# Create user
gcloud sql users create sandbox-user \
--instance=stateset-sandbox \
--password="YOUR_STRONG_PASSWORD"
# Get connection name for Kubernetes
gcloud sql instances describe stateset-sandbox --format='value(connectionName)'
Step 5: Create Memorystore (Redis)
# Create Redis instance
gcloud redis instances create stateset-sandbox \
--size=2 \
--region=$REGION \
--network=stateset-sandbox-vpc \
--tier=STANDARD_HA \
--redis-version=redis_7_0 \
--transit-encryption-mode=SERVER_AUTHENTICATION
# Get host and port
gcloud redis instances describe stateset-sandbox --region=$REGION \
--format='value(host,port)'
Step 6: Create Cloud Storage Bucket
# Create bucket
gcloud storage buckets create gs://stateset-sandbox-artifacts-${PROJECT_ID} \
--location=$REGION \
--uniform-bucket-level-access \
--public-access-prevention
# Set lifecycle rule
cat <<EOF > lifecycle.json
{
"rule": [{
"action": {"type": "Delete"},
"condition": {"age": 90, "matchesPrefix": ["artifacts/"]}
}]
}
EOF
gcloud storage buckets update gs://stateset-sandbox-artifacts-${PROJECT_ID} \
--lifecycle-file=lifecycle.json
Step 7: Set Up Secret Manager
# Create secrets
echo -n "$(openssl rand -base64 32)" | \
gcloud secrets create jwt-secret --data-file=-
echo -n "$(openssl rand -hex 32)" | \
gcloud secrets create database-encryption-key --data-file=-
echo -n "your-stripe-secret-key" | \
gcloud secrets create stripe-secret-key --data-file=-
# Create service account for secrets access
gcloud iam service-accounts create sandbox-controller \
--display-name="StateSet Sandbox Controller"
# Grant secrets access
gcloud secrets add-iam-policy-binding jwt-secret \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
gcloud secrets add-iam-policy-binding database-encryption-key \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
gcloud secrets add-iam-policy-binding stripe-secret-key \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
# Grant GCS access
gsutil iam ch serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com:objectAdmin \
gs://stateset-sandbox-artifacts-${PROJECT_ID}
Step 8: Configure Workload Identity
# Allow Kubernetes service account to use GCP service account
gcloud iam service-accounts add-iam-policy-binding \
sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="serviceAccount:${PROJECT_ID}.svc.id.goog[stateset-sandbox/sandbox-controller]"
Step 9: Deploy StateSet Sandbox
Create Namespace
kubectl apply -f k8s/namespace.yaml
Configure Service Account for Workload Identity
# gcp-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: sandbox-controller
namespace: stateset-sandbox
annotations:
iam.gke.io/gcp-service-account: sandbox-controller@YOUR_PROJECT_ID.iam.gserviceaccount.com
Create ConfigMap
# gcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-controller-config
namespace: stateset-sandbox
data:
# GCP-specific configuration
SANDBOX_IMAGE: "us-central1-docker.pkg.dev/${PROJECT_ID}/stateset/sandbox:latest"
# Database (Cloud SQL via private IP)
DATABASE_URL: "postgres://sandbox-user:PASSWORD@CLOUD_SQL_PRIVATE_IP:5432/sandbox?sslmode=require"
# Redis (Memorystore)
REDIS_HOST: "MEMORYSTORE_HOST"
REDIS_PORT: "6379"
REDIS_TLS: "true"
# Storage (GCS)
STORAGE_PROVIDER: "gcs"
STORAGE_BUCKET: "stateset-sandbox-artifacts-${PROJECT_ID}"
GCS_PROJECT_ID: "${PROJECT_ID}"
# Isolation
DEFAULT_ISOLATION: "gvisor"
SANDBOX_RUNTIME_CLASS: "gvisor"
# Warm pool
WARM_POOL_ENABLED: "true"
WARM_POOL_SIZE: "5"
# Exec agent runtime (optional)
# Set to "go" to use the Go exec agent (execd) instead of the default Node.js agent.
# The Go binary is included in the sandbox base image and has a significantly lower
# memory footprint (~8MB RSS vs ~45MB for Node.js).
# EXEC_AGENT_RUNTIME: "go"
# CORS
CORS_ORIGIN: "https://sandbox.yourdomain.com"
APP_URL: "https://api.sandbox.yourdomain.com"
# Logging (Cloud Logging integration)
LOG_LEVEL: "info"
NODE_ENV: "production"
Create External Secrets
# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace
# gcp-external-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: gcp-secret-manager
namespace: stateset-sandbox
spec:
provider:
gcpsm:
projectID: YOUR_PROJECT_ID
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: sandbox-controller-secrets
namespace: stateset-sandbox
spec:
refreshInterval: 1h
secretStoreRef:
name: gcp-secret-manager
kind: SecretStore
target:
name: sandbox-controller-auth
data:
- secretKey: jwt-secret
remoteRef:
key: jwt-secret
- secretKey: database-encryption-key
remoteRef:
key: database-encryption-key
- secretKey: stripe-secret-key
remoteRef:
key: stripe-secret-key
Apply Custom Resource Definitions
CRDs must be applied before the main deployment so the API server recognises StateSet resource types:# Apply Custom Resource Definitions (before main deployment)
kubectl apply -f k8s/crds/
# Verify CRDs are registered
kubectl get crd | grep stateset
# statesandboxes.sandbox.stateset.io
# warmpools.sandbox.stateset.io
# Optional: Enable CRD-based sandbox management
# Set SANDBOX_BACKEND=crd in your configmap to use declarative sandbox management
Deploy Components
kubectl apply -f gcp-service-account.yaml
kubectl apply -f gcp-configmap.yaml
kubectl apply -f gcp-external-secrets.yaml
kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/network-policy.yaml
kubectl apply -f k8s/resource-quota.yaml
Create gVisor RuntimeClass
# gvisor-runtimeclass.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
scheduling:
nodeSelector:
sandbox.gke.io/runtime: gvisor
tolerations:
- key: stateset.com/sandbox
operator: Exists
effect: NoSchedule
kubectl apply -f gvisor-runtimeclass.yaml
CRD Usage Examples
With the CRDs installed you can manage sandboxes and warm pools declaratively:# Example: Create a sandbox via CRD
apiVersion: sandbox.stateset.io/v1alpha1
kind: StateSandbox
metadata:
name: my-sandbox
namespace: stateset-sandbox
spec:
orgId: "org-123"
image: "stateset/sandbox:latest"
resources:
cpu: "2"
memory: "2Gi"
timeoutSeconds: 600
isolation: container
kubectl apply -f sandbox.yaml
kubectl get ssb # short name
kubectl get warmpools # or kubectl get wp
Step 10: Configure Load Balancer & SSL
Managed SSL Certificate
# Create managed certificate
gcloud compute ssl-certificates create stateset-sandbox-cert \
--domains=api.sandbox.yourdomain.com \
--global
# Reserve static IP
gcloud compute addresses create stateset-sandbox-ip --global
Ingress with GCE Load Balancer
# gcp-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sandbox-controller
namespace: stateset-sandbox
annotations:
kubernetes.io/ingress.global-static-ip-name: "stateset-sandbox-ip"
networking.gke.io/managed-certificates: "sandbox-controller-cert"
kubernetes.io/ingress.class: "gce"
# Enable Cloud Armor (optional)
# networking.gke.io/v1beta1.FrontendConfig: "sandbox-frontend-config"
spec:
rules:
- host: api.sandbox.yourdomain.com
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: sandbox-controller
port:
number: 80
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: sandbox-controller-cert
namespace: stateset-sandbox
spec:
domains:
- api.sandbox.yourdomain.com
kubectl apply -f gcp-ingress.yaml
# Get the external IP
kubectl get ingress sandbox-controller -n stateset-sandbox
# Update DNS to point to this IP
Cloud Armor Security Policy (Optional)
# Create security policy
gcloud compute security-policies create stateset-sandbox-policy \
--description="StateSet Sandbox security policy"
# Add rate limiting rule
gcloud compute security-policies rules create 1000 \
--security-policy=stateset-sandbox-policy \
--action=rate-based-ban \
--rate-limit-threshold-count=100 \
--rate-limit-threshold-interval-sec=60 \
--ban-duration-sec=300 \
--conform-action=allow \
--exceed-action=deny-429 \
--enforce-on-key=IP
# Add geo-blocking (optional)
gcloud compute security-policies rules create 2000 \
--security-policy=stateset-sandbox-policy \
--action=deny-403 \
--expression="origin.region_code == 'CN'"
# Create frontend config
cat <<EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: sandbox-frontend-config
namespace: stateset-sandbox
spec:
sslPolicy: modern-ssl-policy
redirectToHttps:
enabled: true
EOF
Step 11: Set Up Monitoring
Cloud Monitoring
GKE automatically sends metrics to Cloud Monitoring. Enable additional features:# Enable managed Prometheus
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--enable-managed-prometheus
Custom Dashboards
Create monitoring dashboard in Cloud Console or via API:# Create dashboard from JSON
gcloud monitoring dashboards create \
--config-from-file=monitoring/gcp-dashboard.json
Alerting Policies
# Create alert for high error rate
gcloud alpha monitoring policies create \
--display-name="StateSet Sandbox High Error Rate" \
--condition-display-name="Error rate > 5%" \
--condition-filter='metric.type="custom.googleapis.com/stateset/http_error_rate" resource.type="k8s_container"' \
--condition-threshold-value=0.05 \
--condition-threshold-duration=300s \
--notification-channels=CHANNEL_ID
Cloud Logging
Query logs:gcloud logging read 'resource.type="k8s_container" resource.labels.namespace_name="stateset-sandbox"' \
--limit=100 \
--format=json
Step 12: Configure Autoscaling
Cluster Autoscaler
Already enabled when creating node pool with--enable-autoscaling. Configure behavior:
# Update autoscaling profile
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--autoscaling-profile=optimize-utilization
Horizontal Pod Autoscaler
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: sandbox-controller
namespace: stateset-sandbox
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: sandbox-controller
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Security Hardening
Binary Authorization
# Enable Binary Authorization
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--enable-binauthz
# Create attestor and policy
gcloud container binauthz attestors create sandbox-attestor \
--attestation-authority-note=sandbox-note \
--attestation-authority-note-project=$PROJECT_ID
VPC Service Controls (Enterprise)
# Create service perimeter
gcloud access-context-manager perimeters create stateset-sandbox \
--title="StateSet Sandbox Perimeter" \
--resources="projects/${PROJECT_NUMBER}" \
--restricted-services="storage.googleapis.com,sqladmin.googleapis.com" \
--policy=POLICY_ID
Cloud KMS for Encryption
# Create keyring and key
gcloud kms keyrings create stateset-sandbox --location=$REGION
gcloud kms keys create sandbox-secrets-key \
--location=$REGION \
--keyring=stateset-sandbox \
--purpose=encryption
# Use for database encryption
gcloud sql instances patch stateset-sandbox \
--disk-encryption-key=projects/$PROJECT_ID/locations/$REGION/keyRings/stateset-sandbox/cryptoKeys/sandbox-secrets-key
Confidential Computing (Optional)
For maximum isolation with Confidential VMs:# Create confidential computing node pool
gcloud container node-pools create sandbox-confidential \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=n2d-standard-8 \
--enable-confidential-nodes \
--disk-size=100GB \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=10 \
--enable-autoscaling \
--node-labels=node-type=sandbox,isolation=confidential \
--node-taints=stateset.com/sandbox=true:NoSchedule
Cost Optimization
Committed Use Discounts
Purchase committed use discounts for predictable controller workloads:gcloud compute commitments create stateset-controller \
--region=$REGION \
--resources=vcpu=12,memory=48GB \
--plan=twelve-month
Preemptible VMs for Sandboxes
# Create preemptible node pool
gcloud container node-pools create sandbox-preemptible \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=c2-standard-8 \
--preemptible \
--num-nodes=2 \
--min-nodes=0 \
--max-nodes=20 \
--enable-autoscaling \
--sandbox=type=gvisor \
--node-labels=node-type=sandbox,lifecycle=preemptible \
--node-taints=stateset.com/sandbox=true:NoSchedule
Estimated Monthly Costs
| Resource | Configuration | Est. Cost |
|---|---|---|
| GKE Management | 1 cluster | $73 |
| Controller Nodes | 3x n2-standard-4 | ~$290 |
| Sandbox Nodes | 3-10x c2-standard-8 (preemptible) | ~150−500 |
| Cloud SQL | db-custom-4-15360 HA | ~$350 |
| Memorystore Redis | 2GB Standard HA | ~$175 |
| Cloud Load Balancer | 1 LB + traffic | ~$35 |
| Cloud Storage | Storage + requests | ~$20 |
| Egress | ~500GB/month | ~$60 |
| Total | ~1,150−1,500/month |
Troubleshooting
Common Issues
Cloud SQL connection issues# Check connectivity
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
psql "postgres://user:pass@CLOUD_SQL_IP:5432/sandbox?sslmode=require"
# Check node pool status
gcloud container node-pools describe sandbox-gvisor \
--cluster=stateset-sandbox \
--region=$REGION
# Check RuntimeClass
kubectl get runtimeclass gvisor -o yaml
# Check managed certificate status
kubectl describe managedcertificate sandbox-controller-cert -n stateset-sandbox
Useful Commands
# View cluster events
kubectl get events -n stateset-sandbox --sort-by='.lastTimestamp'
# Stream logs
kubectl logs -f -l app=sandbox-controller -n stateset-sandbox
# Check node conditions
kubectl describe nodes | grep -A5 "Conditions:"