GCP Deployment Guide (GKE)
Complete guide for deploying StateSet Sandbox on Google Kubernetes Engine (GKE).Prerequisites
- gcloud CLI configured with appropriate permissions
- kubectl installed
- Helm 3.x installed
- Domain name for API endpoint
- GCP Project with billing enabled
Quick Start (Kustomize Overlay)
Use the GCP-specific overlay to apply the correct ingress/controller defaults:kubectl apply -k k8s/overlays/gcp
# or
./scripts/deploy-k8s.sh gcp
# or (CLI)
stateset-sandbox deploy gcp
stateset-sandbox deploy gcp --preflight --wait --smoke
stateset-sandbox is the @stateset/sandbox-cli binary. It also installs a stateset alias, but that
name collides with @stateset/cli, whose stateset has no deploy command, so use the long name.
For smoke checks, export credentials before running deploy:
export STATESET_API_KEY="your_api_key"
export STATESET_API_URL="https://api.sandbox.stateset.app/api/v1"
read -rsp "Admin password: " ADMIN_PASSWORD; echo
bash scripts/create-controller-secret.sh \
--admin-username admin \
--admin-password "$ADMIN_PASSWORD"
unset ADMIN_PASSWORD
Architecture Overview
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Google Cloud │
│ ┌───────────────────────────────────────────────────────────────────────────┐ │
│ │ VPC (10.0.0.0/16) │ │
│ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ GKE Autopilot / Standard │ │ │
│ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │
│ │ │ │ Node Pool: │ │ Node Pool: │ │ Node Pool: │ │ │ │
│ │ │ │ Controller │ │ gVisor │ │ Confidential │ │ │ │
│ │ │ │ n2-standard-4 │ │ c2-standard-8 │ │ c2d-standard-8 │ │ │ │
│ │ │ │ (3 nodes) │ │ (auto-scale) │ │ (optional) │ │ │ │
│ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │ Cloud SQL │ │ Memorystore │ │ Cloud Storage│ │ Secret Manager │ │
│ │ PostgreSQL │ │ Redis │ │ Artifacts │ │ API Keys / Secrets │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Cloud Load Balancer (HTTPS) │ │
│ │ api.sandbox.yourdomain.com │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
Step 1: Set Up GCP Project
# Set project
export PROJECT_ID="your-project-id"
export REGION="us-central1"
export ZONE="us-central1-a"
gcloud config set project $PROJECT_ID
gcloud config set compute/region $REGION
gcloud config set compute/zone $ZONE
# Enable required APIs
gcloud services enable \
container.googleapis.com \
sqladmin.googleapis.com \
redis.googleapis.com \
secretmanager.googleapis.com \
certificatemanager.googleapis.com \
compute.googleapis.com \
artifactregistry.googleapis.com \
cloudkms.googleapis.com
Step 2: Create VPC Network
# Create VPC
gcloud compute networks create stateset-sandbox-vpc \
--subnet-mode=custom
# Create subnet for GKE nodes
gcloud compute networks subnets create stateset-sandbox-subnet \
--network=stateset-sandbox-vpc \
--region=$REGION \
--range=10.0.0.0/20 \
--secondary-range=pods=10.4.0.0/14,services=10.8.0.0/20
# Create Cloud NAT for egress
gcloud compute routers create stateset-sandbox-router \
--network=stateset-sandbox-vpc \
--region=$REGION
gcloud compute routers nats create stateset-sandbox-nat \
--router=stateset-sandbox-router \
--region=$REGION \
--nat-all-subnet-ip-ranges \
--auto-allocate-nat-external-ips
Step 3: Create GKE Cluster
Option A: GKE Standard with gVisor
# Create cluster
gcloud container clusters create stateset-sandbox \
--region=$REGION \
--num-nodes=1 \
--machine-type=n2-standard-4 \
--disk-size=50GB \
--network=stateset-sandbox-vpc \
--subnetwork=stateset-sandbox-subnet \
--cluster-secondary-range-name=pods \
--services-secondary-range-name=services \
--enable-ip-alias \
--enable-network-policy \
--workload-pool=${PROJECT_ID}.svc.id.goog \
--enable-shielded-nodes \
--shielded-secure-boot \
--shielded-integrity-monitoring \
--release-channel=regular \
--node-labels=node-type=controller \
--addons=HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver
# Create gVisor-enabled node pool for sandboxes
gcloud container node-pools create sandbox-gvisor \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=c2-standard-8 \
--disk-size=100GB \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=20 \
--enable-autoscaling \
--sandbox=type=gvisor \
--node-labels=node-type=sandbox,isolation=gvisor \
--node-taints=stateset.com/sandbox=true:NoSchedule \
--shielded-secure-boot \
--shielded-integrity-monitoring
# Get credentials
gcloud container clusters get-credentials stateset-sandbox --region=$REGION
Option B: GKE Autopilot
# Create Autopilot cluster (automatically manages nodes)
gcloud container clusters create-auto stateset-sandbox \
--region=$REGION \
--network=stateset-sandbox-vpc \
--subnetwork=stateset-sandbox-subnet \
--cluster-secondary-range-name=pods \
--services-secondary-range-name=services \
--workload-pool=${PROJECT_ID}.svc.id.goog \
--release-channel=regular
# Get credentials
gcloud container clusters get-credentials stateset-sandbox --region=$REGION
Step 4: Create Cloud SQL (PostgreSQL)
# Create Cloud SQL instance
gcloud sql instances create stateset-sandbox \
--database-version=POSTGRES_15 \
--tier=db-custom-4-15360 \
--region=$REGION \
--network=stateset-sandbox-vpc \
--no-assign-ip \
--enable-google-private-path \
--storage-size=100GB \
--storage-type=SSD \
--storage-auto-increase \
--backup-start-time=03:00 \
--availability-type=REGIONAL \
--deletion-protection
# Create database
gcloud sql databases create sandbox --instance=stateset-sandbox
# Create user
gcloud sql users create sandbox-user \
--instance=stateset-sandbox \
--password="YOUR_STRONG_PASSWORD"
# Get connection name for Kubernetes
gcloud sql instances describe stateset-sandbox --format='value(connectionName)'
Step 5: Create Memorystore (Redis)
# Create Redis instance
gcloud redis instances create stateset-sandbox \
--size=2 \
--region=$REGION \
--network=stateset-sandbox-vpc \
--tier=STANDARD_HA \
--redis-version=redis_7_0 \
--transit-encryption-mode=SERVER_AUTHENTICATION
# Get host and port
gcloud redis instances describe stateset-sandbox --region=$REGION \
--format='value(host,port)'
Step 6: Create Cloud Storage Bucket
# Create bucket
gcloud storage buckets create gs://stateset-sandbox-artifacts-${PROJECT_ID} \
--location=$REGION \
--uniform-bucket-level-access \
--public-access-prevention
# Set lifecycle rule
cat <<EOF > lifecycle.json
{
"rule": [{
"action": {"type": "Delete"},
"condition": {"age": 90, "matchesPrefix": ["artifacts/"]}
}]
}
EOF
gcloud storage buckets update gs://stateset-sandbox-artifacts-${PROJECT_ID} \
--lifecycle-file=lifecycle.json
Step 7: Set Up Secret Manager
# Create secrets
echo -n "$(openssl rand -base64 32)" | \
gcloud secrets create jwt-secret --data-file=-
echo -n "$(openssl rand -hex 32)" | \
gcloud secrets create database-encryption-key --data-file=-
echo -n "your-stripe-secret-key" | \
gcloud secrets create stripe-secret-key --data-file=-
# Create service account for secrets access
gcloud iam service-accounts create sandbox-controller \
--display-name="Stateset Sandbox Controller"
# Grant secrets access
gcloud secrets add-iam-policy-binding jwt-secret \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
gcloud secrets add-iam-policy-binding database-encryption-key \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
gcloud secrets add-iam-policy-binding stripe-secret-key \
--member="serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"
# Grant GCS access
gsutil iam ch serviceAccount:sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com:objectAdmin \
gs://stateset-sandbox-artifacts-${PROJECT_ID}
Step 8: Configure Workload Identity
# Allow Kubernetes service account to use GCP service account
gcloud iam service-accounts add-iam-policy-binding \
sandbox-controller@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="serviceAccount:${PROJECT_ID}.svc.id.goog[stateset-sandbox/sandbox-controller]"
Step 9: Deploy StateSet Sandbox
Create Namespace
kubectl apply -f k8s/namespace.yaml
Configure Service Account for Workload Identity
# gcp-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: sandbox-controller
namespace: stateset-sandbox
annotations:
iam.gke.io/gcp-service-account: sandbox-controller@YOUR_PROJECT_ID.iam.gserviceaccount.com
Create ConfigMap
# gcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-controller-config
namespace: stateset-sandbox
data:
# GCP-specific configuration
SANDBOX_IMAGE: "us-central1-docker.pkg.dev/${PROJECT_ID}/stateset/sandbox:latest"
# Database (Cloud SQL via private IP)
DATABASE_URL: "postgres://sandbox-user:PASSWORD@CLOUD_SQL_PRIVATE_IP:5432/sandbox?sslmode=require"
# Redis (Memorystore)
REDIS_HOST: "MEMORYSTORE_HOST"
REDIS_PORT: "6379"
REDIS_TLS: "true"
# Storage (GCS)
STORAGE_PROVIDER: "gcs"
STORAGE_BUCKET: "stateset-sandbox-artifacts-${PROJECT_ID}"
GCS_PROJECT_ID: "${PROJECT_ID}"
# Isolation
DEFAULT_ISOLATION: "gvisor"
SANDBOX_RUNTIME_CLASS: "gvisor"
# Warm pool
WARM_POOL_ENABLED: "true"
WARM_POOL_SIZE: "5"
# Exec agent runtime (optional)
# Set to "go" to use the Go exec agent (execd) instead of the default Node.js agent.
# The Go binary is included in the sandbox base image and has a significantly lower
# memory footprint (~8MB RSS vs ~45MB for Node.js).
# EXEC_AGENT_RUNTIME: "go"
# CORS
CORS_ORIGIN: "https://sandbox.yourdomain.com"
APP_URL: "https://api.sandbox.yourdomain.com"
# Logging (Cloud Logging integration)
LOG_LEVEL: "info"
NODE_ENV: "production"
Create External Secrets
# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace
# gcp-external-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: gcp-secret-manager
namespace: stateset-sandbox
spec:
provider:
gcpsm:
projectID: YOUR_PROJECT_ID
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: sandbox-controller-secrets
namespace: stateset-sandbox
spec:
refreshInterval: 1h
secretStoreRef:
name: gcp-secret-manager
kind: SecretStore
target:
name: sandbox-controller-auth
data:
- secretKey: jwt-secret
remoteRef:
key: jwt-secret
- secretKey: database-encryption-key
remoteRef:
key: database-encryption-key
- secretKey: stripe-secret-key
remoteRef:
key: stripe-secret-key
Apply Custom Resource Definitions
CRDs must be applied before the main deployment so the API server recognises StateSet resource types:# Apply Custom Resource Definitions (before main deployment)
kubectl apply -f k8s/crds/
# Verify CRDs are registered
kubectl get crd | grep stateset
# statesandboxes.sandbox.stateset.io
# warmpools.sandbox.stateset.io
# Optional: Enable CRD-based sandbox management
# Set SANDBOX_BACKEND=crd in your configmap to use declarative sandbox management
Deploy Components
kubectl apply -f gcp-service-account.yaml
kubectl apply -f gcp-configmap.yaml
kubectl apply -f gcp-external-secrets.yaml
kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/network-policy.yaml
kubectl apply -f k8s/resource-quota.yaml
Create gVisor RuntimeClass
# gvisor-runtimeclass.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
scheduling:
nodeSelector:
sandbox.gke.io/runtime: gvisor
tolerations:
- key: stateset.com/sandbox
operator: Exists
effect: NoSchedule
kubectl apply -f gvisor-runtimeclass.yaml
CRD Usage Examples
With the CRDs installed you can manage sandboxes and warm pools declaratively:# Example: Create a sandbox via CRD
apiVersion: sandbox.stateset.io/v1alpha1
kind: StateSandbox
metadata:
name: my-sandbox
namespace: stateset-sandbox
spec:
orgId: "org-123"
image: "stateset/sandbox:latest"
resources:
cpu: "2"
memory: "2Gi"
timeoutSeconds: 600
isolation: container
kubectl apply -f sandbox.yaml
kubectl get ssb # short name
kubectl get warmpools # or kubectl get wp
Step 10: Configure Load Balancer & SSL
Managed SSL Certificate
# Create managed certificate
gcloud compute ssl-certificates create stateset-sandbox-cert \
--domains=api.sandbox.yourdomain.com \
--global
# Reserve static IP
gcloud compute addresses create stateset-sandbox-ip --global
Ingress with GCE Load Balancer
# gcp-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sandbox-controller
namespace: stateset-sandbox
annotations:
kubernetes.io/ingress.global-static-ip-name: "stateset-sandbox-ip"
networking.gke.io/managed-certificates: "sandbox-controller-cert"
kubernetes.io/ingress.class: "gce"
# Enable Cloud Armor (optional)
# networking.gke.io/v1beta1.FrontendConfig: "sandbox-frontend-config"
spec:
rules:
- host: api.sandbox.yourdomain.com
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: sandbox-controller
port:
number: 80
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: sandbox-controller-cert
namespace: stateset-sandbox
spec:
domains:
- api.sandbox.yourdomain.com
kubectl apply -f gcp-ingress.yaml
# Get the external IP
kubectl get ingress sandbox-controller -n stateset-sandbox
# Update DNS to point to this IP
Cloud Armor Security Policy (Optional)
# Create security policy
gcloud compute security-policies create stateset-sandbox-policy \
--description="Stateset Sandbox security policy"
# Add rate limiting rule
gcloud compute security-policies rules create 1000 \
--security-policy=stateset-sandbox-policy \
--action=rate-based-ban \
--rate-limit-threshold-count=100 \
--rate-limit-threshold-interval-sec=60 \
--ban-duration-sec=300 \
--conform-action=allow \
--exceed-action=deny-429 \
--enforce-on-key=IP
# Add geo-blocking (optional)
gcloud compute security-policies rules create 2000 \
--security-policy=stateset-sandbox-policy \
--action=deny-403 \
--expression="origin.region_code == 'CN'"
# Create frontend config
cat <<EOF | kubectl apply -f -
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: sandbox-frontend-config
namespace: stateset-sandbox
spec:
sslPolicy: modern-ssl-policy
redirectToHttps:
enabled: true
EOF
Step 11: Set Up Monitoring
Cloud Monitoring
GKE automatically sends metrics to Cloud Monitoring. Enable additional features:# Enable managed Prometheus
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--enable-managed-prometheus
Custom Dashboards
Create monitoring dashboard in Cloud Console or via API:# Create dashboard from JSON
gcloud monitoring dashboards create \
--config-from-file=monitoring/gcp-dashboard.json
Alerting Policies
# Create alert for high error rate
gcloud alpha monitoring policies create \
--display-name="Stateset Sandbox High Error Rate" \
--condition-display-name="Error rate > 5%" \
--condition-filter='metric.type="custom.googleapis.com/stateset/http_error_rate" resource.type="k8s_container"' \
--condition-threshold-value=0.05 \
--condition-threshold-duration=300s \
--notification-channels=CHANNEL_ID
Cloud Logging
Query logs:gcloud logging read 'resource.type="k8s_container" resource.labels.namespace_name="stateset-sandbox"' \
--limit=100 \
--format=json
Step 12: Configure Autoscaling
Cluster Autoscaler
Already enabled when creating node pool with--enable-autoscaling. Configure behavior:
# Update autoscaling profile
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--autoscaling-profile=optimize-utilization
Horizontal Pod Autoscaler
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: sandbox-controller
namespace: stateset-sandbox
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: sandbox-controller
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Security Hardening
Binary Authorization
# Enable Binary Authorization
gcloud container clusters update stateset-sandbox \
--region=$REGION \
--enable-binauthz
# Create attestor and policy
gcloud container binauthz attestors create sandbox-attestor \
--attestation-authority-note=sandbox-note \
--attestation-authority-note-project=$PROJECT_ID
VPC Service Controls (Enterprise)
# Create service perimeter
gcloud access-context-manager perimeters create stateset-sandbox \
--title="Stateset Sandbox Perimeter" \
--resources="projects/${PROJECT_NUMBER}" \
--restricted-services="storage.googleapis.com,sqladmin.googleapis.com" \
--policy=POLICY_ID
Cloud KMS for Encryption
# Create keyring and key
gcloud kms keyrings create stateset-sandbox --location=$REGION
gcloud kms keys create sandbox-secrets-key \
--location=$REGION \
--keyring=stateset-sandbox \
--purpose=encryption
# Use for database encryption
gcloud sql instances patch stateset-sandbox \
--disk-encryption-key=projects/$PROJECT_ID/locations/$REGION/keyRings/stateset-sandbox/cryptoKeys/sandbox-secrets-key
Confidential Computing (Optional)
For maximum isolation with Confidential VMs:# Create confidential computing node pool
gcloud container node-pools create sandbox-confidential \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=n2d-standard-8 \
--enable-confidential-nodes \
--disk-size=100GB \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=10 \
--enable-autoscaling \
--node-labels=node-type=sandbox,isolation=confidential \
--node-taints=stateset.com/sandbox=true:NoSchedule
Cost Optimization
Committed Use Discounts
Purchase committed use discounts for predictable controller workloads:gcloud compute commitments create stateset-controller \
--region=$REGION \
--resources=vcpu=12,memory=48GB \
--plan=twelve-month
Preemptible VMs for Sandboxes
# Create preemptible node pool
gcloud container node-pools create sandbox-preemptible \
--cluster=stateset-sandbox \
--region=$REGION \
--machine-type=c2-standard-8 \
--preemptible \
--num-nodes=2 \
--min-nodes=0 \
--max-nodes=20 \
--enable-autoscaling \
--sandbox=type=gvisor \
--node-labels=node-type=sandbox,lifecycle=preemptible \
--node-taints=stateset.com/sandbox=true:NoSchedule
Estimated Monthly Costs
| Resource | Configuration | Est. Cost |
|---|---|---|
| GKE Management | 1 cluster | $73 |
| Controller Nodes | 3x n2-standard-4 | ~$290 |
| Sandbox Nodes | 3-10x c2-standard-8 (preemptible) | ~150−500 |
| Cloud SQL | db-custom-4-15360 HA | ~$350 |
| Memorystore Redis | 2GB Standard HA | ~$175 |
| Cloud Load Balancer | 1 LB + traffic | ~$35 |
| Cloud Storage | Storage + requests | ~$20 |
| Egress | ~500GB/month | ~$60 |
| Total | ~1,150−1,500/month |
Troubleshooting
Common Issues
Cloud SQL connection issues# Check connectivity
kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
psql "postgres://user:pass@CLOUD_SQL_IP:5432/sandbox?sslmode=require"
# Check node pool status
gcloud container node-pools describe sandbox-gvisor \
--cluster=stateset-sandbox \
--region=$REGION
# Check RuntimeClass
kubectl get runtimeclass gvisor -o yaml
# Check managed certificate status
kubectl describe managedcertificate sandbox-controller-cert -n stateset-sandbox
Useful Commands
# View cluster events
kubectl get events -n stateset-sandbox --sort-by='.lastTimestamp'
# Stream logs
kubectl logs -f -l app=sandbox-controller -n stateset-sandbox
# Check node conditions
kubectl describe nodes | grep -A5 "Conditions:"
Support Resources
The GCP cluster above runs sandboxed, untrusted code. Do not reuse an existing
GKE cluster that also runs your own services: the isolation story in the
security guide assumes a
dedicated node pool, and a shared one gives a sandboxed process neighbours
worth attacking.
Next steps
Security guide
Read before this cluster takes untrusted code — network policy is what contains it.
Operations
Health probes, metrics and runbooks for the cluster you just built.
Production guide
Capacity, warm pools and the settings that decide cost.
Other providers
The same deployment on the other two clouds.