AWS Deployment Guide (EKS)
Complete guide for deploying StateSet Sandbox on Amazon Elastic Kubernetes Service (EKS).Prerequisites
- AWS CLI configured with appropriate permissions
- kubectl installed
- eksctl installed (recommended)
- Helm 3.x installed
- Domain name for API endpoint
Quick Start (Kustomize Overlay)
Use the AWS-specific overlay to apply the correct ingress/controller defaults:kubectl apply -k k8s/overlays/aws
# or
./scripts/deploy-k8s.sh aws
# or (CLI)
stateset-sandbox deploy aws
stateset-sandbox deploy aws --preflight --wait --smoke
stateset-sandbox is the @stateset/sandbox-cli binary. It also installs a stateset alias, but that
name collides with @stateset/cli, whose stateset has no deploy command, so use the long name.
For smoke checks, export credentials before running deploy:
export STATESET_API_KEY="your_api_key"
export STATESET_API_URL="https://api.sandbox.stateset.app/api/v1"
read -rsp "Admin password: " ADMIN_PASSWORD; echo
bash scripts/create-controller-secret.sh \
--admin-username admin \
--admin-password "$ADMIN_PASSWORD"
unset ADMIN_PASSWORD
Architecture Overview
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AWS Cloud │
│ ┌───────────────────────────────────────────────────────────────────────────┐ │
│ │ VPC (10.0.0.0/16) │ │
│ │ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐ │ │
│ │ │ Public Subnet 1 │ │ Public Subnet 2 │ │ Public Subnet 3 │ │ │
│ │ │ 10.0.1.0/24 │ │ 10.0.2.0/24 │ │ 10.0.3.0/24 │ │ │
│ │ │ ┌──────────────┐ │ │ ┌──────────────┐ │ │ ┌──────────────┐ │ │ │
│ │ │ │ NAT Gateway │ │ │ │ NAT Gateway │ │ │ │ NAT Gateway │ │ │ │
│ │ │ └──────────────┘ │ │ └──────────────┘ │ │ └──────────────┘ │ │ │
│ │ └────────────────────┘ └────────────────────┘ └────────────────────┘ │ │
│ │ │ │
│ │ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐ │ │
│ │ │ Private Subnet 1 │ │ Private Subnet 2 │ │ Private Subnet 3 │ │ │
│ │ │ 10.0.11.0/24 │ │ 10.0.12.0/24 │ │ 10.0.13.0/24 │ │ │
│ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │ │ ┌────────────────┐ │ │ │
│ │ │ │ EKS Node Group │ │ │ │ EKS Node Group │ │ │ │ EKS Node Group │ │ │ │
│ │ │ │ Controller │ │ │ │ Sandboxes │ │ │ │ Sandboxes │ │ │ │
│ │ │ └────────────────┘ │ │ └────────────────┘ │ │ └────────────────┘ │ │ │
│ │ └────────────────────┘ └────────────────────┘ └────────────────────┘ │ │
│ └───────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │ Amazon RDS │ │ ElastiCache │ │ S3 │ │ AWS Secrets Manager │ │
│ │ PostgreSQL │ │ Redis │ │ Artifacts │ │ API Keys / Secrets │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ AWS Load Balancer (ALB) │ │
│ │ api.sandbox.yourdomain.com │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
Step 1: Create EKS Cluster
Option A: Using eksctl (Recommended)
Create a cluster configuration file:# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: stateset-sandbox
region: us-east-1
version: "1.29"
vpc:
cidr: 10.0.0.0/16
nat:
gateway: HighlyAvailable
iam:
withOIDC: true
managedNodeGroups:
# Controller nodes
- name: controller
instanceType: t3.large
desiredCapacity: 3
minSize: 2
maxSize: 5
volumeSize: 50
labels:
node-type: controller
tags:
environment: production
iam:
withAddonPolicies:
imageBuilder: true
autoScaler: true
ebs: true
albIngress: true
cloudWatch: true
# Sandbox nodes (gVisor-enabled)
- name: sandbox-gvisor
instanceType: c5.2xlarge
desiredCapacity: 3
minSize: 1
maxSize: 20
volumeSize: 100
labels:
node-type: sandbox
isolation: gvisor
taints:
- key: stateset.com/sandbox
value: "true"
effect: NoSchedule
preBootstrapCommands:
# Install gVisor
- |
set -o errexit
set -o nounset
set -o pipefail
ARCH=$(uname -m)
URL="https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}"
wget "${URL}/runsc" "${URL}/containerd-shim-runsc-v1" -P /tmp
chmod a+rx /tmp/runsc /tmp/containerd-shim-runsc-v1
mv /tmp/runsc /tmp/containerd-shim-runsc-v1 /usr/local/bin/
cat <<EOF | tee /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
EOF
systemctl restart containerd
addons:
- name: vpc-cni
version: latest
- name: coredns
version: latest
- name: kube-proxy
version: latest
- name: aws-ebs-csi-driver
version: latest
attachPolicyARNs:
- arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
cloudWatch:
clusterLogging:
enableTypes: ["api", "audit", "authenticator", "controllerManager", "scheduler"]
eksctl create cluster -f cluster.yaml
Option B: Using Terraform
See our Terraform module for infrastructure-as-code deployment.Step 2: Install Required Components
AWS Load Balancer Controller
# Create IAM OIDC provider
eksctl utils associate-iam-oidc-provider \
--cluster stateset-sandbox \
--approve
# Create IAM policy
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.0/docs/install/iam_policy.json
aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam_policy.json
# Create service account
eksctl create iamserviceaccount \
--cluster=stateset-sandbox \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--approve
# Install controller
helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=stateset-sandbox \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
External DNS (Optional)
# Create IAM policy
cat <<EOF > external-dns-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["route53:ChangeResourceRecordSets"],
"Resource": ["arn:aws:route53:::hostedzone/*"]
},
{
"Effect": "Allow",
"Action": ["route53:ListHostedZones", "route53:ListResourceRecordSets"],
"Resource": ["*"]
}
]
}
EOF
aws iam create-policy \
--policy-name ExternalDNSPolicy \
--policy-document file://external-dns-policy.json
# Install external-dns
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm install external-dns external-dns/external-dns \
--namespace kube-system \
--set provider=aws \
--set aws.zoneType=public \
--set txtOwnerId=stateset-sandbox \
--set policy=sync
cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
# Create ClusterIssuer for Let's Encrypt
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: your-email@example.com
privateKeySecretRef:
name: letsencrypt-production
solvers:
- http01:
ingress:
class: alb
EOF
gVisor RuntimeClass
kubectl apply -f - <<EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
scheduling:
nodeSelector:
isolation: gvisor
tolerations:
- key: stateset.com/sandbox
operator: Exists
effect: NoSchedule
EOF
Step 3: Create AWS Resources
RDS PostgreSQL
# Create RDS subnet group
aws rds create-db-subnet-group \
--db-subnet-group-name stateset-sandbox \
--db-subnet-group-description "Stateset Sandbox DB subnet group" \
--subnet-ids subnet-xxx subnet-yyy subnet-zzz
# Create RDS instance
aws rds create-db-instance \
--db-instance-identifier stateset-sandbox \
--db-instance-class db.r6g.large \
--engine postgres \
--engine-version 15.4 \
--master-username postgres \
--master-user-password "YOUR_STRONG_PASSWORD" \
--allocated-storage 100 \
--storage-type gp3 \
--storage-encrypted \
--db-subnet-group-name stateset-sandbox \
--vpc-security-group-ids sg-xxx \
--multi-az \
--backup-retention-period 7 \
--deletion-protection
ElastiCache Redis
# Create ElastiCache subnet group
aws elasticache create-cache-subnet-group \
--cache-subnet-group-name stateset-sandbox \
--cache-subnet-group-description "Stateset Sandbox Redis subnet group" \
--subnet-ids subnet-xxx subnet-yyy subnet-zzz
# Create Redis cluster
aws elasticache create-replication-group \
--replication-group-id stateset-sandbox \
--replication-group-description "Stateset Sandbox Redis" \
--engine redis \
--engine-version 7.0 \
--cache-node-type cache.r6g.large \
--num-node-groups 1 \
--replicas-per-node-group 2 \
--cache-subnet-group-name stateset-sandbox \
--security-group-ids sg-xxx \
--at-rest-encryption-enabled \
--transit-encryption-enabled \
--automatic-failover-enabled
S3 Bucket for Artifacts
# Create bucket
aws s3 mb s3://stateset-sandbox-artifacts-YOUR_ACCOUNT_ID --region us-east-1
# Enable versioning
aws s3api put-bucket-versioning \
--bucket stateset-sandbox-artifacts-YOUR_ACCOUNT_ID \
--versioning-configuration Status=Enabled
# Enable encryption
aws s3api put-bucket-encryption \
--bucket stateset-sandbox-artifacts-YOUR_ACCOUNT_ID \
--server-side-encryption-configuration '{
"Rules": [{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "aws:kms"
}
}]
}'
# Create lifecycle rule for artifact expiration
aws s3api put-bucket-lifecycle-configuration \
--bucket stateset-sandbox-artifacts-YOUR_ACCOUNT_ID \
--lifecycle-configuration '{
"Rules": [{
"ID": "ExpireOldArtifacts",
"Status": "Enabled",
"Filter": {"Prefix": "artifacts/"},
"Expiration": {"Days": 90},
"NoncurrentVersionExpiration": {"NoncurrentDays": 30}
}]
}'
Secrets Manager
# Create secrets
aws secretsmanager create-secret \
--name stateset-sandbox/jwt-secret \
--secret-string "$(openssl rand -base64 32)"
aws secretsmanager create-secret \
--name stateset-sandbox/database-encryption-key \
--secret-string "$(openssl rand -hex 32)"
aws secretsmanager create-secret \
--name stateset-sandbox/stripe-secret-key \
--secret-string "your-stripe-secret-key"
# Create IAM policy for secrets access
cat <<EOF > secrets-policy.json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue"],
"Resource": "arn:aws:secretsmanager:us-east-1:*:secret:stateset-sandbox/*"
}]
}
EOF
aws iam create-policy \
--policy-name StateSetSandboxSecretsPolicy \
--policy-document file://secrets-policy.json
Step 4: Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace
# Create SecretStore
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: stateset-sandbox
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: sandbox-controller
EOF
# Create ExternalSecret
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: sandbox-controller-secrets
namespace: stateset-sandbox
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: sandbox-controller-auth
data:
- secretKey: jwt-secret
remoteRef:
key: stateset-sandbox/jwt-secret
- secretKey: database-encryption-key
remoteRef:
key: stateset-sandbox/database-encryption-key
- secretKey: stripe-secret-key
remoteRef:
key: stateset-sandbox/stripe-secret-key
EOF
Step 5: Deploy StateSet Sandbox
Create Namespace
kubectl apply -f k8s/namespace.yaml
Create ConfigMap
# aws-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-controller-config
namespace: stateset-sandbox
data:
# AWS-specific configuration
SANDBOX_IMAGE: "YOUR_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/stateset-sandbox:latest"
# Database (RDS endpoint)
DATABASE_URL: "postgres://postgres:PASSWORD@stateset-sandbox.xxx.us-east-1.rds.amazonaws.com:5432/sandbox?sslmode=require"
# Redis (ElastiCache endpoint)
REDIS_HOST: "stateset-sandbox.xxx.use1.cache.amazonaws.com"
REDIS_PORT: "6379"
REDIS_TLS: "true"
# Storage (S3)
STORAGE_PROVIDER: "s3"
STORAGE_BUCKET: "stateset-sandbox-artifacts-YOUR_ACCOUNT_ID"
STORAGE_REGION: "us-east-1"
# Isolation
DEFAULT_ISOLATION: "gvisor"
SANDBOX_RUNTIME_CLASS: "gvisor"
# Warm pool
WARM_POOL_ENABLED: "true"
WARM_POOL_SIZE: "5"
# CORS
CORS_ORIGIN: "https://sandbox.yourdomain.com"
APP_URL: "https://api.sandbox.yourdomain.com"
# Logging
LOG_LEVEL: "info"
NODE_ENV: "production"
kubectl apply -f aws-configmap.yaml
Deploy Controller
kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/network-policy.yaml
kubectl apply -f k8s/resource-quota.yaml
Create Ingress with ALB
# aws-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sandbox-controller
namespace: stateset-sandbox
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:YOUR_ACCOUNT:certificate/xxx
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/healthcheck-path: /health
alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
alb.ingress.kubernetes.io/healthy-threshold-count: "2"
alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
# WAF integration (optional)
# alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:...
spec:
rules:
- host: api.sandbox.yourdomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sandbox-controller
port:
number: 80
kubectl apply -f aws-ingress.yaml
Step 6: Set Up Monitoring
CloudWatch Container Insights
# Install CloudWatch agent
aws eks create-addon \
--cluster-name stateset-sandbox \
--addon-name amazon-cloudwatch-observability \
--configuration-values '{
"containerLogs": {
"enabled": true
},
"agent": {
"config": {
"logs": {
"metrics_collected": {
"kubernetes": {
"cluster_name": "stateset-sandbox"
}
}
}
}
}
}'
Prometheus (Optional)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
# Apply ServiceMonitor
kubectl apply -f k8s/prometheus/service-monitor.yaml
# Apply alert rules
kubectl apply -f k8s/prometheus/alerts.yaml
Step 7: Configure Autoscaling
Cluster Autoscaler
# Create IAM policy
cat <<EOF > cluster-autoscaler-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": "*"
}
]
}
EOF
aws iam create-policy \
--policy-name ClusterAutoscalerPolicy \
--policy-document file://cluster-autoscaler-policy.json
# Install autoscaler
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
--namespace kube-system \
--set autoDiscovery.clusterName=stateset-sandbox \
--set awsRegion=us-east-1
Horizontal Pod Autoscaler
# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: sandbox-controller
namespace: stateset-sandbox
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: sandbox-controller
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Security Hardening
VPC Security Groups
# Create security group for controller
aws ec2 create-security-group \
--group-name stateset-sandbox-controller \
--description "Stateset Sandbox Controller" \
--vpc-id vpc-xxx
# Allow inbound from ALB
aws ec2 authorize-security-group-ingress \
--group-id sg-xxx \
--protocol tcp \
--port 8080 \
--source-group sg-alb-xxx
# Create security group for sandboxes
aws ec2 create-security-group \
--group-name stateset-sandbox-pods \
--description "Stateset Sandbox Pods" \
--vpc-id vpc-xxx
# Allow inbound from controller only
aws ec2 authorize-security-group-ingress \
--group-id sg-yyy \
--protocol tcp \
--port 0-65535 \
--source-group sg-controller-xxx
KMS Encryption
# Create KMS key for secrets
aws kms create-key \
--description "Stateset Sandbox secrets encryption" \
--tags TagKey=Application,TagValue=stateset-sandbox
# Enable EKS secrets encryption
aws eks associate-encryption-config \
--cluster-name stateset-sandbox \
--encryption-config '[{
"resources": ["secrets"],
"provider": {"keyArn": "arn:aws:kms:us-east-1:xxx:key/xxx"}
}]'
Cost Optimization
Spot Instances for Sandbox Nodes
# Add to cluster.yaml
managedNodeGroups:
- name: sandbox-spot
instanceTypes:
- c5.2xlarge
- c5a.2xlarge
- c5n.2xlarge
spot: true
desiredCapacity: 3
labels:
node-type: sandbox
lifecycle: spot
taints:
- key: stateset.com/sandbox
value: "true"
effect: NoSchedule
Savings Plans
Consider purchasing Compute Savings Plans for the controller nodes (always running) and using Spot for sandbox nodes (variable load).Estimated Monthly Costs
| Resource | Configuration | Est. Cost |
|---|---|---|
| EKS Cluster | 1 cluster | $73 |
| Controller Nodes | 3x t3.large (on-demand) | ~$175 |
| Sandbox Nodes | 3-10x c5.2xlarge (spot) | ~200−800 |
| RDS PostgreSQL | db.r6g.large Multi-AZ | ~$350 |
| ElastiCache Redis | cache.r6g.large 2 replicas | ~$400 |
| ALB | 1 ALB + traffic | ~$50 |
| S3 | Storage + requests | ~$25 |
| Data Transfer | ~500GB/month | ~$45 |
| Total | ~1,300−1,900/month |
Troubleshooting
Common Issues
Pod stuck in Pendingkubectl describe pod <pod-name> -n stateset-sandbox
# Check for resource constraints or node affinity issues
kubectl describe ingress sandbox-controller -n stateset-sandbox
# Check ALB controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller
# Check node has gVisor installed
kubectl get nodes -l isolation=gvisor
kubectl debug node/<node-name> -it --image=busybox -- ls /usr/local/bin/runsc
Support Resources
The AWS cluster above runs sandboxed, untrusted code. Do not reuse an existing
EKS cluster that also runs your own services: the isolation story in the
security guide assumes a
dedicated node pool, and a shared one gives a sandboxed process neighbours
worth attacking.
Next steps
Security guide
Read before this cluster takes untrusted code — network policy is what contains it.
Operations
Health probes, metrics and runbooks for the cluster you just built.
Production guide
Capacity, warm pools and the settings that decide cost.
Other providers
The same deployment on the other two clouds.