> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stateset.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sandbox Operations

> Health checks, metrics, monitoring, and day-to-day operation of the sandbox.

# Operations Guide

This guide covers monitoring, metrics, health checks, alerting, and operational procedures for running StateSet Sandbox in production.

## Health Endpoints

The controller exposes several health endpoints for monitoring and orchestration:

### GET /health

Basic health check returning current status.

```bash theme={null}
curl http://localhost:8080/health
```

Response:

```json theme={null}
{
  "status": "healthy",
  "timestamp": "2024-01-21T12:00:00Z",
  "database": "connected",
  "redis": "connected"
}
```

**Status values:**

* `healthy` - All systems operational
* `connected` / `disconnected` - Component status
* `not_configured` - Optional component not enabled

### GET /ready

Kubernetes readiness probe. Returns 200 when ready to accept traffic, 503 otherwise.

```bash theme={null}
curl http://localhost:8080/ready
```

Response:

```json theme={null}
{
  "status": "ready",
  "checks": {
    "sandbox_manager": true,
    "redis": true
  }
}
```

**Use in Kubernetes:**

```yaml theme={null}
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
```

### GET /health/detailed

Comprehensive health with component details. Useful for debugging.

```bash theme={null}
curl http://localhost:8080/health/detailed
```

Response:

```json theme={null}
{
  "timestamp": "2024-01-21T12:00:00Z",
  "environment": "production",
  "version": "1.0.0",
  "status": "healthy",
  "checks": {
    "database": {
      "status": "healthy"
    },
    "redis": {
      "status": "healthy",
      "details": {
        "required": false,
        "connected": true
      }
    },
    "sandbox_manager": {
      "status": "healthy"
    }
  }
}
```

### GET /metrics

Prometheus metrics endpoint. Requires `INTERNAL_API_KEY` in production.

```bash theme={null}
curl -H "X-Internal-Token: $INTERNAL_API_KEY" http://localhost:8080/metrics
```

## Prometheus Metrics

All metrics use the `stateset_sandbox_` prefix.

### HTTP Metrics

| Metric                          | Type      | Labels               | Description         |
| ------------------------------- | --------- | -------------------- | ------------------- |
| `http_requests_total`           | Counter   | method, path, status | Total HTTP requests |
| `http_request_duration_seconds` | Histogram | method, path, status | Request latency     |

### Sandbox Lifecycle

| Metric                               | Type      | Labels          | Description               |
| ------------------------------------ | --------- | --------------- | ------------------------- |
| `sandbox_creations_total`            | Counter   | status          | Sandbox creation attempts |
| `sandbox_startup_duration_seconds`   | Histogram | phase           | Startup time by phase     |
| `active_sandboxes`                   | Gauge     | -               | Current active sandboxes  |
| `sandbox_executions_total`           | Counter   | status          | Command executions        |
| `sandbox_execution_duration_seconds` | Histogram | -               | Command execution time    |
| `sandbox_terminations_total`         | Counter   | reason, org\_id | Terminations by reason    |

### Warm Pool

| Metric                         | Type    | Labels                          | Description           |
| ------------------------------ | ------- | ------------------------------- | --------------------- |
| `warm_pool_hits_total`         | Counter | result                          | hit, miss, compatible |
| `warm_pool_profile_hits_total` | Counter | result, cpus, memory, isolation | Hits by profile       |

### Kubernetes Operations

| Metric                           | Type      | Labels               | Description         |
| -------------------------------- | --------- | -------------------- | ------------------- |
| `k8s_operations_total`           | Counter   | operation, status    | K8s API calls       |
| `k8s_operation_duration_seconds` | Histogram | operation            | K8s API latency     |
| `pod_scheduling_delay_seconds`   | Histogram | isolation, scheduled | Pod scheduling time |
| `pod_startup_events_total`       | Counter   | event\_type, reason  | Pod events          |

### API Keys & Auth

| Metric                          | Type    | Labels             | Description               |
| ------------------------------- | ------- | ------------------ | ------------------------- |
| `api_key_validations_total`     | Counter | status, cache\_hit | Validation attempts       |
| `api_key_update_failures_total` | Counter | error\_type        | Failed last\_used updates |
| `api_key_usage_total`           | Counter | key\_id, operation | Usage by key              |

### Plan & Billing

| Metric                        | Type    | Labels                           | Description      |
| ----------------------------- | ------- | -------------------------------- | ---------------- |
| `plan_limit_violations_total` | Counter | org\_id, plan\_type, limit\_type | Limit violations |
| `org_resource_utilization`    | Gauge   | org\_id, resource\_type          | Resource usage   |
| `billing_events_total`        | Counter | event\_type, plan\_type          | Billing events   |

### Errors

| Metric                             | Type    | Labels                          | Description            |
| ---------------------------------- | ------- | ------------------------------- | ---------------------- |
| `operation_errors_total`           | Counter | operation, error\_code, org\_id | Operation errors       |
| `background_task_failures_total`   | Counter | task\_name, error\_type         | Background task errors |
| `background_task_executions_total` | Counter | task\_name, status              | Task executions        |

### MicroVM & Advanced (when enabled)

| Metric                                      | Type      | Labels            | Description     |
| ------------------------------------------- | --------- | ----------------- | --------------- |
| `microvm_startup_duration_seconds`          | Histogram | phase, provider   | MicroVM startup |
| `microvm_snapshot_operations_total`         | Counter   | operation, status | Snapshot ops    |
| `microvm_snapshot_restore_duration_seconds` | Histogram | profile           | Restore time    |
| `exec_latency_seconds`                      | Histogram | phase, method     | Exec breakdown  |
| `gpu_allocations_total`                     | Counter   | gpu\_type, status | GPU allocations |
| `numa_scheduling_decisions_total`           | Counter   | decision          | NUMA decisions  |

### eBPF Observability (when enabled)

| Metric                   | Type    | Labels               | Description       |
| ------------------------ | ------- | -------------------- | ----------------- |
| `network_rx_bytes_total` | Counter | sandbox\_id, org\_id | Bytes received    |
| `network_tx_bytes_total` | Counter | sandbox\_id, org\_id | Bytes transmitted |
| `syscall_count_total`    | Counter | sandbox\_id, syscall | Syscall counts    |
| `ebpf_enabled`           | Gauge   | -                    | eBPF status (1/0) |
| `network_collector_type` | Gauge   | collector\_type      | Active collector  |

## Prometheus Configuration

```yaml theme={null}
# prometheus.yml
scrape_configs:
  - job_name: 'sandbox-controller'
    static_configs:
      - targets: ['sandbox-controller:8080']
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials: '<INTERNAL_API_KEY>'
    scrape_interval: 15s
```

## Grafana Dashboards

### Key Panels

**Overview:**

* Active sandboxes gauge
* Sandbox creation rate
* Error rate
* P99 latency

**Performance:**

* Cold start histogram
* Warm pool hit rate
* Exec latency distribution
* K8s API latency

**Resources:**

* CPU utilization by org
* Memory utilization by org
* Network bandwidth
* Storage usage

### Example Queries

```promql theme={null}
# Sandbox creation success rate (5m window)
sum(rate(stateset_sandbox_sandbox_creations_total{status="success"}[5m]))
/
sum(rate(stateset_sandbox_sandbox_creations_total[5m]))

# P99 cold start latency
histogram_quantile(0.99,
  sum(rate(stateset_sandbox_sandbox_startup_duration_seconds_bucket[5m])) by (le)
)

# Warm pool hit rate
sum(rate(stateset_sandbox_warm_pool_hits_total{result="hit"}[5m]))
/
sum(rate(stateset_sandbox_warm_pool_hits_total[5m]))

# Error rate by operation
sum(rate(stateset_sandbox_operation_errors_total[5m])) by (operation)

# Active sandboxes over time
stateset_sandbox_active_sandboxes
```

## Alerting Rules

### Prometheus AlertManager

```yaml theme={null}
# alerts.yml
groups:
  - name: sandbox-controller
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          sum(rate(stateset_sandbox_operation_errors_total[5m]))
          / sum(rate(stateset_sandbox_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Controller down
      - alert: ControllerDown
        expr: up{job="sandbox-controller"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Sandbox controller is down"

      # Database connection issues
      - alert: DatabaseUnhealthy
        expr: |
          stateset_sandbox_background_task_failures_total{task_name="database_health"} > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Database connection issues"

      # Warm pool exhausted
      - alert: WarmPoolExhausted
        expr: |
          rate(stateset_sandbox_warm_pool_hits_total{result="miss"}[5m])
          / rate(stateset_sandbox_warm_pool_hits_total[5m]) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Warm pool miss rate too high"
          description: "Consider increasing WARM_POOL_SIZE"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(stateset_sandbox_http_request_duration_seconds_bucket[5m])) by (le)
          ) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency exceeds 5 seconds"

      # Sandbox creation failures
      - alert: SandboxCreationFailures
        expr: |
          rate(stateset_sandbox_sandbox_creations_total{status="failure"}[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Sandbox creation failures detected"
```

## Logging

### Log Levels

| Level   | Use                                   |
| ------- | ------------------------------------- |
| `fatal` | Unrecoverable errors causing shutdown |
| `error` | Errors that affect functionality      |
| `warn`  | Unexpected but handled conditions     |
| `info`  | Normal operational events             |
| `debug` | Detailed debugging information        |
| `trace` | Very verbose tracing                  |

### Log Format

Logs are JSON-formatted for easy parsing:

```json theme={null}
{
  "level": "info",
  "time": "2024-01-21T12:00:00.000Z",
  "msg": "Sandbox created",
  "sandbox_id": "sb-abc123",
  "org_id": "org-xyz",
  "duration_ms": 1234
}
```

### Log Aggregation

**Kubernetes with Loki:**

```yaml theme={null}
# promtail config
scrape_configs:
  - job_name: sandbox-controller
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: sandbox-controller
        action: keep
    pipeline_stages:
      - json:
          expressions:
            level: level
            msg: msg
            sandbox_id: sandbox_id
```

### Useful Log Queries

```logql theme={null}
# Errors in last hour
{app="sandbox-controller"} |= "error" | json | level="error"

# Specific sandbox logs
{app="sandbox-controller"} | json | sandbox_id="sb-abc123"

# Slow requests
{app="sandbox-controller"} | json | duration_ms > 5000
```

## Operational Runbooks

### High Error Rate

1. **Check metrics** for error patterns:
   ```promql theme={null}
   sum(rate(stateset_sandbox_operation_errors_total[5m])) by (operation, error_code)
   ```

2. **Review logs** for stack traces:
   ```bash theme={null}
   kubectl logs -l app=sandbox-controller --since=10m | grep error
   ```

3. **Check dependencies**:
   * Database: `curl /health | jq .database`
   * Redis: `curl /health | jq .redis`
   * K8s API: `kubectl get pods -n stateset-sandbox`

4. **Common causes**:
   * Database connection pool exhausted
   * K8s API rate limiting
   * Node resource pressure
   * Network issues

### Slow Cold Starts

1. **Identify phase** causing delay:
   ```promql theme={null}
   histogram_quantile(0.99,
     sum(rate(stateset_sandbox_sandbox_startup_duration_seconds_bucket[5m])) by (phase, le)
   )
   ```

2. **Check warm pool**:
   ```promql theme={null}
   rate(stateset_sandbox_warm_pool_hits_total{result="miss"}[5m])
   ```

3. **Check scheduling delays**:
   ```promql theme={null}
   histogram_quantile(0.99,
     sum(rate(stateset_sandbox_pod_scheduling_delay_seconds_bucket[5m])) by (le)
   )
   ```

4. **Remediation**:
   * Increase `WARM_POOL_SIZE`
   * Add more warm pool profiles
   * Check node resources
   * Verify image is pre-pulled

### Database Issues

1. **Check connection**:
   ```bash theme={null}
   curl http://localhost:8080/health/detailed | jq .checks.database
   ```

2. **Check pool stats**:
   ```sql theme={null}
   SELECT state, count(*) FROM pg_stat_activity
   WHERE datname = 'sandbox' GROUP BY state;
   ```

3. **Check for locks**:
   ```sql theme={null}
   SELECT * FROM pg_locks WHERE NOT granted;
   ```

4. **Remediation**:
   * Restart controller to reset pool
   * Check database server health
   * Review connection limits

### Warm Pool Empty

1. **Check pool status**:
   ```promql theme={null}
   stateset_sandbox_warm_pool_hits_total
   ```

2. **Check pod creation**:
   ```bash theme={null}
   kubectl get pods -n stateset-sandbox -l stateset.com/warm-pool=true
   ```

3. **Review events**:
   ```bash theme={null}
   kubectl get events -n stateset-sandbox --sort-by='.lastTimestamp' | head -20
   ```

4. **Remediation**:
   * Check cluster capacity
   * Review warm pool configuration
   * Check for image pull issues
   * Verify RBAC permissions

### Memory Pressure

1. **Check controller memory**:
   ```bash theme={null}
   kubectl top pod -l app=sandbox-controller -n stateset-sandbox
   ```

2. **Check for leaks**:
   ```promql theme={null}
   process_resident_memory_bytes{job="sandbox-controller"}
   ```

3. **Remediation**:
   * Increase memory limits
   * Review connection pool sizes
   * Check for large response payloads
   * Consider horizontal scaling

## Scaling

### Horizontal Scaling

The controller supports horizontal scaling with some considerations:

1. **Database**: Shared state via PostgreSQL
2. **Redis**: Required for distributed warm pool
3. **Leader election**: Not currently implemented (future)

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: controller
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
```

### Vertical Scaling

For single-instance deployments:

| Sandboxes | CPU | Memory |
| --------- | --- | ------ |
| 1-50      | 0.5 | 512Mi  |
| 50-200    | 1   | 1Gi    |
| 200-500   | 2   | 2Gi    |
| 500+      | 4   | 4Gi    |

## Backup & Recovery

### Database Backup

```bash theme={null}
# Backup
pg_dump "$DATABASE_URL" > backup.sql

# Restore
psql "$DATABASE_URL" < backup.sql
```

### Critical Data

Ensure backup of:

* API keys table (encrypted)
* Organizations and users
* Billing/usage records
* Audit logs (if enabled)

### Recovery Procedure

1. Deploy new controller instance
2. Restore database from backup
3. Verify connectivity
4. Run health checks
5. Re-enable traffic

## Security Operations

### Rotating Secrets

**JWT Secret:**

1. Generate new secret
2. Update in Kubernetes secret
3. Restart controller (all JWTs invalidated)

**Database Encryption Key:**

1. Cannot rotate without re-encrypting data
2. Plan data migration if needed

**API Keys:**

```bash theme={null}
# Revoke compromised key
curl -X DELETE -H "Authorization: Bearer $ADMIN_KEY" \
  http://localhost:8080/api/admin/api-keys/<key-id>
```

### Audit Log Review

If `AUDIT_LOGS_ENABLED=true`:

```sql theme={null}
SELECT * FROM audit_logs
WHERE action = 'sandbox.create'
AND created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC;
```

### Incident Response

1. **Contain**: Suspend affected organization
2. **Investigate**: Review audit logs and metrics
3. **Remediate**: Revoke keys, patch vulnerabilities
4. **Recover**: Restore normal operation
5. **Document**: Post-incident review