Operate

Operations

GitHub Edit page

Operations

Read this when you:

  • deploy or validate either coordinator runtime;
  • change coordinator secrets, routes, ingress, or provider credentials;
  • check cost limits or lease cleanup behavior;
  • need to decide whether a failure lives in the local CLI, the broker, a provider, or runner state.

Crabbox operations span three layers:

local CLI -> coordinator (Cloudflare or Node/PostgreSQL) -> provider VM

The CLI owns local config, per-lease SSH keys, sync, and remote command execution. The coordinator owns auth, lease state, provider credentials, cost guardrails, and cleanup. Providers own VM creation, network reachability, and deletion. For the full request flow see Architecture and How It Works.

#Daily Health Check

Run these before a release or after changing secrets:

go test ./...
npm run check --prefix worker
npm test --prefix worker
node --test scripts/*.test.js
scripts/check-docs.sh
bin/crabbox doctor
bin/crabbox whoami
bin/crabbox list --json
bin/crabbox usage --scope all --json
bin/crabbox history --limit 5
  • crabbox doctor checks local prerequisites and coordinator/provider readiness.
  • crabbox whoami verifies broker identity.
  • crabbox list confirms the broker can answer lease state.
  • crabbox usage proves the cost-accounting path is reachable.
  • crabbox history proves recorded-run history is reachable.

When broker/provider credentials are available and infra changed, run the live smoke:

CRABBOX_LIVE=1 CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh

scripts/live-smoke.sh defaults to aws,hetzner. Narrow the matrix with CRABBOX_LIVE_PROVIDERS:

CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=aws               CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=hetzner           CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=azure CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_AZURE_TYPE=Standard_D2s_v5 CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=blacksmith-testbox CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=e2b               CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=modal             CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=coder CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_CODER_TEMPLATE=go-dev CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=daytona           CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=namespace-devbox  CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=namespace-instance CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=semaphore         CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=sprites           CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=tenki CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=wandb CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_REPO=/path/to/my-app scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=kubevirt CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_KUBEVIRT_TEMPLATE=/path/to/vm.yaml scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=external CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_EXTERNAL_COMMAND=/path/to/provider scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=morph CRABBOX_LIVE_COORDINATOR=0 CRABBOX_LIVE_MORPH_SNAPSHOT=snapshot_xxx scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=scaleway CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=docker-sandbox CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=smolvm CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=superserve CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=vercel-sandbox CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=aws-lambda-microvm CRABBOX_AWS_LAMBDA_MICROVM_IMAGE=arn:aws:lambda:eu-west-1:123456789012:microvm-image:crabbox-runner scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=apple-container CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=local-container CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=multipass CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=tart CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=apple-vm CRABBOX_LIVE_COORDINATOR=0 CRABBOX_BIN=./bin/crabbox scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=linode scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=digitalocean scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=nebius scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=ovh scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=nvidia-brev scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=phala CRABBOX_LIVE_COORDINATOR=0 CRABBOX_BIN=./bin/crabbox scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=anthropic-sandbox-runtime scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=opensandbox CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=proxmox CRABBOX_LIVE_COORDINATOR=0 CRABBOX_BIN=./bin/crabbox scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=xcp-ng CRABBOX_LIVE_COORDINATOR=0 CRABBOX_BIN=./bin/crabbox scripts/live-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=agent-sandbox CRABBOX_LIVE_COORDINATOR=0 scripts/live-smoke.sh

Per-provider smoke prerequisites:

  • Blacksmith — a workflow containing a useblacksmith/testbox, useblacksmith/begin-testbox, or useblacksmith/run-testbox step; set CRABBOX_BLACKSMITH_WORKFLOW when the default path is wrong. scripts/live-smoke.sh refuses to call Blacksmith until it can derive an org and validate the selected Testbox workflow, then lists inventory and runs one delegated command through the configured workflow/job/ref.
  • E2B — CRABBOX_E2B_API_KEY or E2B_API_KEY. scripts/live-smoke.sh refuses to call E2B until an API key is exported, then creates one sandbox, runs one no-sync command, lists normalized inventory, and stops the lease.
  • Modal — an authenticated Modal Python client (python3 -m modal setup or Modal token env vars). scripts/live-smoke.sh refuses to call Modal until the configured Python binary can import the Modal client, then creates one sandbox, waits for status, runs one no-sync command, lists normalized inventory, and stops the lease.
  • Coder — authenticated coder CLI on PATH plus an explicit disposable template from CRABBOX_LIVE_CODER_TEMPLATE, CRABBOX_CODER_TEMPLATE, or coder.template. scripts/live-smoke.sh refuses to mutate Coder until a template is selected, then proves doctor, dry-run cleanup, stop-by-default warmup/run/stop/status, delete-on-release warmup/run/stop, list, and final dry-run cleanup.
  • Sealos DevBox — kubectl, an inherited kubeconfig or readable configured kubeconfig, an explicit sealosDevbox.context, namespace RBAC for the DevBox CRD, sealosDevbox.image, and a configured SSHGate or NodePort route. scripts/live-smoke.sh refuses to mutate Sealos resources until those prerequisites and doctor --json pass, then proves dry-run cleanup, one retained DevBox warmup, status, SSH command rendering, a synced command, stop, post-stop status, and final dry-run cleanup.
  • Semaphore — CRABBOX_SEMAPHORE_HOST, CRABBOX_SEMAPHORE_PROJECT, and CRABBOX_SEMAPHORE_TOKEN, or the equivalent user config. scripts/live-smoke.sh refuses to call Semaphore until those values are configured, then creates one testbox, runs one no-sync command, lists normalized inventory, and stops the lease.
  • Daytona — CRABBOX_DAYTONA_SNAPSHOT, DAYTONA_SNAPSHOT, or daytona.snapshot. scripts/live-smoke.sh refuses to call Daytona until a snapshot is configured, then runs one delegated command and normalized list proof.
  • Namespace — the authenticated devbox CLI on PATH. scripts/live-smoke.sh refuses to call Namespace until devbox is available, then creates a delete-on-release Devbox, runs one no-sync command, and prints a normalized list proof.
  • Namespace Compute — the authenticated nsc CLI on PATH; run nsc login first. scripts/live-smoke.sh runs doctor, snapshots existing Crabbox-owned inventory, creates one short-lived Compute Instance, runs the normal SSH lease lifecycle, stops the lease, and fails if the post-smoke inventory changed.
  • Sprites — the authenticated sprite CLI on PATH plus a Sprites token in the environment. scripts/live-smoke.sh refuses to call Sprites until the CLI is available, then creates one sprite, verifies SSH, runs one command, lists normalized inventory, and stops the lease.
  • Tenki — the authenticated tenki CLI on PATH; run tenki login and complete the browser flow. scripts/live-smoke.sh refuses to call Crabbox Tenki lifecycle commands until tenki status --json reports a logged-in CLI, then creates one session, runs one no-sync command, verifies paused-session status waits do not resume it, and stops the lease.
  • KubeVirt — kubectl, virtctl, a namespace with KubeVirt access, and an SSH-ready VM template.
  • Agent Sandbox — kubectl, an absolute kubeconfig or inherited KUBECONFIG, an explicit context, a namespace, and a configured SandboxWarmPool. scripts/live-agent-sandbox-smoke.sh is coordinator-free, creates a short-lived SandboxClaim, verifies archive sync, env forwarding, retained-claim reuse, replacement sync, status/list, and claim deletion.
  • External — a configured provider executable through external.command or CRABBOX_LIVE_EXTERNAL_COMMAND, or a declarative external.lifecycle.acquire configuration. scripts/live-smoke.sh refuses to call External lifecycle commands until one path is configured, then runs the normal SSH lease lifecycle, lists normalized inventory, and stops the lease.
  • Morph — CRABBOX_MORPH_API_KEY, MORPH_API_KEY, or morph.apiKey, plus CRABBOX_LIVE_MORPH_SNAPSHOT. scripts/live-smoke.sh refuses to call Morph until both are configured, then creates one delete-on-release instance, runs the normal SSH lease lifecycle, lists normalized inventory, and stops the lease.
  • Scaleway — SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_ORGANIZATION_ID, SCW_DEFAULT_PROJECT_ID, SCW_DEFAULT_REGION, and SCW_DEFAULT_ZONE, or equivalent CRABBOX_SCALEWAY_* overrides for project/location fields. scripts/live-scaleway-smoke.sh is coordinator-free, requires an empty Crabbox-owned inventory before provisioning, creates one short-lived DEV1-S instance by default, verifies status/run/list, stops the lease, and proves the inventory is empty afterward.
  • Docker Sandbox — the standalone sbx CLI on PATH or configured with CRABBOX_DOCKER_SANDBOX_CLI; run sbx login first when your account requires authentication.
  • SmolVM — CRABBOX_SMOLVM_API_KEY, SMOLMACHINES_API_KEY, or SMK_API_KEY.
  • Superserve — CRABBOX_SUPERSERVE_API_KEY or SUPERSERVE_API_KEY.
  • Vercel Sandbox — authenticated sandbox CLI on PATH; project and team scope may come from CRABBOX_VERCEL_SANDBOX_PROJECT_ID plus CRABBOX_VERCEL_SANDBOX_TEAM_ID, CRABBOX_VERCEL_SANDBOX_SCOPE, or the Vercel OIDC environment.
  • AWS Lambda MicroVM — standard AWS SDK credentials, an explicitly exported CRABBOX_AWS_LAMBDA_MICROVM_IMAGE, a launch Region, and quota. scripts/live-aws-lambda-microvm-smoke.sh is coordinator-free; it proves archive sync, retained reuse, pause/resume, termination, and empty local inventory after cleanup.
  • Apple Container — Apple silicon macOS with Apple's container CLI on PATH and container system start already run. scripts/live-smoke.sh uses the normal SSH lease lifecycle with a short TTL and no coordinator.
  • Local Container — a working Docker-compatible CLI and daemon such as Docker Desktop, OrbStack, or Colima. scripts/live-smoke.sh uses the normal SSH lease lifecycle with a short TTL and no coordinator.
  • Multipass — Canonical Multipass installed locally and able to launch Ubuntu VMs. scripts/live-smoke.sh uses the normal SSH lease lifecycle with a short TTL and no coordinator.
  • Tart — Apple silicon macOS with the tart CLI installed, a reachable base image, and guest login credentials configured when the selected image needs a password. scripts/live-smoke.sh uses the normal SSH lease lifecycle with a longer TTL and no coordinator.
  • Apple VZ — Apple silicon macOS, a locally built Crabbox binary (CRABBOX_BIN), and the bundled or explicit crabbox-apple-vm-helper. scripts/live-smoke.sh uses the normal SSH lease lifecycle and preserves CRABBOX_LIVE_APPLE_VM_HELPER for the whole run when set.
  • W&B — WANDB_ENTITY_NAME plus CRABBOX_WANDB_API_KEY or WANDB_API_KEY (from wandb login). scripts/live-smoke.sh refuses to call W&B until an API key is exported, then runs doctor, executes one no-sync command, and prints normalized inventory.
  • Incus — local jq and rg, plus Crabbox config or env resolving incus.socket, incus.address, or incus.remote. scripts/live-smoke.sh refuses to call Incus until local preflight tools are available, then proves delete-on-release and retained-reuse SSH lease lifecycles.
  • Linode — LINODE_TOKEN with Linode instance, image, type, SSH key, and tag access.
  • DigitalOcean — DIGITALOCEAN_TOKEN with account-read, Droplet, image-read, SSH key, and tag scopes. scripts/live-digitalocean-smoke.sh is coordinator-free, requires an empty Crabbox-owned inventory, creates a small short-lived Droplet, verifies status and execution, and prints a final cleanup classification.
  • Nebius — authenticated Nebius CLI profile plus nebius.parentId and nebius.subnetId. scripts/live-nebius-smoke.sh is coordinator-free, requires explicit CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=nebius, creates one short-lived CPU-default VM, verifies status and echo ok, stops the lease, runs dry-run cleanup, and prints a final classification.
  • OVHcloud — OVH application credentials plus ovh.projectId and ovh.region. scripts/live-ovh-smoke.sh is coordinator-free, requires an empty Crabbox-owned OVH inventory, creates one short-lived b3-8 instance by default, verifies status and echo ok, stops the lease, runs dry-run cleanup, and prints a final classification.
  • NVIDIA Brev — authenticated brev CLI on PATH plus enough GPU quota and capacity for the selected workspace type. scripts/live-nvidia-brev-smoke.sh is coordinator-free, creates one short-lived GPU workspace, verifies nvidia-smi, deletes the workspace with crabbox stop, and prints a final classification.
  • Phala — authenticated phala CLI on PATH (or CRABBOX_PHALA_CLI), Phala auth, and enough confidential CVM quota. scripts/live-phala-smoke.sh is coordinator-free, classifies missing auth/quota before or during provisioning, verifies sync/env forwarding on a tiny Git fixture, and hard-deletes any created CVM on failure.
  • Anthropic Sandbox Runtime — srt and curl on PATH plus host sandbox support. scripts/live-anthropic-sandbox-runtime-smoke.sh is coordinator-free and local-only; it verifies doctor, one-shot command execution, allowed temp-file access, denied secret reads, and denied network access.
  • OpenSandbox — CRABBOX_OPENSANDBOX_API_KEY or OPEN_SANDBOX_API_KEY, plus CRABBOX_OPENSANDBOX_API_URL or OPEN_SANDBOX_API_URL. scripts/live-opensandbox-smoke.sh is coordinator-free, proves archive sync, off-argv environment forwarding, retained sandbox reuse, staged sync.delete replacement, list/status, and cleanup.
  • Proxmox — a locally built Crabbox binary (CRABBOX_BIN), Proxmox API credentials/config, and jq/perl. scripts/proxmox-live-smoke.sh is coordinator-free and read-only by default; set CRABBOX_PROXMOX_LIVE_SMOKE=1 only after doctor is green to run the guarded warmup/status/ssh/stop proof.
  • XCP-ng — a locally built Crabbox binary (CRABBOX_BIN), XCP-ng API credentials/config, and python3. scripts/xcpng-live-smoke.sh is coordinator-free and read-only by default; pass --mutate through the standalone script and set CRABBOX_XCP_NG_LIVE_MUTATE=1 only after doctor is green.

For a direct-provider smoke (no coordinator), disable the broker with a scratch config and run the same lease lifecycle manually:

tmp="$(mktemp)"
printf 'provider: hetzner\n' > "$tmp"
CRABBOX_CONFIG="$tmp" CRABBOX_COORDINATOR= bin/crabbox warmup --provider hetzner --class standard --ttl 15m --idle-timeout 4m
CRABBOX_CONFIG="$tmp" CRABBOX_COORDINATOR= bin/crabbox run --provider hetzner --id <slug> --no-sync -- echo direct-hetzner-ok
CRABBOX_CONFIG="$tmp" CRABBOX_COORDINATOR= bin/crabbox stop --provider hetzner <slug>
rm -f "$tmp"

Use --provider aws with AWS SDK credentials for the direct AWS equivalent. Use scripts/live-smoke.sh or scripts/live-digitalocean-smoke.sh for the repeatable direct DigitalOcean equivalent; it builds or reuses bin/crabbox, creates a guarded digitalocean scratch config, and verifies the Crabbox-owned inventory is empty before create and after stop/cleanup. Use scripts/live-smoke.sh or scripts/live-nebius-smoke.sh for the repeatable direct Nebius equivalent; it builds or reuses bin/crabbox, uses the documented Nebius config and CLI profile, creates a unique nebius-smoke-* lease, and verifies the slug is absent after stop and dry-run cleanup. Use scripts/live-smoke.sh or scripts/live-ovh-smoke.sh for the repeatable direct OVHcloud equivalent; it builds or reuses bin/crabbox, uses the documented OVH credentials and project settings, creates a unique ovh-smoke-* lease, and verifies the Crabbox-owned inventory is empty after stop and dry-run cleanup.

#Deployment

Choose one runtime for a coordinator installation:

RuntimeDurable state and schedulingDeployment shape
CloudflareFleet Durable Object, alarms, scheduled Worker triggerWrangler-managed edge service
Node.jsPostgreSQL 13+ plus pg-bossInitial single-replica container or process behind TLS/WebSocket ingress

Both expose the same API and portal. They do not automatically copy state between Durable Object storage and PostgreSQL. Cloudflare is the established deployment; complete the Node deployment-proof checklist in Portable Coordinator Runtime before production cutover.

#Cloudflare Worker

Worker source lives in worker/. Run the gate, then deploy:

npm ci --prefix worker
npm run format:check --prefix worker
npm run lint --prefix worker
npm run check --prefix worker
npm test --prefix worker
npm run build --prefix worker
npx wrangler deploy --config worker/wrangler.jsonc

The repeatable deploy proof is:

scripts/deploy-worker-smoke.sh

It runs Worker format, lint, typecheck, tests, dry-run build, deploy, and public health checks for the comma-separated deployment URLs in CRABBOX_DEPLOY_SMOKE_URLS. To include a short AWS lease smoke after deploy:

CRABBOX_DEPLOY_SMOKE_URLS="https://$BROKER_HOST/v1/health" \
  CRABBOX_DEPLOY_SMOKE_AWS=1 \
  CRABBOX_LIVE_REPO=/path/to/my-app \
  scripts/deploy-worker-smoke.sh

#Node.js And PostgreSQL

Requirements: Node.js 22.12+, PostgreSQL 13+, one always-on service replica, and an ingress that preserves WebSocket upgrades. Use TLS with hostname and CA verification for a remote database. Build and run directly:

npm ci --prefix worker
npm run format:check --prefix worker
npm run lint --prefix worker
npm run check:node --prefix worker
npm test --prefix worker
npm run build:node --prefix worker

DATABASE_URL='postgresql://crabbox:password@db.example.com/crabbox?sslmode=verify-full&sslrootcert=/run/secrets/postgres-ca.pem' \
CRABBOX_PUBLIC_URL=https://broker.example.com \
npm run start:node --prefix worker

Or build the OCI image with worker/ as context:

docker build -f worker/Dockerfile.node -t crabbox-coordinator:local worker
docker run --rm -p 8080:8080 \
  --env-file /secure/path/crabbox.env \
  --mount type=bind,src=/secure/path/postgres-ca.pem,dst=/run/secrets/postgres-ca.pem,readonly \
  crabbox-coordinator:local

The checked-in runtime Dockerfiles keep readable base-image tags but pin them to multi-platform manifest digests. When refreshing a base image, resolve the current manifest-list digest, update the tag and digest together, build the affected image, and run the repository check:

docker buildx imagetools inspect <image>:<tag> --format '{{.Manifest.Digest}}'
node scripts/check-docker-base-images.mjs

The service creates PostgreSQL schemas crabbox and crabbox_jobs. Use GET /v1/health for liveness and GET /v1/ready for database readiness. SIGTERM and SIGINT stop new requests, drain active HTTP/WebSocket and provisioning work, then close PostgreSQL; CRABBOX_SHUTDOWN_TIMEOUT_MS defaults to 120000.

For a VM, run the same image or Node process under the host service manager. For Kubernetes or another scheduler, use one replica, a Recreate deployment strategy, readiness on /v1/ready, and a termination grace period longer than the shutdown timeout. PostgreSQL state and pg-boss jobs are durable, but lifecycle serialization and live bridge ownership remain process-local. Do not horizontally scale yet.

#Dedicated AWS private-workspace service

Use the checked-in ECS Fargate deployment when one Node/PostgreSQL coordinator must own small, private, SSM-only Linux workspaces in one AWS account and Region. The stack owns its single-replica ECS control plane, task roles, workspace instance role/profile, retained SSM log group, internal HTTPS load balancer, and separate security groups. It injects the database URL and route-scoped workspace bearer from Secrets Manager and uses refreshable task-role credentials rather than static AWS keys.

The runbook in Private AWS Workspaces covers the exact account/Region preflight, small instance allowlist, 20 GiB encrypted gp3 disk, private subnet, no public IP/SSH, IMDSv2, SSM bootstrap, CloudWatch evidence, client URL/bearer contract, idempotent cleanup, rollback, and retirement. Deployment and the paid create/delete canary require a separate AWS GO; code merge alone is not approval.

#Minimum coordinator configuration

Configure CRABBOX_PUBLIC_URL, one auth model, and at least one brokered provider. Shared-token automation needs CRABBOX_SHARED_TOKEN and CRABBOX_SHARED_OWNER; browser login needs the GitHub OAuth settings below. Provider choices are HETZNER_TOKEN, AWS credentials from the default chain or an existing static credential contract, an Azure service principal, a GCP service account, or DAYTONA_CRABBOX_KEY. Node additionally requires DATABASE_URL.

GitHub OAuth start routes remain unauthenticated so a new user can bootstrap login. The coordinator limits active attempts to ten per caller source and 100 globally for both CLI and portal login, after removing expired attempts. Node deployments behind a reverse proxy must configure CRABBOX_TRUSTED_PROXY_CIDRS; otherwise caller limits use the direct peer address and ignore forwarded addresses.

For any portal that exposes browser Code, configure CRABBOX_CODE_ORIGIN_TEMPLATE=https://{lease}.code.example.com and route the matching wildcard hostname to the same coordinator with TLS and WebSocket support. This preserves the normal Code links while moving each lease's proxied HTML and JavaScript to a separate browser origin.

#Conditional coordinator secrets and settings

AWS_SESSION_TOKEN                  optional
CRABBOX_HOST_ID                    optional; admin-only except owner reactivation of a retained Mac instance
CRABBOX_AWS_MAC_HOST_ID            optional legacy AWS alias for CRABBOX_HOST_ID
CRABBOX_SHARED_OWNER              optional fixed owner identity for shared-token automation
CRABBOX_ADMIN_TOKEN               required for admin routes and image promotion
CRABBOX_WORKSPACE_SSH_PUBLIC_KEY  required for SSH-based /v1/workspaces provisioning; unused by private AWS mode
CRABBOX_WORKSPACE_SSH_PRIVATE_KEY required for SSH-based terminal attachment; unused by private AWS mode
CRABBOX_WORKSPACE_PROVIDER        optional workspace provider; hetzner, aws, azure, or gcp
CRABBOX_WORKSPACE_CLASS           optional workspace machine class; default standard
CRABBOX_WORKSPACE_PREWARM_COUNT   optional ready spares per active organization; default 0, maximum 4
CRABBOX_RUNTIME_ADAPTER_TOKEN     route-scoped workspace API credential
CRABBOX_RUNTIME_ADAPTER_OWNER     stable service owner for route-scoped access
CRABBOX_RUNTIME_ADAPTER_ORG       stable organization for route-scoped access
DAYTONA_CRABBOX_KEY               required for brokered Daytona leases
CRABBOX_DAYTONA_*                 optional Daytona API, snapshot, target, user, work-root, and SSH-token settings
CRABBOX_RUN_RETENTION_DAYS        terminal run history retention; default 30 days, minimum 1
CRABBOX_GITHUB_CLIENT_ID          required for browser login
CRABBOX_GITHUB_CLIENT_SECRET      required for browser login
CRABBOX_SESSION_SECRET            required for browser login; must differ from CRABBOX_SHARED_TOKEN
CRABBOX_CODE_ORIGIN_TEMPLATE      required for browser Code; per-lease HTTPS origin template
CRABBOX_GITHUB_ALLOWED_ORG or CRABBOX_GITHUB_ALLOWED_ORGS
CRABBOX_GITHUB_ALLOWED_TEAMS      optional
CRABBOX_GITHUB_REVOKED_USERS      optional narrow login/email revocation list
CRABBOX_GITHUB_MEMBERSHIP_CACHE_SECONDS optional; default 300, range 0-3600
CRABBOX_ACCESS_TEAM_DOMAIN        required for Access JWT verification
CRABBOX_ACCESS_AUD                required for Access JWT verification
CRABBOX_TAILSCALE_CLIENT_ID       required for brokered --tailscale
CRABBOX_TAILSCALE_CLIENT_SECRET   required for brokered --tailscale
CRABBOX_TAILSCALE_TAILNET         optional
CRABBOX_TAILSCALE_TAGS            optional
CRABBOX_TAILSCALE_ENABLED         optional; set 0 to disable brokered Tailscale
CRABBOX_TAILSCALE_INSTALL_MODE    optional; package (default) or pinned static archive
CRABBOX_TAILSCALE_VERSION         optional pinned static build version
CRABBOX_TAILSCALE_SHA256_AMD64    optional pinned amd64 archive checksum
CRABBOX_TAILSCALE_SHA256_ARM64    optional pinned arm64 archive checksum
CRABBOX_ARTIFACTS_BACKEND         optional; enables brokered artifact publishing
CRABBOX_ARTIFACTS_BUCKET          required when artifact backend is enabled
CRABBOX_ARTIFACTS_PREFIX          optional
CRABBOX_ARTIFACTS_BASE_URL        optional; public URL prefix used only with public reads
CRABBOX_ARTIFACTS_PUBLIC_READS    optional; set 1 to intentionally return public non-expiring links
CRABBOX_ARTIFACTS_REGION          optional
CRABBOX_ARTIFACTS_ENDPOINT_URL    optional; required for R2/custom S3 endpoints
CRABBOX_ARTIFACTS_ACCESS_KEY_ID   required when artifact backend is enabled
CRABBOX_ARTIFACTS_SECRET_ACCESS_KEY required when artifact backend is enabled
CRABBOX_ARTIFACTS_SESSION_TOKEN   optional
CRABBOX_ARTIFACTS_UPLOAD_EXPIRES_SECONDS optional
CRABBOX_ARTIFACTS_URL_EXPIRES_SECONDS    optional
CRABBOX_AWS_ORPHAN_SWEEP_ENABLED  optional; defaults on when AWS broker credentials exist
CRABBOX_AWS_ORPHAN_SWEEP_DELETE   optional; set 1 to terminate coordinator-owned orphan EC2 instances
CRABBOX_AWS_ORPHAN_SWEEP_INTERVAL_SECONDS optional; default 3600
CRABBOX_AWS_ORPHAN_SWEEP_GRACE_SECONDS    optional; default 900
CRABBOX_AWS_MAC_HOST_SWEEP_RELEASE optional; set 1 to release stale pending EC2 Mac hosts during orphan sweep

Normal SSH-based AWS workspace bridges use a dedicated crabbox-workspaces security group, separate from ordinary runner ingress. Workers TCP egress has no published allowlist, so that group accepts key-only SSH from 0.0.0.0/0; workspace keys are deployment specific, host keys are pinned, and leases expire automatically. The dedicated private AWS mode does not use this group or any SSH key: it requires a separate no-ingress group and SSM-only bootstrap.

Workspace leases currently use their hard TTL for provider expiry because the adapter does not yet receive a trustworthy activity signal. Workspace TTLs must be at least 1,800 seconds so a durable claim and ambiguity-recovery window both fit before hard TTL.

The workspace SSH public and private keys must be a matching dedicated key pair. The coordinator installs the public key on provisioned workspaces and uses the private key only for authenticated terminal attachment. Each workspace also receives a coordinator-generated SSH host identity whose fingerprint is persisted before provisioning, so first attachment does not rely on TOFU. The versioned workspace attachUrl is a bearer-authenticated server-to-server endpoint for control planes such as Crabfleet, not a browser portal URL.

Private AWS workspaces intentionally omit terminal attachment. Their client contract is the dedicated service URL, CRABBOX_RUNTIME_ADAPTER_TOKEN, and POST/GET/DELETE /v1/workspaces; a ready status means SSM registration and bootstrap succeeded. Client labels and Region-shaped metadata do not choose placement.

Desktop workspaces report capabilities.nativeVnc=true when the native CLI handoff is available. This does not imply a browser desktop endpoint: capabilities.vnc and capabilities.desktop remain false unless POST /v1/workspaces/:id/connections/desktop is supported. POST /v1/workspaces/:id/connections/native-vnc mints a one-minute, single-use grant. The native CLI passes that grant on stdin, and the coordinator uses its dedicated workspace SSH key to relay the loopback VNC service over an authenticated WebSocket. The workspace SSH private key never leaves the coordinator.

When CRABBOX_WORKSPACE_PREWARM_COUNT is positive, the coordinator keeps that many hidden ready workspaces for each organization with active workspace demand. Any owner in the organization can atomically adopt a matching spare. The coordinator replenishes adopted spares and drains them after the organization has no provisioning or ready workspaces.

#Artifact backend

The artifact backend vars are ordinary coordinator settings except CRABBOX_ARTIFACTS_ACCESS_KEY_ID, CRABBOX_ARTIFACTS_SECRET_ACCESS_KEY, and optional CRABBOX_ARTIFACTS_SESSION_TOKEN, which must use the runtime's secret injection. These object-store keys let the coordinator sign short-lived artifact upload/read URLs. Scope them to the artifact bucket or prefix; they should not carry Cloudflare account, Worker deployment, lease-provider, or VM permissions.

A typical R2-compatible configuration looks like:

CRABBOX_ARTIFACTS_BACKEND=r2
CRABBOX_ARTIFACTS_BUCKET=my-crabbox-artifacts
CRABBOX_ARTIFACTS_PREFIX=crabbox-artifacts
CRABBOX_ARTIFACTS_BASE_URL=https://artifacts.example.com
CRABBOX_ARTIFACTS_PUBLIC_READS=1
CRABBOX_ARTIFACTS_REGION=auto
CRABBOX_ARTIFACTS_ENDPOINT_URL=<account>.r2.cloudflarestorage.com

Omit CRABBOX_ARTIFACTS_PUBLIC_READS to return expiring signed read URLs even when a base URL is configured. Enable it only when the artifact origin is intentionally public; each public grant receives a random capability namespace.

Deploy the matching access key id and secret access key as coordinator secrets, not local CLI defaults. End users run crabbox artifacts publish without holding any S3/R2 credentials.

#Cost-control secrets and settings

CRABBOX_COST_RATES_JSON
CRABBOX_EUR_TO_USD
CRABBOX_MAX_ACTIVE_LEASES
CRABBOX_MAX_ACTIVE_LEASES_PER_OWNER
CRABBOX_MAX_ACTIVE_LEASES_PER_ORG
CRABBOX_MAX_MONTHLY_USD
CRABBOX_MAX_MONTHLY_USD_PER_OWNER
CRABBOX_MAX_MONTHLY_USD_PER_ORG
CRABBOX_DEFAULT_ORG

Monthly cost checks use reserved cost, not only elapsed runtime. Long TTLs, prewarmed leases, and failed provisioning attempts can therefore consume budget headroom faster than the provider bill. Keep active-lease and per-owner limits as the primary safety rails, and size fleet/org monthly caps with enough room for TTL-based reservations during busy test bursts.

#Routes And Access

A deployment exposes one canonical route:

https://broker.example.com          # CLI, API, portal, browser login, WebSockets

Cloudflare deployments can expose the same Worker at https://broker-access.example.com behind Cloudflare Access. Node deployments can use a conventional TLS/WebSocket ingress and optionally trust an authenticated user header only from CRABBOX_TRUSTED_PROXY_CIDRS. Bearer-token CLI automation authenticates with CRABBOX_SHARED_TOKEN / CRABBOX_COORDINATOR_TOKEN; GitHub browser login stores a user-scoped signed token (prefix cbxu_). See Auth and Admin and Broker Auth and Routing.

Test the Cloudflare Access layer through the protected route:

CRABBOX_COORDINATOR=https://broker-access.example.com bin/crabbox doctor
CRABBOX_COORDINATOR=https://broker-access.example.com bin/crabbox whoami
CRABBOX_LIVE=1 CRABBOX_AUTH_SMOKE_ACCESS=1 \
  CRABBOX_COORDINATOR=https://broker-access.example.com \
  CRABBOX_BIN=bin/crabbox scripts/live-auth-smoke.sh
CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=aws \
  CRABBOX_COORDINATOR=https://broker-access.example.com \
  CRABBOX_BIN=bin/crabbox scripts/live-smoke.sh

doctor should report access=service-token. scripts/live-auth-smoke.sh proves the auth boundary without leasing a machine: requests missing Access headers are denied at the edge, shared-token user auth works, raw Access-identity spoofing is ignored, shared-token admin calls fail, and admin-token admin calls pass. A raw request without Access headers to https://broker-access.example.com/v1/health should return a Cloudflare Access 403.

Confirm which URL and provider the CLI will use:

bin/crabbox config show

#Cleanup

Brokered cleanup belongs to the coordinator scheduler: Durable Object alarms on Cloudflare or pg-boss jobs on Node. The CLI refuses provider cleanup when a coordinator is configured, because deleting machines behind the broker can remove live leases:

machine cleanup is disabled when a coordinator is configured;
coordinator TTL alarms own brokered cleanup

For brokered fleets, inspect and end leases through the broker:

bin/crabbox list
bin/crabbox admin leases --state active
bin/crabbox inspect --id blue-lobster --json
bin/crabbox stop blue-lobster

Azure release persists an immutable managed-disk cleanup claim before deleting the VM. Retries use that durable claim and recheck any live disk attachment; they never treat names or Crabbox-written ownership tags as sufficient proof.

Trusted operators can use crabbox admin release or crabbox admin delete --force for stuck leases.

After AWS credential or account rotation, scan old provider accounts directly for Crabbox-tagged EC2 instances that the current coordinator can no longer see:

scripts/aws-crabbox-orphan-audit.sh --profile old-crabbox-account

The audit is read-only. It skips keep=true instances, protects active coordinator leases by lease tag or EC2 instance ID, and applies the same grace window as the broker sweep before reporting stale labels. The script intentionally refuses --terminate: a local AWS scan cannot atomically lock coordinator lease state before deleting an instance. For broker-owned accounts, use the coordinator AWS orphan sweep below. For rotated legacy accounts, treat the JSON output as investigation evidence and delete through an explicit operator or infrastructure workflow only after confirming no active coordinator can still claim the instance.

Direct-provider cleanup is only for debug mode without a coordinator:

bin/crabbox cleanup --dry-run
bin/crabbox cleanup

#AWS orphan sweep

The coordinator schedules an AWS orphan sweep when AWS broker credentials are configured. Cloudflare uses the Durable Object alarm plus its scheduled trigger; Node uses pg-boss plus recurring reconciliation, so cleanup does not depend on new lease traffic after deploy. The sweep scans CRABBOX_AWS_REGION plus CRABBOX_CAPACITY_REGIONS for crabbox=true EC2 instances and compares their lease tags with active coordinator leases. Active matching leases always win, because provider expires_at tags are written at launch and can be older than a heartbeat-extended lease.

The sweep reports a candidate when an instance is past its provider expires_at tag, has no active lease, is missing a lease label, or points at an active lease whose current cloud ID differs. It skips keep=true instances and applies the grace window before reporting missing or mismatched lease state. Provider tags are discovery metadata, not deletion authority. Set CRABBOX_AWS_ORPHAN_SWEEP_DELETE=1 to terminate candidates only when an exact retained coordinator lease binds the same instance and region; tag-only and legacy candidates remain report-only. With CRABBOX_AWS_MAC_HOST_SWEEP_RELEASE=1 also set, the same rule permits release of a stale pending EC2 Mac Dedicated Host only when a retained coordinator lease binds that exact host.

Trusted admins can inspect or trigger the sweep:

curl -H "Authorization: Bearer $CRABBOX_COORDINATOR_ADMIN_TOKEN" \
  https://broker.example.com/v1/admin/aws-orphan-sweep

curl -X POST -H "Authorization: Bearer $CRABBOX_COORDINATOR_ADMIN_TOKEN" \
  https://broker.example.com/v1/admin/aws-orphan-sweep

See Lifecycle and Cleanup for the full lease-expiry model.

#AWS Security Guardrails

Apply the cheap account-wide guardrails before adding heavier audit services:

account_id="$(aws sts get-caller-identity --query Account --output text)"

aws s3control put-public-access-block \
  --account-id "$account_id" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

aws iam update-account-password-policy \
  --minimum-password-length 14 \
  --require-symbols \
  --require-numbers \
  --require-uppercase-characters \
  --require-lowercase-characters \
  --allow-users-to-change-password \
  --max-password-age 90 \
  --password-reuse-prevention 24

S3 account-level Block Public Access and the IAM account password policy are account-wide. IAM Access Analyzer external-access analyzers are regional, so create one in each AWS capacity region:

for region in eu-west-1 eu-west-2 eu-central-1 us-east-1 us-west-2; do
  if ! aws accessanalyzer get-analyzer \
    --region "$region" \
    --analyzer-name crabbox-external-access >/dev/null 2>&1; then
    aws accessanalyzer create-analyzer \
      --region "$region" \
      --analyzer-name crabbox-external-access \
      --type ACCOUNT
  fi
done

List active external-access findings across the same pool:

for region in eu-west-1 eu-west-2 eu-central-1 us-east-1 us-west-2; do
  arn="$(aws accessanalyzer get-analyzer \
    --region "$region" \
    --analyzer-name crabbox-external-access \
    --query 'analyzer.arn' \
    --output text)"
  aws accessanalyzer list-findings \
    --region "$region" \
    --analyzer-arn "$arn" \
    --filter '{"status":{"eq":["ACTIVE"]}}'
done

Do not treat these as spend caps or compliance audit trails. CloudTrail, AWS Config, Security Hub, and GuardDuty are separate choices with different cost and retention tradeoffs. See Security.

#Cost Guardrails

The coordinator reserves worst-case lease cost before provisioning. A request that would exceed active-lease or monthly cost limits fails (HTTP 429 cost_limit_exceeded) before any VM is created.

bin/crabbox usage
bin/crabbox usage --scope user --user alice@example.com
bin/crabbox usage --scope org --org example-org
bin/crabbox usage --scope all --json

Cost is an estimate for compute leases, not an invoice. See Cost and Usage.

#Release Checklist

The authoritative serialized release contract is Release engineering. No event automatically publishes a tag or updates Homebrew.

Before creating or reusing a signed release tag:

  • Rebase release preparation on the current main, restore the full changelog from the latest tag if concurrent work regressed it, and verify every published version remains represented.
  • Reorder CHANGELOG.md with the user-facing changes first, date the release section, and keep contributor thanks / co-author notes intact.
  • Update every package metadata file that carries the project version. The current release surface is worker/package.json plus both root package entries in worker/package-lock.json; the removed root plugin package must not be recreated.
  • go vet ./...
  • go test -race ./...
  • scripts/test-go-modules.sh
  • go build -trimpath -o bin/crabbox ./cmd/crabbox
  • scripts/check-go-coverage.sh 90.0
  • Worker gate: npm run format:check --prefix worker && npm run lint --prefix worker && npm run check --prefix worker && npm test --prefix worker && npm run build --prefix worker
  • node --test scripts/*.test.js
  • scripts/check-docs.sh
  • git diff --check
  • Live smoke at least one coordinator-backed crabbox run, then verify crabbox attach, crabbox events, crabbox logs, and lease cleanup.
  • Push, pull, and wait for CI green on the release commit.

Then advance exactly one gate at a time:

  1. Tag trust. Create or reuse an annotated signed vX.Y.Z tag. Verify it against the repository-pinned signer policy, capture the tag-object and peeled commit IDs, and require that commit to be an ancestor of the current protected main. If a valid tag already exists, preserve it; never move or recreate it merely because verifier hardening landed later.
  2. Local candidate. Build the exact eight-asset payload described in Release engineering. Ordinary builds remain credential-free. The macOS producer uses the managed release keychain to sign both native CLI architectures, the Apple Silicon helper, and its embedded VMD as Developer ID Application: OpenClaw Foundation (FWJYW4S8P8), with hardened runtime and secure timestamps, then requires accepted notarization and online codesign --check-notarization proof before packaging.
  3. Private draft. With separate explicit authorization, create exactly one GitHub draft for the captured pre-existing signed tag. Its title, exact eight assets, and body copied byte-for-byte from the tagged CHANGELOG.md section are immutable candidate inputs.
  4. Native draft verification. Dispatch only the protected-default verifier for that numeric draft ID and pinned tag identities. Its Apple Silicon and Intel jobs download assets with narrowly scoped credentials, remove all API, Actions, OIDC, and Homebrew credentials, then verify and execute the matching candidates in a clean environment.
  5. Publication. Stop for an explicit publication gate. Re-read and compare the unchanged draft, successful native proofs, tag, protected verifier SHA, notes, asset IDs, sizes, and digests. Publication is a single draft-state transition; it does not rebuild, replace, or delete anything.
  6. Published verification. Re-download the public assets by immutable asset ID and repeat the exact metadata, checksum, signature, notarization, native execution, and notes proof. The proof must be newer than publication and every release or asset mutation.
  7. Homebrew. Only after published verification, grant a separate tap-update gate. Bind every formula URL and SHA-256 to the frozen release record, then run the documented downstream verifier on clean native Apple Silicon and Intel hosts. The verifier re-fetches the current public release and run, authenticates both supplied native proof ZIPs against GitHub artifact digests, and requires that successful run to be newer than publication and every release or asset update. It then performs brew update, a fresh install or reinstall, brew test, archive-to-install byte comparison, signature/notarization checks, exact crabbox --version, and the Apple Silicon helper's non-mutating vmd-info check. Those checks are the bounded installed-binary smoke; they do not create a provider lease or authorize another mutation.

On cancellation or uncertainty, stop all release and tap writes. Inspect and record the exact draft/public release and tap state, but do not delete a partial draft or release, replace assets, rewrite the tag, redispatch, publish, or update Homebrew. Resume only from a newly authorized serialized gate.