Skip to main content

Background Coding Agents: End-to-End Testing with Modal Sandboxes and Signadot

Modal Sandboxes are secure, ephemeral cloud containers built for running agent code. They start in seconds, which makes them a good home for background coding agents. What a Modal Sandbox can't do on its own is test a change to one microservice against the services it depends on, because those run in your Kubernetes cluster.

Signadot connects the two. The Signadot CLI in the Modal Sandbox connects to your cluster with only an API key. A Signadot sandbox then routes requests that carry its routing key to the service running in the Modal Sandbox. All other traffic keeps reaching the shared baseline.

In this tutorial, Claude Code runs in a Modal Sandbox and changes the route service of HotROD, a demo ride-sharing app. You then verify the change end to end through HTTP, Kafka and gRPC.

Time: about 20 minutes.
Example files: signadot/examples/modal-sandboxes-tutorial

Two kinds of sandbox

The Modal Sandbox is the cloud container where Claude Code and your changed service run. The Signadot sandbox is the routing rule in your cluster that sends tagged requests to it.

How it works​

Two things make this work from a Modal Sandbox:

  • No cluster credentials. The CLI's ControlPlaneProxy connection relays traffic through the Signadot control plane. The Modal Sandbox needs only SIGNADOT_ORG and SIGNADOT_API_KEY: no kubeconfig, no VPN and no network path to the cluster API.
  • Unprivileged mode. Modal Sandboxes run on gVisor, which doesn't implement netfilter NAT, so signadot local connect runs with --unprivileged. Traffic from the cluster still tunnels into the Modal Sandbox. There's no cluster DNS, so you reach in-cluster services through signadot local proxy port mappings instead.

HotROD makes this a strong test of context propagation. frontend publishes each dispatch to Kafka, driver consumes it and calls route over gRPC, and the routing key rides the whole way in OpenTelemetry baggage.

Prerequisites​

  • A Kubernetes cluster with the Signadot Operator installed, and HotROD running in the hotrod namespace:

    kubectl create namespace hotrod
    kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'
    kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m

  • A Signadot API key. In the Signadot dashboard, go to Admin → Service Accounts, create a service account, and create an API key for it.

  • A Modal account, with the Modal CLI installed and logged in on your laptop: pip install modal && modal token new.

  • A Claude subscription or an Anthropic API key, to run Claude Code in the Modal Sandbox.

Clone the example files:

git clone https://github.com/signadot/examples.git
cd examples/modal-sandboxes-tutorial

Step 1: Create the Modal secrets​

Credentials reach the Modal Sandbox only as Modal Secrets, never baked into the image. Create one for Signadot:

modal secret create signadot-credentials SIGNADOT_ORG=<your-org> SIGNADOT_API_KEY=<your-api-key>

Create one for Claude Code. With a Claude subscription, run claude setup-token on your laptop and store the token it prints:

modal secret create claude-code-auth CLAUDE_CODE_OAUTH_TOKEN=<token>

Or store an Anthropic API key instead:

modal secret create claude-code-auth ANTHROPIC_API_KEY=<key>

Step 2: Launch the Modal Sandbox​

Run the launcher. Replace my-cluster with your cluster's name in Signadot:

modal run launch_sandbox.py --cluster my-cluster

launch_sandbox.py builds an image with Go, Node.js, Claude Code and the Signadot CLI. The first build takes about 2 minutes; later launches start in seconds. It then creates a Modal Sandbox with both secrets attached and a 4-hour timeout. The Modal Sandbox belongs to a persistent app, so it keeps running after modal run returns.

Finally, the launcher runs in-sandbox/setup.sh in the Modal Sandbox. It writes a machine ID, which gVisor doesn't provide, then configures the CLI and connects:

cat > "$HOME/.signadot/config.yaml" <<EOF
local:
  connections:
  - cluster: ${CLUSTER}
    type: ControlPlaneProxy
EOF

signadot local connect --unprivileged --wait connect --wait-timeout 120s

There's no signadot auth login: the CLI reads SIGNADOT_ORG and SIGNADOT_API_KEY from the environment. The output ends with the ID of your Modal Sandbox:

signadot local connect has been started ✓
✓ Local connection healthy!
    * operator version 1.3.2
    * devbox 4a0e7f2218ae964ef68b08405c959d7b connected
    * sandboxes watcher is running

Connect with:  modal shell sb-Dx4LUDetYGvzz2DqKJMjE5

Step 3: Create the Signadot sandbox​

Open a shell in the Modal Sandbox, using the ID from the previous step. Run every command from here until Clean up in this shell:

modal shell sb-Dx4LUDetYGvzz2DqKJMjE5

/root/workspace/local-route.yaml defines a Signadot sandbox that maps the in-cluster route Deployment to port 8083 in the Modal Sandbox:

name: modal-route-dev
spec:
  description: HotROD route service under development in a Modal sandbox
  cluster: "@{cluster}"
  ttl:
    duration: 1d
  local:
  - name: modal-route
    from:
      kind: Deployment
      namespace: "@{namespace}"
      name: route
    mappings:
    - port: 8083
      toLocal: "localhost:8083"
  defaultRouteGroup:
    endpoints:
    - name: frontend
      target: "http://frontend.@{namespace}.svc:8080"

Apply it:

signadot sandbox apply -f /root/workspace/local-route.yaml --set cluster=my-cluster --set namespace=hotrod
Created sandbox "modal-route-dev" (routing key: ug86wvt9f5nhq) in cluster "my-cluster".
✓ Sandbox status: Ready: All desired workloads are available.

Requests in the cluster that carry this routing key and reach route now go to localhost:8083 in the Modal Sandbox. Nothing listens there yet.

Step 4: Have Claude Code change the service​

Clone HotROD and start Claude Code:

git clone https://github.com/signadot/hotrod /root/workspace/hotrod
cd /root/workspace/hotrod
claude

/root/workspace/CLAUDE.md already describes this environment to Claude Code. Ask for a change that's easy to spot:

In services/route/server.go, change the ETA computation so every computed ETA is exactly 2 minutes. Keep the FAST_ROUTE override and rounding logic intact. Smallest possible diff.

Claude Code makes this edit:

-	// Generate a random number between 3 and 45 with decimals
-	eta := time.Duration((rand.Float64()*(45-3) + 3) * float64(time.Minute))
+	eta := 2 * time.Minute
 	if os.Getenv("FAST_ROUTE") != "" {
 		eta = time.Second
 	}

Baseline ETAs are random, between 3 and 45 minutes, so an ETA of exactly 2m0s can only come from the changed code.

Step 5: Run the changed service​

Exit Claude Code and start route on port 8083:

cd /root/workspace/hotrod
ROUTE_CALC_DELAY=1ms go run ./cmd/hotrod/main.go route > /root/route.log 2>&1 &

ROUTE_CALC_DELAY=1ms removes the service's simulated 50 ms of work. The driver service gives each route call one second, and calls tunneled through ControlPlaneProxy already carry a few hundred milliseconds of latency.

Step 6: Test end to end​

Open two proxies to the in-cluster frontend. The first runs in the Signadot sandbox's context and adds its routing key to every request; the second reaches the baseline:

signadot local proxy --sandbox modal-route-dev --map http://frontend.hotrod.svc:8080@localhost:18080 > /root/proxy-sandbox.log 2>&1 &
signadot local proxy --cluster my-cluster --map http://frontend.hotrod.svc:8080@localhost:18081 > /root/proxy-baseline.log 2>&1 &

Dispatch a ride through each proxy. HotROD processes dispatches asynchronously through Kafka, so read the results from the notifications endpoint a few seconds later:

curl -s -X POST localhost:18080/dispatch -H "Content-Type: application/json" \
  -d '{"SessionID": 9001, "RequestID": 41, "PickupLocationID": 123, "DropoffLocationID": 1}'
curl -s -X POST localhost:18081/dispatch -H "Content-Type: application/json" \
  -d '{"SessionID": 9002, "RequestID": 42, "PickupLocationID": 123, "DropoffLocationID": 1}'

sleep 15
curl -s "localhost:18080/notifications?sessionID=9001&cursor=0" | jq -r '.notifications[] | .id + " | rk=" + .context.routingKey + " | " + .body'
curl -s "localhost:18081/notifications?sessionID=9002&cursor=0" | jq -r '.notifications[] | .id + " | rk=" + .context.routingKey + " | " + .body'

The request with the routing key gets the 2-minute ETA from the code running in the Modal Sandbox:

req-41-location-resolve | rk=ug86wvt9f5nhq | Resolving locations
req-41-finding-driver | rk=ug86wvt9f5nhq | Finding an available driver
req-41-dispatched-driver | rk=ug86wvt9f5nhq | Driver T729782C arriving in 2m0s

The baseline request is untouched, with a random ETA and no routing key:

req-42-location-resolve | rk= | Resolving locations
req-42-finding-driver | rk= | Finding an available driver
req-42-route-resolve | rk= | Resolving routes
req-42-dispatched-driver | rk= | Driver T782962C arriving in 3m10s

The first request went from the baseline frontend through Kafka to the baseline driver. Only its gRPC call to route left the cluster for the Modal Sandbox. Every other service was shared.

You can also test from anywhere through the Signadot sandbox's preview URL, without proxies:

curl -s -X POST -H "signadot-api-key: $SIGNADOT_API_KEY" -H "Content-Type: application/json" \
  -d '{"SessionID": 8001, "RequestID": 31, "PickupLocationID": 123, "DropoffLocationID": 1}' \
  https://frontend--modal-route-dev.preview.signadot.com/dispatch

Clean up​

In the Modal Sandbox, delete the Signadot sandbox:

signadot sandbox delete modal-route-dev

On your laptop, terminate the Modal Sandbox:

python3 -c "import modal; modal.Sandbox.from_id('sb-Dx4LUDetYGvzz2DqKJMjE5').terminate()"

If you forget, nothing leaks: the Modal Sandbox stops after its 4-hour timeout, and the Signadot sandbox expires after its 1-day TTL.

Use it with your own services​

  • Your service. In local-route.yaml, point from at your Deployment and set its port. Update in-sandbox/CLAUDE.md with how your service runs outside the cluster, and add its toolchain to the image in launch_sandbox.py.

  • Headless runs. Background agents and CI don't need modal shell. From any Python process, run Claude Code with sb.exec():

    sb = modal.Sandbox.from_id("sb-...")
    p = sb.exec("bash", "-lc",
        'cd /root/workspace/hotrod && claude -p "fix the failing ETA test" --allowedTools "Edit,Read,Bash"')
    print(p.stdout.read())

  • Lower latency. ControlPlaneProxy is rate-limited and meant for setup and evaluation. For team use, change the connection in setup.sh to PortForward, with a kubeconfig for a scoped service account delivered as another Modal Secret, or to ProxyAddress, with the Signadot Operator's SOCKS5 proxy exposed on a network Modal can reach. See Connection Configs. Both also remove the need for ROUTE_CALC_DELAY.

Troubleshooting​

  • 409 Conflict: devbox ... is already claimed in session .... Another environment, often your laptop, holds a local-connect session for the same user. Run signadot local disconnect there, then connect again in the Modal Sandbox.
  • couldn't read machine-id. gVisor has no /etc/machine-id. setup.sh generates one; if you skipped it, create the file yourself.
  • iptables ... exit status 1 or other localnet errors. You ran signadot local connect without --unprivileged. Modal Sandboxes can't do netfilter NAT.
  • The driver logs DeadlineExceeded and no final notification appears. Tunneled route calls took longer than one second. Run route with ROUTE_CALC_DELAY=1ms, or use a lower-latency connection type.
  • The proxies or route stopped unexpectedly. Avoid broad patterns like pkill -f hotrod, which also match the proxies' arguments. Stop processes by port instead: kill $(lsof -t -i :8083). If the connection drops, run signadot local connect --unprivileged again; the Signadot sandbox reattaches within seconds.

Frequently asked questions​

Does the Modal Sandbox need a kubeconfig or VPN to reach my cluster?​

No. The Signadot CLI in the Modal Sandbox uses a ControlPlaneProxy connection, which relays all cluster traffic through the Signadot control plane. The Modal Sandbox needs only SIGNADOT_ORG and SIGNADOT_API_KEY: no kubeconfig, no VPN and no network path to your cluster's API server.

Why does signadot local connect need --unprivileged in a Modal Sandbox?​

Modal Sandboxes run on gVisor, which doesn't implement netfilter NAT, so the CLI can't set up the virtual IPs and cluster DNS of its default privileged mode. Unprivileged mode still tunnels sandbox traffic into the Modal Sandbox. To call in-cluster services from the Modal Sandbox, use signadot local proxy port mappings.

Does testing from a Modal Sandbox affect other people using the cluster?​

No. The Signadot sandbox only receives requests that carry its routing key. Every other request, including your teammates' traffic and other sandboxes, keeps reaching the baseline services.

Can a background agent run this without an interactive shell?​

Yes. Call sb.exec() from the Modal Python SDK to run Claude Code headlessly with claude -p, the same way a CI job or an agent orchestrator would. The routing and tests work the same as in modal shell.

Is ControlPlaneProxy suitable for production use?​

It is intended for setup and evaluation: it is rate-limited and adds latency. For team-scale use, switch the connection to PortForward, with a kubeconfig delivered as a Modal Secret, or to ProxyAddress, with the Signadot Operator's proxy exposed on a network the Modal Sandbox can reach.

See also​