Skip to main content

Validate Devin's Changes End to End with Signadot Sandboxes

Devin writes code and runs unit tests in its own cloud VM. What it can't easily check there is whether a change to one microservice still works with the services it talks to. That usually waits for a staging deploy, after the pull request is open.

With the Signadot CLI and the Signadot MCP server in its environment, Devin closes that loop itself. It runs the changed service in its VM, creates a sandbox that routes tagged cluster traffic to that process, runs your end-to-end tests in the cluster as a Signadot Job, and reports back only after they pass.

In this tutorial, Devin changes the route service of HotROD, a demo ride-sharing app, and validates it with HotROD's Playwright suite.

Time: about 30 minutes, including about 8 minutes for Devin's session.
Example files: signadot/examples/devin-tutorial

How it works​

Two things make this work from a cloud VM:

  • No cluster credentials. The CLI's ControlPlaneProxy connection tunnels through the Signadot control plane. Devin needs only an API key, so the cluster can even run on your laptop.
  • A full VM. Devin has passwordless sudo, so signadot local connect runs in its default privileged mode. In-cluster names such as redis.hotrod.svc resolve from the VM, and the local service can call its real dependencies.

Prerequisites​

  • A Kubernetes cluster with the Signadot Operator installed. A local cluster such as minikube, kind, k3d or Docker Desktop works.
  • The Signadot CLI logged in with signadot auth login, and kubectl pointed at the cluster.
  • A Devin organization where you can edit the environment blueprint and add MCP servers (the Manage MCP Servers permission).

Step 1: Install HotROD and a Playwright runner​

Skip this block if HotROD already runs in the hotrod namespace:

kubectl create namespace hotrod
kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'
kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m

Create the job runner group that runs HotROD's Playwright suite inside the cluster. Replace <your-cluster> with your Signadot cluster name:

git clone https://github.com/signadot/hotrod.git
cd hotrod
kubectl create namespace signadot-tests
signadot jobrunnergroup apply -f .signadot/testing/playwright-runner.yaml \
  --set cluster=<your-cluster> --set env=dev

Step 2: Create an API key for Devin​

In the Signadot dashboard, go to Admin → Service Accounts, create a service account named devin-tutorial, and create an API key for it. Copy the key; you'll paste it into Devin twice.

The service account keeps Devin's sandboxes and devboxes separate from yours. Revoking its key cuts Devin off without touching your own login.

Step 3: Store the key as a Devin secret​

In Devin, go to Settings → Secrets → Add secret. Create a Raw secret named SIGNADOT_API_KEY with the key as its value. Choose Organization scope to share it with your team, or Personal to keep it to your own sessions.

Step 4: Add the Signadot CLI to Devin's blueprint​

A Devin blueprint describes what goes into the snapshot every session's VM starts from. Go to Settings → Environment → Blueprints → Organization blueprint → Edit and paste devin/blueprint.yaml. Before you save, set your Signadot org and cluster names at the top of the Configure Signadot CLI step:

      SIGNADOT_ORG_NAME=my-org
      SIGNADOT_CLUSTER=my-cluster

The blueprint has three parts:

  • initialize installs Go and a pinned version of the Signadot CLI, then writes ~/.signadot/config.yaml with a ControlPlaneProxy connection to your cluster.
  • maintenance clones HotROD into ~/repos/hotrod and downloads its Go modules.
  • knowledge is never executed. It tells Devin how to connect, create the sandbox and run the tests. Copy this section into your own blueprints when you adapt the tutorial.
Full blueprint.yaml
initialize:
  - name: Install Go
    uses: github.com/actions/setup-go@v5
    with:
      go-version: "1.27.1"

  - name: Install Signadot CLI
    run: |
      # Pin the version: without it the installer asks the GitHub API for the
      # latest release, which is rate limited from Devin's shared build IPs.
      curl -sSLf https://raw.githubusercontent.com/signadot/cli/main/scripts/install.sh \
        | SIGNADOT_CLI_VERSION=v1.9.0 sh
      signadot --version

  - name: Configure Signadot CLI
    run: |
      # Edit these two values to match your Signadot org and cluster name.
      SIGNADOT_ORG_NAME=my-org
      SIGNADOT_CLUSTER=my-cluster
      mkdir -p ~/.signadot
      cat > ~/.signadot/config.yaml <<EOF
      org: ${SIGNADOT_ORG_NAME}
      local:
        connections:
        - cluster: ${SIGNADOT_CLUSTER}
          type: ControlPlaneProxy
      EOF

maintenance:
  - name: Clone or update HotROD
    run: |
      if [ ! -d ~/repos/hotrod ]; then
        mkdir -p ~/repos
        git clone https://github.com/signadot/hotrod ~/repos/hotrod
      fi
      cd ~/repos/hotrod
      git pull --ff-only
      go mod download

knowledge:
  - name: signadot
    contents: |
      This VM can test code changes end to end against a shared Kubernetes
      cluster with Signadot. The HotROD baseline runs in namespace `hotrod`
      of the Signadot cluster named in ~/.signadot/config.yaml. HotROD
      source: ~/repos/hotrod (Go; one binary, `go run ./cmd/hotrod <service>`;
      services: frontend, location, driver, route).

      Credentials: the `signadot` CLI reads SIGNADOT_API_KEY from the
      environment (Devin secret SIGNADOT_API_KEY). Bind it to every
      `signadot` command.

      1. Connect this VM to the cluster (once per session, before creating
         sandboxes). This also makes in-cluster service names such as
         redis.hotrod.svc resolvable from this VM. Every Devin VM boots with
         the same /etc/machine-id, which Signadot uses to pick the devbox, so
         give this session its own ID first; otherwise a second concurrent
         session fails with "409 Conflict: devbox ... is already claimed":
           [ -s ~/.signadot/machine-id ] || tr -d - < /proc/sys/kernel/random/uuid > ~/.signadot/machine-id
           export MACHINE_ID_FILE=~/.signadot/machine-id
           signadot local connect --wait connect --wait-timeout 120s
           signadot local status
      2. Run the changed service locally. For route (gRPC, port 8083):
           cd ~/repos/hotrod
           go run ./cmd/hotrod route > ~/route.log 2>&1 &
      3. Create a sandbox with the Signadot MCP `create_sandbox` tool: a
         `local` workload mapping Deployment `route` (namespace hotrod) port
         8083 to localhost:8083, `connection.devboxId` from `list_devboxes`
         (this VM's devbox, the one `signadot local status` reports), plus a
         defaultRouteGroup endpoint for http://frontend.hotrod.svc:8080.
      4. Run HotROD's Playwright e2e suite against the sandbox as a Signadot
         Job (runs in the cluster on runner group `playwright`):
           cd ~/repos/hotrod
           signadot job submit -f .signadot/testing/demo-playwright-job.yaml \
             --set branch=main --set namespace=hotrod \
             --set sandbox=<sandbox-name> --wait --timeout 15m
         Inspect a run with `signadot job get <job-name>`,
         `signadot logs --job <job-name>`, and
         `signadot artifact download --job <job-name> hotrod/playwright-report.tar.gz`.

      Never print the value of SIGNADOT_API_KEY. Do not kill processes with
      broad patterns like `pkill -f hotrod` (that matches signadot processes
      too); kill by port: `kill $(lsof -t -i :8083)`.

Click Save blueprint. Devin builds a new snapshot in about two minutes; the page shows Last built when it's ready.

Step 5: Add the Signadot MCP server​

The MCP server is a subcommand of the CLI you just installed, so it runs inside Devin's VM over STDIO. In Devin, go to Customize → MCPs, select Organization, and choose Add MCP → Add custom MCP:

FieldValue
Server nameSignadot
Transport typeSTDIO
Commandsignadot
Argumentsmcp
Environment variablesSIGNADOT_API_KEY = $SIGNADOT_API_KEY

Click Add and confirm the security notice. The server shows Needs connection because Devin doesn't share Settings → Secrets with MCP servers. Open the server, paste the API key into its Secrets field, and save. The status changes to Connected.

Step 6: Give Devin the task​

Start a new Devin session and paste this prompt (prompt.md), with my-cluster changed to your cluster name:

In ~/repos/hotrod, change the route service so that every computed ETA is
exactly 2 minutes (services/route/server.go). Keep the FAST_ROUTE override
and the rounding logic intact. Smallest possible diff.

Before you report back, validate the change end to end against the shared
cluster with Signadot (see the `signadot` knowledge for commands):

1. Connect this VM to the cluster with `signadot local connect`.
2. Run your modified route service locally on port 8083.
3. Use the Signadot MCP to create a sandbox named `devin-route-eta` in
   cluster `my-cluster` that maps Deployment `route` in namespace `hotrod` to
   localhost:8083 on this VM's devbox. I approve that spec; no need to ask.
4. Run HotROD's Playwright e2e suite against the sandbox as a Signadot Job
   and wait for it to finish. If it fails, find out why, fix it, and rerun.
5. Prove the behavior change: dispatch one ride with the sandbox routing key
   and one without (baseline), then compare the driver notifications.

Report the diff, the sandbox routing key, the job name and result, and the
two notifications. Leave the sandbox and the route service running.

Baseline ETAs are random, between 3 and 45 minutes, so an ETA of exactly 2m0s can only come from Devin's change. Keep the approval in step 3: the create_sandbox tool tells agents to confirm the spec with the user first, and without it Devin stops to ask.

What Devin does​

  1. Makes the change. Devin edits services/route/server.go, then builds and vets the package:

    -	// Generate a random number between 3 and 45 with decimals
    -	eta := time.Duration((rand.Float64()*(45-3) + 3) * float64(time.Minute))
    +	eta := 2 * time.Minute

  2. Connects its VM to the cluster with signadot local connect, then starts the changed route service on port 8083.

  3. Creates the sandbox through the MCP server. Devin calls list_clusters, resolve_workload, resolve_workload_port and list_devboxes, then create_sandbox with a spec like this:

    name: devin-route-eta
    spec:
      cluster: my-cluster
      connection:
        devboxId: <this VM's devbox ID>
      local:
      - name: local-route
        from:
          kind: Deployment
          namespace: hotrod
          name: route
        mappings:
        - port: 8083
          toLocal: localhost:8083
      defaultRouteGroup:
        endpoints:
        - name: frontend
          target: http://frontend.hotrod.svc:8080

    Requests in the cluster that carry the sandbox's routing key now reach route in Devin's VM. Everything else still reaches the baseline.

  4. Runs the Playwright suite as a Job, and fixes what fails. In our test run, the first Job failed on "Resolving routes": route defaults to REDIS_ADDR=redis:6379, a short name that resolves only inside the cluster. Devin read the Job's logs with signadot logs, downloaded the Playwright report, and restarted route with REDIS_ADDR=redis.hotrod.svc:6379. No code change was needed, and the second Job passed.

  5. Proves the behavior change by dispatching one ride with the routing key and one without. Its report includes:

    Job: hotrod-playwright-e2e-zihjipq succeeded, 2 of 2 tests passed.

    With the sandbox key: Driver T792260C arriving in 2m0s
    Baseline: Driver T775730C arriving in 8m36s

The whole session took Devin about eight minutes. You can open each Job, with its logs and the uploaded Playwright report, in the Signadot dashboard under Validations → Jobs.

Clean up​

Ask Devin to delete the sandbox and stop route before you end the session, or delete the sandbox from your machine:

signadot sandbox delete devin-route-eta

Once the Devin VM shuts down, its devbox claim lapses within about a minute.

Use it with your own services​

  • Your repository. Connect Devin to GitHub, add your repository under Settings → Environment → Repositories, and commit the blueprint as .devin/blueprint.yaml. In its knowledge section, write down how each service runs outside the cluster: environment variables, ports and dependencies.
  • Pull requests with evidence. Add "open a pull request and link the passing Signadot Job" to the prompt, so reviewers see the end-to-end result next to the diff.
  • Automation. The Devin API starts a session from a prompt, so a ticket or alert can kick off the same write, sandbox and test loop.
  • Agent skills. The signadot-validate skill packages this validation loop. Devin loads skills from .agents/skills/ in your repository.

Troubleshooting​

  • Blueprint build fails with signadot: command not found. The CLI installer couldn't resolve the latest release. Pin it with SIGNADOT_CLI_VERSION, as the blueprint does.
  • 409 Conflict: devbox ... is already claimed. Another Devin session with the same API key holds the devbox. Make sure Devin sets a per-session MACHINE_ID_FILE (step 1 of the blueprint's knowledge), or end the other session.
  • The MCP server shows "Needs connection". Paste the API key into the MCP server's own Secrets field.
  • list_clusters asks which cluster to use. Your org has more than one cluster. Name it in the prompt.
  • The driver logs DeadlineExceeded. The driver gives each route call one second, and traffic to Devin's VM goes through the control plane. Run route with ROUTE_CALC_DELAY=1ms to remove its simulated 50 ms of work.

Frequently asked questions​

How does Devin reach my Kubernetes cluster?​

Through the Signadot control plane. The blueprint configures the Signadot CLI with a ControlPlaneProxy connection, so Devin's VM needs only a Signadot API key: no kubeconfig and no network path to your cluster's API server. This also works for a local cluster such as minikube, kind, k3d or Docker Desktop.

Does Devin's sandbox affect other people using the cluster?​

No. A Signadot sandbox only receives requests that carry its routing key. Every other request, including your teammates' traffic and other sandboxes, keeps reaching the baseline services.

Can several Devin sessions use Signadot at the same time?​

Yes. Every Devin VM boots with the same /etc/machine-id, which Signadot uses to pick a devbox. The blueprint tells Devin to write a fresh machine ID for each session and point MACHINE_ID_FILE at it, so each session gets its own devbox and sandbox under one service account.

Which Signadot MCP tools does Devin use?​

list_clusters, resolve_workload, resolve_workload_port, list_devboxes and create_sandbox, plus get_workload_object while debugging. The MCP server has no Job tools yet, so Devin runs the end-to-end suite with signadot job submit.

What does my application need for this to work?​

Your services must propagate the routing key from incoming to outgoing requests, for example as OpenTelemetry baggage, so that requests tagged for the sandbox reach Devin's copy of the service. HotROD already does this with OpenTelemetry. The Signadot docs on context propagation show how to set it up in your own services.

See also​