Skip to main content

Run Tests with Plans

Overview​

Signadot runs your tests against sandboxes in two ways. Pick the one that fits how you want to work:

  • Jobs: you run your test suite the way your CI does. You write the script, choose the container image, and submit it from the CLI or a CI pipeline. See Run Tests with Jobs.
  • Plans: your coding agent sets up and runs the tests for you. It turns your tests into a reusable plan, runs the plan to check its own changes, and CI can run the same plan by name.

This quickstart covers Plans. A coding agent wraps the Playwright suite that already lives in the HotROD repository in a plan, without changing the tests, and then uses that plan to check a code change. It uses the two Signadot agent skills:

  • signadot-plan creates the plan from one prompt.
  • signadot-validate runs the plan against each change the agent makes, and fixes the change until the tests pass.

Before you begin​

  • A Signadot account (sign up here) and the Signadot CLI v1.6.0 or later (install).
  • A Kubernetes cluster with the Signadot Operator v1.3.1 or later. Local clusters such as minikube, k3s or Docker Desktop work. Playground Clusters don't run plans.
  • HotROD in the hotrod namespace: 
    kubectl create ns hotrod || true
    kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'
  • Plans enabled on your cluster: on the Managed Runners page, select the cluster and set Plan Runners to 1 (details).
  • The Playwright image allowed for plans. An org admin adds mcr.microsoft.com/playwright:* on the same page, under Settings > Plan Runner Actions > Configure > run-container > Allowed images.
  • A local clone of signadot/hotrod.
  • signadot local connect running in a separate terminal, for Step 4.
  • A coding agent such as Claude Code, Cursor or Codex, with the Signadot skills installed: 
    npx skills add signadot/agent-skills

The tests​

HotROD's Playwright suite is in playwright-tests/. It requests a ride in the web UI and checks the driver details on the page. Its playwright.config.ts adds a routing-key header to every request, which is what sends the test traffic to a sandbox:

playwright.config.ts (excerpt)
extraHTTPHeaders: {
  baggage: `sd-routing-key=${process.env.SIGNADOT_ROUTING_KEY}`,
},

If your own suite doesn't send this header yet, adding it is usually the only change it needs.

Step 1: Create a sandbox​

Create a sandbox with a changed driver service. The change (signadot/hotrod#260) adds an SD- prefix to driver license plates. Save this as driver-plates.yaml:

name: driver-plates
spec:
  description: Adding SD- prefix to driver plates
  cluster: "@{cluster}"
  forks:
    - forkOf:
        kind: Deployment
        namespace: hotrod
        name: driver
      customizations:
        images:
          - image: signadot/hotrod:f788b05bca80429425b814da62b2f17c3c564089-linux-amd64
            container: hotrod
signadot sandbox apply -f ./driver-plates.yaml --set cluster=<your-cluster>

Step 2: Have your agent create the plan​

Open your agent in the HotROD clone and ask:

Create a Signadot plan that runs this repo's existing Playwright tests against a sandbox. Use the run-container action with the mcr.microsoft.com/playwright:v1.45.1-jammy image to clone the repo and run the tests with their own config. Keep the HTML report. Run it against the driver-plates sandbox, and tag it hotrod-e2e once it passes.

The signadot-plan skill reads your test config and the action catalog, writes the plan, creates it, runs it against driver-plates, fixes anything that fails, and applies the tag. The Playwright image is large, so a run can take a few minutes. The plan it writes looks similar to this one, though details such as param names can differ:

Generated plan spec
spec:
  selectionHint: "Run HotROD's existing Playwright suite (playwright-tests/ in signadot/hotrod) against a sandbox. Pass sandbox; optionally branch."
  cluster:
    fromSandbox: sandbox
  params:
  - name: sandbox
    required: true
  - name: branch
    default: main
  steps:
  - id: e2e
    action:
      actionID: <run-container-action-id>
    routingContext:
      ref:
        sandboxRef: params.sandbox
    args:
      values:
        image: mcr.microsoft.com/playwright:v1.45.1-jammy
        timeout: 15m
        script: |
          set -e
          git clone --depth 1 -b "$(cat ./context/branch)" https://github.com/signadot/hotrod.git
          cd hotrod
          CYPRESS_INSTALL_BINARY=0 npm ci
          export HOTROD_NAMESPACE=hotrod CI=true
          set +e
          npx playwright test --reporter=list,html
          ec=$?
          tar czf "$OLDPWD/outputs/report" playwright-report
          exit $ec
      refs:
        branch: params.branch
    extraInputs:
    - name: branch
    extraOutputs:
    - name: report
      metadata:
        contentType: application/gzip
  output:
    exit_code: steps.e2e.outputs.exit_code
    report: steps.e2e.outputs.report

The plan has a single step that does what a CI job would: clone the repository, install dependencies and run npx playwright test. routingContext puts the sandbox's routing key in the step's environment, and the suite's config sends it with every request. The tests stay in Git, so changing a test never means changing the plan.

Confirm the tag:

signadot plan tag get hotrod-e2e -o json | jq '{name, plan: .plan.id, selectionHint: .plan.spec.selectionHint}'

Step 3: Run the plan yourself​

Anyone can now run the suite against any sandbox by tag, the same way CI would:

signadot plan run --tag hotrod-e2e --param sandbox=driver-plates
Created execution ckm23zqmyptbj for plan k6zvc4kmmjuqy
Phase:       completed
Steps:       1/1 completed
...
Outputs:
  exit_code   ← 0   (inline, from e2e)
  report      (artifact, 4.129MB, ready, application/gzip, from e2e)

Download the report and open it, using the execution ID from the first line:

signadot plan x get-output <execution-id> report > report.tar.gz
tar xzf report.tar.gz
npx playwright show-report playwright-report

Open the check driver series test and step through its trace: the driver plates start with SD-. The tests didn't change, but their requests reached the driver service in your sandbox.

signadot plan run exits with 1 when a test fails, so the same command works as a CI check. A failed step doesn't keep its outputs, though; Set up Tests using Plans shows how to keep the report from failed runs too.

Step 4: Let the agent validate a change​

Ask your agent:

In services/driver/driverstore.go, add an SD- prefix to the license plates that FindDriverIDs generates. Then validate the change against the hotrod-e2e plan on cluster <your-cluster>.

The agent makes the change, and the signadot-validate skill takes over:

  1. It creates a sandbox that sends driver traffic to the changed service, which it runs on your machine through local connect.
  2. It runs hotrod-e2e against that sandbox, so the tests' requests to driver reach the process on your machine.
  3. It reads the result. If a test fails, it fixes the change and runs the plan again until the tests pass.
  4. It reports the sandbox, the files it changed, the result, and the cleanup commands.

HotROD's test accepts driver plates with or without the prefix, so a passing run shows that the change didn't break the ride flow. To see the new plates themselves, open the run's report as in Step 3.

Step 5: Clean up​

Stop the local processes with the commands from the agent's report, then:

signadot sandbox delete <sandbox-from-the-report>
signadot sandbox delete driver-plates
signadot local disconnect

The plan stays in your org for the next change. To remove its tag, run signadot plan tag delete hotrod-e2e.

Next steps​