Set up Tests using Plans

Overview

A Signadot plan is a reusable test run against a sandbox. You, your CI pipeline and your coding agents run it by name, passing the sandbox to test.

A plan is a small workflow. Each step calls an action from your org's catalog, and steps pass values to each other through references, which also decide the order they run in. A typical plan has one step that runs the tests and a few around it:

Before the tests: compose a value with eval, or call an API with request-http to seed data or check that a dependency is up.
The tests: a framework action such as playwright or k6, or run-container to run your CI test command in your CI image.
After the tests: record pass or fail with check, or fail the plan with a gate step.

There are two ways to get tests into a plan:

	Tests written into the plan	Tests from your repository
How it works	Your coding agent writes the test into the plan, using a framework action such as `playwright`, `request-http` or `k6`	A `run-container` step clones your repository and runs your CI test command in your CI image
Good for	New checks of a user flow or an API, written from a description	Suites you already have, with their config, fixtures and dependencies
Changing a test	Create a new plan and move the tag	Push a commit; the plan doesn't change
Extra setup	None	Allow your CI image for `run-container`

For a hands-on walk-through that wraps an existing suite, start with the Run Tests with Plans quickstart.

Let your coding agent write the plan

You rarely need to write a plan spec by hand. The signadot-plan agent skill writes it from a description: it reads your org's action catalog and each action's contract, reads your tests and their config when there are any, drafts the spec, creates the plan, runs it once against a sandbox and applies a tag. You review the result.

Install the skills in your coding agent, such as Claude Code, Cursor or Codex:

npx skills add signadot/agent-skills

Then describe what you want. To have the agent write the test:

Create a Signadot plan tagged checkout-flow that uses Playwright to add an item to the cart in our web app, check out, and check that the order confirmation page shows the order number. Run it against a sandbox I pass in, and make the plan fail when the test fails.

To wrap a suite you already have:

Our checkout suite runs in CI with npm ci && npx playwright test in the mcr.microsoft.com/playwright:v1.45.1-jammy image. Create a Signadot plan tagged checkout-e2e that runs that suite unchanged against a sandbox, clones the repo at a branch I pass in, and keeps the HTML report.

The sections below explain what goes into such plans, so you can review what the agent produced or adjust it. Every field is described in the Plan specification. Once a plan is tagged, the signadot-validate skill runs it against the sandbox for each change the agent makes.

Before you start

An org admin sets these up once, on the Managed Runners page:

Plans on each cluster that runs tests. Select the cluster and raise Plan Runners to 1. Plans require Signadot Operator v1.3.1 or later. See Enabling Plan Runner Groups for node requirements.
The images run-container may use, if you wrap existing suites. Actions such as playwright bring their own image. run-container runs the image each step names, which has to be on its allowed list: under Settings > Plan Runner Actions > Configure, add each image to the Allowed images list of run-container. Entries must be fully qualified (docker.io/library/python:3.12, not python:3.12) and can be globs (ghcr.io/acme/ci-*:*). The images your CI test jobs already use are the natural starting point. Plan creation fails with image ... is not in this org's allowed-images list for anything else.

1. Send the routing key with every request

A test reaches the sandbox when its requests carry the sandbox's routing key. A plan step that targets a sandbox gets the key in the SIGNADOT_ROUTING_KEY environment variable; the test sends it as a header. Set it once, at the HTTP layer of the framework:

Playwright: playwright.config.ts, or test.use() in a test file
use: {
  extraHTTPHeaders: {
    baggage: `sd-routing-key=${process.env.SIGNADOT_ROUTING_KEY}`,
  },
},

pytest with requests: conftest.py
import os, pytest, requests

@pytest.fixture
def http():
    s = requests.Session()
    s.headers["baggage"] = f"sd-routing-key={os.environ['SIGNADOT_ROUTING_KEY']}"
    return s

When an agent writes the test into the plan, it adds the header for you. For suites in your repository, this is usually the only change they need. Point the tests at the in-cluster service addresses (http://<service>.<namespace>.svc:<port>), or at your usual staging hostname if the cluster serves it. The testing framework integrations cover Cypress, Playwright and Postman in more detail, and Context Propagation explains how the header travels between services.

2. Write tests into the plan

Framework actions run a test that lives in the plan itself. The playwright action runs one test file in its own Playwright image and produces a JSON report, an exit_code and, with capture_artifacts, the traces and videos. Its base_url input becomes the BASE_URL environment variable.

This plan, written for the HotROD demo app, requests a ride and checks the driver's license plate:

spec:
  selectionHint: "End-to-end check of HotROD's ride request: request a ride in the web UI and check the driver's plate. Pass sandbox."
  cluster:
    fromSandbox: sandbox
  params:
  - name: sandbox
    required: true
  steps:
  - id: e2e
    action:
      actionID: <playwright-action-id>
    routingContext:
      ref:
        sandboxRef: params.sandbox
    args:
      values:
        base_url: http://frontend.hotrod.svc:8080
        capture_artifacts: "true"
        script: |
          import { test, expect } from '@playwright/test';

          test.use({
            baseURL: process.env.BASE_URL,
            trace: 'on',
            extraHTTPHeaders: {
              baggage: `sd-routing-key=${process.env.SIGNADOT_ROUTING_KEY}`,
            },
          });

          test('request a ride and check the driver', async ({ page }) => {
            await page.goto('/');
            await page.getByRole('combobox').first().selectOption('1');
            await page.getByRole('combobox').nth(1).selectOption('123');
            await page.getByRole('button', { name: 'Request Ride' }).click();
            await expect(
              page.locator("//div[p[2][contains(text(), 'driver')]]/p[4]").last(),
            ).toHaveText(/T7\d{5}C/);
          });
  output:
    exit_code: steps.e2e.outputs.exit_code
    report: steps.e2e.outputs.report
    test_artifacts: steps.e2e.outputs.test_artifacts

Keep in mind:

One self-contained file. The action saves the script as a single .js file, so imports of other files don't resolve, and TypeScript-only syntax such as interfaces and type annotations fails to parse. Settings that would normally live in playwright.config.ts, such as the base URL, headers or tracing, go in a test.use(...) call.
The step always succeeds. The result is in exit_code (0 when every test passed) and report. Add a step after it to record the result or to fail the plan when a test fails.
Changing the test means a new plan. Plans are immutable, so the agent creates a new plan from the edited spec and moves the tag with signadot plan tag apply. Anything that runs the plan by tag picks up the change.

Other framework actions work the same way: request-http for API checks and k6 for load tests (see the k6 load testing tutorial).

3. Wrap tests from your repository

The run-container action runs a script in the image the step names. Use the image and the commands from your CI job, and the suite runs with its own config, fixtures and dependencies:

spec:
  selectionHint: "Run the checkout service integration suite against a sandbox. Pass sandbox; optionally ref."
  cluster:
    fromSandbox: sandbox
  params:
  - name: sandbox
    required: true
  - name: ref
    default: main
  steps:
  - id: tests
    action:
      actionID: <run-container-action-id>
    routingContext:
      ref:
        sandboxRef: params.sandbox
    args:
      values:
        image: <your-ci-test-image>
        timeout: 20m
        script: |
          set -e
          git clone --depth 1 -b "$(cat ./context/ref)" https://github.com/<org>/<repo>.git src
          cd src
          <install dependencies, as in CI>
          <run the suite, as in CI>
      refs:
        ref: params.ref
    extraInputs:
    - name: ref

Get an action's ID with signadot plan action get run-container -o json | jq -r .id, then create and tag the plan:

signadot plan create -f ./checkout-tests-plan.yaml -o json | jq -r .id
signadot plan tag apply checkout-tests --plan <plan-id>
signadot plan run --tag checkout-tests --param sandbox=<sandbox-name>

Keep in mind:

Inputs arrive as files. A value you wire into the step through extraInputs and refs appears as ./context/<name>, so the script reads it with cat. Routing values are the exception: they're environment variables set from routingContext, and you don't pass them as params.
Use the CI command as is. Run the same npm run, pytest, go test or make target your CI runs, with its usual config file. If your CI sets environment variables for the suite, set the same ones in the script.
Install only what the suite needs. The step starts from a clean container each run, so skip anything slow that the suite doesn't use, such as browser or binary downloads for other test tools.
Prefer slim images. Each step starts a fresh container from its image. Small images start in seconds; large browser images take noticeably longer. The runner caches images after the first pull.
Changing a test needs no new plan, because the tests come from your repository. You only create a new plan, and move the tag to it, when the image or the command changes.

4. Add steps before and after the tests

Because a plan is a workflow, the step that runs your tests doesn't have to stand alone. Add steps from the catalog around it, and wire them together with references:

eval composes a value from params and outputs, such as a URL, a header or a request body.
request-http calls an API and captures the full request and response, for example to seed test data through your API before the tests, or to check a response after them.
check evaluates a boolean expression against earlier outputs and records the result, such as "no tests failed" or "the response status was 200".
k6 adds a load test next to the functional one.

A step runs once the steps it references have finished, and a condition can skip it. For example, this check step reads the playwright step's JSON report and records whether every test ran and passed:

  - id: passed
    action:
      actionID: <check-action-id>
    args:
      values:
        name: all tests passed
        expression: report.stats.expected > 0 && report.stats.unexpected == 0 && len(report.errors) == 0
      refs:
        report: steps.e2e.outputs.report
    extraInputs:
    - name: report
      schema:
        type: object

Read the result from report rather than exit_code: the playwright action writes exit_code as plain text, which can't be passed to an input with a schema, and check gives its inputs one. Checking report.errors as well catches a test file that fails to load, which reports no failed tests. List the actions enabled in your org with signadot plan action list, and read one's contract with signadot plan action get <name>. The full catalog is in signadot/actions.

5. Fail the plan when tests fail

When CI runs a plan, a failing test should fail the plan, so that signadot plan run exits with 1. A check step records whether its expression passed, but a failed check doesn't fail the plan.

Tests from your repository

A run-container step needs nothing extra. When the script exits with the suite's exit code, a failing test fails the step and the plan.

The trade-off is that a failed step doesn't keep its outputs, so a run with failing tests has no report.

Keep the report when tests fail

To keep the report and still fail the plan, let the test step always succeed, record the suite's exit code as an output, and add a gate step that exits with it:

  steps:
  - id: tests
    action:
      actionID: <run-container-action-id>
    routingContext:
      ref:
        sandboxRef: params.sandbox
    args:
      values:
        image: mcr.microsoft.com/playwright:v1.45.1-jammy
        timeout: 15m
        script: |
          set -e
          OUT="$PWD/outputs"
          git clone --depth 1 -b "$(cat ./context/ref)" https://github.com/<org>/<repo>.git src
          cd src
          npm ci
          set +e
          npx playwright test --reporter=list,html
          printf '%d' $? > "$OUT/tests_exit_code.json"
          tar czf "$OUT/report" playwright-report
      refs:
        ref: params.ref
    extraInputs:
    - name: ref
    extraOutputs:
    - name: tests_exit_code
      schema:
        type: integer
    - name: report
      metadata:
        contentType: application/gzip
  - id: gate
    action:
      actionID: <run-container-action-id>
    args:
      values:
        image: public.ecr.aws/docker/library/alpine:3.20
        script: |
          code=$(cat ./context/code.json)
          echo "Test exit code: $code"
          exit "$code"
      refs:
        code: steps.tests.outputs.tests_exit_code
    extraInputs:
    - name: code
      schema:
        type: integer
  output:
    tests_exit_code: steps.tests.outputs.tests_exit_code
    report: steps.tests.outputs.report

Outputs and inputs that declare a schema are written as <name>.json, which is why the exit code goes to tests_exit_code.json and the gate reads code.json. See JSON value handling. The gate image has to be on the run-container allowed list.

A failed plan doesn't set its plan-level outputs, so fetch the report from the step that wrote it:

signadot plan x get-output <execution-id> tests/report > report.tar.gz

Tests written into the plan

The playwright step always succeeds, so it needs a gate step to fail the plan. A small run-container step reads the step's exit_code, without a schema because the action writes it as plain text, and exits with it:

  - id: gate
    action:
      actionID: <run-container-action-id>
    args:
      values:
        image: public.ecr.aws/docker/library/alpine:3.20
        script: |
          code=$(cat ./context/code)
          echo "Playwright exit code: $code"
          exit "$code"
      refs:
        code: steps.e2e.outputs.exit_code
    extraInputs:
    - name: code

When a test fails, the e2e step completes with its report and traces, the gate step fails, and so does the plan. Fetch the report and traces from the step, for example with signadot plan x get-output <execution-id> e2e/test_artifacts.

6. Clone private repositories

Keep credentials out of the plan spec. Store the token as an org secret, declare a plan param for it, and bind the param to the secret when you run the plan:

signadot secret create github-test-token --value-file ./token.txt \
  --description "Read-only token for cloning test repositories"

  params:
  - name: sandbox
    required: true
  - name: git_token
    required: true
  steps:
  - id: tests
    # ...
    args:
      values:
        script: |
          set -e
          git clone --depth 1 \
            "https://x-access-token:$(cat ./context/git_token)@github.com/<org>/<repo>.git" src
          # ...
      refs:
        git_token: params.git_token
    extraInputs:
    - name: git_token

signadot plan run --tag checkout-tests \
  --param sandbox=<sandbox-name> \
  --param-secret git_token=github-test-token

The control plane resolves the secret when the run starts and never returns its value in API responses. Don't echo it from the script. See spec.secrets for the rules.

7. Run plans from CI

In a pull request workflow, create the sandbox first, then run the tagged plan against it. signadot plan run exits with 0 when the plan completes, 1 when it fails and 2 when it's cancelled.

To publish the report whether or not the tests pass, use the keep-the-report pattern and read the report from the step that wrote it, since a failed plan doesn't set its plan-level outputs:

set +e
signadot plan run --tag checkout-tests \
  --param sandbox="${SANDBOX_NAME}" \
  --param ref="${BRANCH_NAME}" \
  -o json > plan-run.json 2>&1
status=$?
set -e
signadot plan x get-output "$(jq -r .id plan-run.json)" tests/report > report.tar.gz
exit $status

With -o json, the CLI prints the execution as JSON, on stderr when the plan fails, so the command captures both streams. When you only need outputs from passing runs, --output-dir ./plan-outputs saves every output of a completed plan.

The CLI authenticates with the SIGNADOT_ORG and SIGNADOT_API_KEY environment variables. See CI/CD integrations for creating sandboxes from GitHub Actions, GitLab, Bitbucket and Jenkins.

8. Let coding agents run your tests

A tagged plan is also something a coding agent can run. The signadot-validate agent skill lists your plan tags, picks the one whose selectionHint matches the change it's validating, and runs it against the sandbox it created. Write the selectionHint the way you'd brief a teammate: which tests the plan runs, what they cover, and which params to pass.

Jobs or Plans?

Both run your tests against sandboxes:

Choose Plans if you want coding agents to create and run tests for you, want steps before and after the tests, or want one named test run that developers, CI and agents all call the same way.
Choose Jobs if you want to run suites exactly the way your CI does and control the pod they run in: its image, resources, mounted secrets and service account.

Overview​

Let your coding agent write the plan​

Before you start​

1. Send the routing key with every request​

2. Write tests into the plan​

3. Wrap tests from your repository​

4. Add steps before and after the tests​

5. Fail the plan when tests fail​

Tests from your repository​

Keep the report when tests fail​

Tests written into the plan​

6. Clone private repositories​

7. Run plans from CI​

8. Let coding agents run your tests​

Jobs or Plans?​

See also​