Skip to main content

Run Tests with Jobs

Overview​

Signadot runs your tests against sandboxes in two ways. Pick the one that fits how you want to work:

  • Jobs: you run your test suite the way your CI does. You write the script, choose the container image, and submit it from the CLI or a CI pipeline.
  • Plans: your coding agent sets up and runs the tests for you. It turns your tests into a reusable plan, runs the plan to check its own changes, and CI can run the same plan by name. See Run Tests with Plans.

This quickstart covers Jobs. A Job runs your script in a pod in your cluster, with the sandbox's routing key in its environment. The script clones your repository and runs your normal test command, and the routing key sends the test traffic to the services you changed while everything else stays on the baseline.

You'll take the Playwright suite that already lives in the HotROD repository and run it, unchanged, against a sandbox:

  • Create a sandbox with a backend change
  • Create a Job Runner Group that provides Playwright-ready pods
  • Submit a Job that clones the repository and runs npx playwright test
  • Download the test report and confirm the tests exercised the sandbox

Setup​

Before you begin, you'll need:

  • Signadot account (sign up here)
  • Signadot CLI (installation instructions)
  • A Kubernetes cluster with the Signadot Operator installed. Local clusters such as minikube, k3s or Docker Desktop work fine.
  • The HotROD demo application, installed with the overlay that matches your cluster: 
    kubectl create ns hotrod --dry-run=client -o yaml | kubectl apply -f -
    kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'

The existing test suite​

HotROD is a ride-sharing application with four microservices: frontend, driver, route and location. Its repository ships a small Playwright suite, the kind of end-to-end suite many teams already run in CI:

  • playwright-tests/basic.spec.ts requests a ride and checks the route and driver details on the page.
  • playwright.config.ts points the tests at the in-cluster frontend and adds a header to every request.

The header is the only Signadot-specific part of the suite:

playwright.config.ts (excerpt)
use: {
  baseURL: `http://frontend.${process.env.HOTROD_NAMESPACE}:8080`,
  extraHTTPHeaders: {
    baggage: `sd-routing-key=${process.env.SIGNADOT_ROUTING_KEY}`,
  },
},

Signadot sets SIGNADOT_ROUTING_KEY when the tests run against a sandbox. The header carries it to the cluster, which routes each request to the sandboxed version of a service when there is one, and to the baseline otherwise. If your own suite doesn't send a routing key yet, adding a header like this is usually the only change it needs.

Step 1: Create a sandbox​

Create a sandbox that runs a changed version of the driver service. The change (signadot/hotrod#260) adds an SD- prefix to driver license plates. Save this as driver-plates.yaml:

name: driver-plates
spec:
  description: Adding SD- prefix to driver plates
  cluster: "@{cluster}"
  forks:
    - forkOf:
        kind: Deployment
        namespace: hotrod
        name: driver
      customizations:
        images:
          - image: signadot/hotrod:f788b05bca80429425b814da62b2f17c3c564089-linux-amd64
            container: hotrod

Apply it, replacing <cluster> with the name of your cluster from the Clusters page:

signadot sandbox apply -f ./driver-plates.yaml --set cluster=<cluster>

Step 2: Create a Job Runner Group​

A Job Runner Group is a set of pods in your cluster that run Jobs. You choose the image, so the pods already have the tools your tests need. Here that's the official Playwright image, at the same version the HotROD suite pins.

Job runner pods live in a namespace that you create:

kubectl create ns signadot-tests

Save this as playwright-runner.yaml. It's the same spec as .signadot/testing/playwright-runner.yaml in the HotROD repository:

name: playwright
spec:
  cluster: "@{cluster}"
  labels:
    env: "@{env}"
  namespace: signadot-tests
  jobTimeout: 30m
  image: mcr.microsoft.com/playwright:v1.45.1-jammy
  scaling:
    manual:
      desiredPods: 1

Apply it and wait for the runner pod to become ready:

signadot jobrunnergroup apply -f ./playwright-runner.yaml --set cluster=<cluster> --set env=quickstart
signadot jobrunnergroup get playwright
Name:             playwright
Status:           1/1 pods ready
Dashboard page:   https://app.signadot.com/testing/runner-groups/playwright

desiredPods sets how many Jobs can run at the same time. See the Job Runner Group specification for every field.

Step 3: Run the suite as a Job​

The Job spec holds the script to run and the sandbox to run it against. Save this as playwright-job.yaml. It matches .signadot/testing/demo-playwright-job.yaml in the HotROD repository:

spec:
  namePrefix: hotrod-playwright-e2e
  runnerGroup: playwright
  script: |
    #!/bin/bash
    set -e

    # Clone the repository at the branch under test
    git clone --single-branch -b "@{branch}" \
      https://github.com/signadot/hotrod.git
    cd hotrod

    # Run the existing suite with its own config
    export HOTROD_NAMESPACE="@{namespace}"
    export CI=true
    npm ci
    set +e
    npm run e2e:playwright --spec playwright-tests/basic.spec.ts
    E2E_EXIT_CODE=$?
    set -e

    tar czf playwright-report.tar.gz playwright-report
    exit $E2E_EXIT_CODE
  routingContext:
    sandbox: "@{sandbox}"
  uploadArtifact:
    - path: hotrod/playwright-report/index.html
    - path: hotrod/playwright-report.tar.gz

A few things to notice:

  • The tests come from Git at run time. The runner image only provides the toolchain, so changing a test means pushing a commit, not rebuilding an image.
  • The test command is the one the repository already uses (npm run e2e:playwright), and it reads its own playwright.config.ts.
  • routingContext.sandbox tells Signadot which sandbox the Job targets. Signadot injects SIGNADOT_ROUTING_KEY and SIGNADOT_SANDBOX_NAME into the script's environment.
  • uploadArtifact keeps the HTML report after the pod is gone.
  • The script's exit code is the Job's result. A failing test fails the Job.

Submit the Job and stream its output:

signadot job submit -f ./playwright-job.yaml \
  --set branch=main \
  --set namespace=hotrod \
  --set sandbox=driver-plates \
  --attach
Job hotrod-playwright-e2e-ag7ycsa queued on Job Runner Group: playwright
Dashboard page: https://app.signadot.com/testing/jobs/hotrod-playwright-e2e-ag7ycsa
Cloning signadot repo
Cloning into 'hotrod'...
> e2e:playwright
> playwright test playwright-tests/basic.spec.ts
Running 2 tests using 1 worker
  2 passed (6.3s)

With --attach, signadot job submit waits for the Job to finish and exits non-zero when it fails, so the same command works as a check in CI.

Step 4: Inspect the results​

List the Job's status and artifacts, using the Job name from the previous output:

signadot job get hotrod-playwright-e2e-ag7ycsa
Job Name:           hotrod-playwright-e2e-ag7ycsa
Job Runner Group:   playwright
Status:             Succeeded
Environment:        sandbox=driver-plates

Artifacts
PATH                                  SIZE
hotrod/playwright-report/index.html   437.7 kB
hotrod/playwright-report.tar.gz       3.6 MB
@stderr                               25 B
@stdout                               224 B

Download and open the report:

signadot artifact download --job hotrod-playwright-e2e-ag7ycsa hotrod/playwright-report.tar.gz
tar xzf playwright-report.tar.gz
npx playwright show-report playwright-report

HotROD's config records a trace and a video for every test. Open the check driver series test and step through its trace: the driver plates on the page now start with SD-. The tests didn't change, but the routing key sent their requests to the driver service in your sandbox instead of the baseline.

The Jobs page in the dashboard shows the same run, with its logs and artifacts.

Cleanup​

signadot sandbox delete driver-plates
signadot jobrunnergroup delete playwright
kubectl delete ns signadot-tests

Conclusion​

You ran an existing end-to-end suite against a sandbox without changing the tests: a Job Runner Group supplied the toolchain, and a Job cloned the repository, ran the suite's own command and kept the report. The same Job spec works for any suite your CI already runs, such as Cypress, Postman/Newman, pytest or Go tests, as long as the runner image has the tools it needs and the tests send the routing key.

Next steps: