Skip to main content

Run Scalable Performance Tests with Grafana k6 and Signadot

Performance regressions slip past functional tests: every response is still correct, it just arrives later. This tutorial load tests a pull request's version of a service with Grafana k6 before merge. You deploy only the changed service as a Signadot sandbox, run k6 inside your Kubernetes cluster as a Signadot job, and run the same test against the baseline and the sandbox. A change that adds a 10-second delay to HotROD's location service still returns status 200 on every request, but its p95 latency crosses a 500 ms threshold and the job fails.

How it works​

Three Signadot resources do the work:

  • A sandbox forks the location Deployment and runs the pull request's image. Everything else, including MySQL and Redis, comes from the baseline.
  • A job runner group keeps pods running the grafana/k6 image in the cluster, ready to run jobs. Raise its pod count to run several k6 jobs at once.
  • A job runs a parameterized k6 script on one of those pods. Its routingContext names the sandbox. With trafficManager.injectRoutingKey: auto, a proxy next to k6 adds the sandbox's routing key to every request. With disabled, requests go to the baseline.

The diagram shows a k6 job's requests, all inside the cluster, reaching either version of location through the same Service address.

k6 runs next to the services it tests, so the latency it measures has no laptop-to-cluster network hop in it, and the baseline and the sandbox are measured under the same conditions.

Before you begin​

You need:

  • A Kubernetes cluster with the Signadot Operator installed.
  • The Signadot CLI, logged in with signadot auth login. Creating a job runner group requires a Signadot admin.
  • kubectl pointed at that cluster, plus git, Docker, Go 1.21 or later, and a container registry your cluster can pull from.

Set two variables for the commands that follow. CLUSTER is your cluster's name in Signadot (signadot cluster list), and REGISTRY is the registry you push images to:

export CLUSTER=my-cluster
export REGISTRY=registry.example.com/my-team

Step 1: Install HotROD​

HotROD is a ride-sharing demo made of several microservices. Clone it, because you'll build a changed location service from it, and install it in the hotrod namespace:

git clone https://github.com/signadot/hotrod.git
cd hotrod
kubectl create namespace hotrod
kubectl -n hotrod apply -k k8s/overlays/prod/devmesh
kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m

The devmesh overlay adds the Signadot DevMesh sidecar, which routes requests that carry a sandbox's routing key. If your cluster uses Istio, apply k8s/overlays/prod/istio instead.

Step 2: Add a delay to the location service​

The pull request in this tutorial makes GET /location slow. In services/location/server.go, add a 10-second sleep at the end of getLocation, just before it writes the response:

services/location/server.go
func (s *Server) getLocation(w http.ResponseWriter, r *http.Request) {
	// ... parse locationID, read the location, send a notification

	// Introduce a 10-second delay before responding
	time.Sleep(10 * time.Second)

	w.Header().Set("Content-Type", "application/json")
	w.Write(data)
}

Build a Linux binary and an image, and push it. This is the build command from HotROD's Makefile without its frontend step (the built frontend assets are committed). For Arm nodes, use GOARCH=arm64 and --platform linux/arm64.

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux/amd64/bin/hotrod ./cmd/hotrod
docker build --platform linux/amd64 -t $REGISTRY/hotrod:location-delay .
docker push $REGISTRY/hotrod:location-delay
Use a private repository for your own services

registry.example.com/my-team is a placeholder. When you build images of your own services this way, push them to a private repository and give the cluster pull access to it, for example with an image pull secret.

Step 3: Create a sandbox for the change​

The sandbox runs your image as a fork of the location Deployment, leaving the baseline untouched. Create a directory for this tutorial's Signadot files:

mkdir -p .signadot/k6

Save the sandbox spec. container: hotrod is the name of the container in HotROD's Deployments:

.signadot/k6/location-sbx.yaml
name: location-sbx
spec:
  description: location service with a 10-second response delay
  cluster: "@{cluster}"
  forks:
    - forkOf:
        kind: Deployment
        namespace: hotrod
        name: location
      customizations:
        images:
          - image: "@{image}"
            container: hotrod

Apply it:

signadot sandbox apply -f .signadot/k6/location-sbx.yaml \
  --set cluster=$CLUSTER \
  --set image=$REGISTRY/hotrod:location-delay

The CLI waits until the sandbox is ready and prints its routing key. If your registry needs credentials, add an imagePullSecrets custom patch to the fork.

Step 4: Create a k6 job runner group​

A job runner group is a set of pods in your cluster that run Signadot jobs. This one runs Grafana's official k6 image:

.signadot/k6/k6-runner.yaml
name: k6-perf-tests
spec:
  cluster: "@{cluster}"
  namespace: signadot
  image: grafana/k6:1.7.1
  jobTimeout: 30m
  scaling:
    manual:
      desiredPods: 1

namespace is where the runner pods run, jobTimeout caps each job's execution time, and desiredPods is the number of runner pods kept ready. Create the group:

signadot jobrunnergroup apply -f .signadot/k6/k6-runner.yaml --set cluster=$CLUSTER

The first pod pulls the k6 image. Run signadot jobrunnergroup get k6-perf-tests until it shows Status: 1/1 pods ready.

Step 5: Write the k6 script and the job​

Keep test logic and configuration apart: the script reads its target URL and virtual user (VU) counts from environment variables, so one script serves every service and environment. Its stages ramp to S1_VUS, then to S2_VUS, then down to S3_VUS, two minutes in all. Two checks record whether each response has status 200 and arrives in under 500 ms.

.signadot/k6/k6-script.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: Number(__ENV.S1_VUS) },
    { duration: '1m', target: Number(__ENV.S2_VUS) },
    { duration: '30s', target: Number(__ENV.S3_VUS) },
  ],
};

export default function () {
  const response = http.get(__ENV.URL); // target passed in by the job
  check(response, {
    'Status is 200': (r) => r.status === 200,
    'Response time < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

The job spec runs that script on the runner group. @{embed: k6-script.js} is a template embedding: the CLI copies the script file into the job when you submit it, and the job writes it back to a file for k6.

.signadot/k6/k6-job.yaml
spec:
  namePrefix: k6-location
  runnerGroup: k6-perf-tests
  routingContext:
    sandbox: "@{sandbox}"
  trafficManager:
    injectRoutingKey: "@{injectRoutingKey}"
  script: |
    #!/bin/sh
    cat > k6-script.js <<'EOF'
    @{embed: k6-script.js}
    EOF
    k6 run \
      -e URL="@{URL}" \
      -e S1_VUS="@{S1_VUS}" -e S2_VUS="@{S2_VUS}" -e S3_VUS="@{S3_VUS}" \
      --summary-export=/tmp/k6-summary.json \
      k6-script.js
  uploadArtifact:
    - path: /tmp/k6-summary.json

injectRoutingKey decides which version of location receives the load:

ValueWhat happensMeasures
disabledRequests leave the job without a routing key.The baseline
autoA proxy next to k6 adds the routing key of the sandbox in routingContext to each request.The sandbox

The proxy injects the key into plain HTTP and gRPC traffic only, not TLS. If k6 calls services over HTTPS, set the header in the script instead. Every job gets the sandbox's key in the SIGNADOT_ROUTING_KEY environment variable:

const params = { headers: { baggage: `sd-routing-key=${__ENV.SIGNADOT_ROUTING_KEY}` } };
const response = http.get(__ENV.URL, params);

uploadArtifact keeps k6's JSON summary with the job, for CI or later comparison.

Step 6: Run k6 against the baseline and the sandbox​

Run the job twice with the same load, changing only injectRoutingKey. --attach streams k6's output and waits for the job to finish. First the baseline:

signadot job submit -f .signadot/k6/k6-job.yaml \
  --set sandbox=location-sbx \
  --set injectRoutingKey=disabled \
  --set URL="http://location.hotrod.svc:8081/location?locationID=1" \
  --set S1_VUS=10 --set S2_VUS=20 --set S3_VUS=5 \
  --attach

k6's end-of-test summary for the baseline, trimmed to the lines that matter here:

     ✓ Status is 200
     ✓ Response time < 500ms

     checks.........................: 100.00% 2178 out of 2178
     http_req_duration..............: avg=303.94ms min=202.59ms med=303.49ms max=403.7ms  p(90)=344.41ms p(95)=357.3ms
       { expected_response:true }...: avg=303.94ms min=202.59ms med=303.49ms max=403.7ms  p(90)=344.41ms p(95)=357.3ms
     http_req_failed................: 0.00%   0 out of 1089
     http_reqs......................: 1089    8.97901/s
     iteration_duration.............: avg=1.3s     min=1.2s     med=1.3s     max=1.4s     p(90)=1.34s    p(95)=1.35s
     iterations.....................: 1089    8.97901/s
     vus_max........................: 20      min=20       max=20

running (2m01.3s), 00/20 VUs, 1089 complete and 0 interrupted iterations
default ✓ [ 100% ] 00/20 VUs  2m0s

Then the sandbox, with the routing key injected:

signadot job submit -f .signadot/k6/k6-job.yaml \
  --set sandbox=location-sbx \
  --set injectRoutingKey=auto \
  --set URL="http://location.hotrod.svc:8081/location?locationID=1" \
  --set S1_VUS=10 --set S2_VUS=20 --set S3_VUS=5 \
  --attach
     ✓ Status is 200
     ✗ Response time < 500ms
      ↳  0% — ✓ 0 / ✗ 135

     checks.........................: 50.00% 135 out of 270
     http_req_duration..............: avg=10.3s    min=10.2s    med=10.3s    max=10.39s   p(90)=10.34s   p(95)=10.35s
       { expected_response:true }...: avg=10.3s    min=10.2s    med=10.3s    max=10.39s   p(90)=10.34s   p(95)=10.35s
     http_req_failed................: 0.00%  0 out of 135
     http_reqs......................: 135    1.030529/s
     iteration_duration.............: avg=11.3s    min=11.21s   med=11.3s    max=11.39s   p(90)=11.34s   p(95)=11.35s
     iterations.....................: 135    1.030529/s
     vus_max........................: 20     min=20       max=20

running (2m11.0s), 00/20 VUs, 135 complete and 0 interrupted iterations
default ✓ [ 100% ] 00/20 VUs  2m0s

These summaries come from an earlier k6 release, which prints metrics as one list. Current k6 releases group the same metrics by category.

Side by side:

MetricBaselineSandbox
Response time < 500ms check100% passed0 of 135 passed
http_req_duration average303.94 ms10.3 s
http_req_duration p95357.3 ms10.35 s
http_req_failed0.00%0.00%
Iterations (requests per second)1,089 (8.98/s)135 (1.03/s)

The change doesn't break a single request: http_req_failed is 0% and every response is a 200. It makes them slow. The baseline's roughly 300 ms comes from HotROD simulating a database call; the sandbox adds the 10-second sleep on top. Because each virtual user waits for its response before the next iteration, the same 20 VUs complete 88% fewer iterations.

Yet both jobs succeeded. A failed check is recorded in the summary, but it doesn't fail the run. The next step makes it fail.

Step 7: Fail the test with k6 thresholds (p95 and error rate)​

Thresholds turn k6 metrics into pass/fail criteria for the whole run. Add them to options in k6-script.js:

.signadot/k6/k6-script.js
export const options = {
  stages: [
    // ... unchanged
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% of requests finish in under 500 ms
    http_req_failed: ['rate<0.01'],   // fewer than 1% of requests fail
  },
};
  • http_req_duration: ['p(95)<500'] applies to a Trend metric, the time each request takes. p(95) is the 95th percentile, so the threshold passes when 95% of requests finish in under 500 ms. avg, med, min, max and other percentiles such as p(99) work the same way.
  • http_req_failed: ['rate<0.01'] applies to a Rate metric, the share of requests that failed. By default k6 counts network errors and responses outside the 200 to 399 status range as failed. Keep this gate even when latency is what you care about: a service that fails fast can pass a latency threshold.

At the end of the run, k6 marks each threshold as passed or crossed in its summary. If any threshold is crossed, k6 exits with code 99. The job script ends with k6 run, so the job fails, and signadot job submit --attach exits with a non-zero code.

Submit both jobs again with the commands from Step 6. The baseline, whose p95 was 357.3 ms, passes. The sandbox, whose p95 was 10.35 s, crosses the http_req_duration threshold and its job fails, which is what a pull request check needs.

Two refinements:

  • To fail on checks, add a threshold on the checks metric, such as checks: ['rate>0.99'].
  • To stop a run as soon as a threshold is crossed, use the object form: http_req_duration: [{ threshold: 'p(95)<500', abortOnFail: true }].

Shared clusters are noisier than dedicated load-test environments. Leave headroom in thresholds, and when a run fails, compare it with a baseline run from the same time rather than with a fixed number alone.

Step 8: Scale k6 across pods (optional)​

A single k6 pod is limited by its CPU and memory. To generate more load, spread the test across pods: set desiredPods: 3 in k6-runner.yaml and apply it again, then submit three jobs at once:

for i in 1 2 3; do
  signadot job submit -f .signadot/k6/k6-job.yaml \
    --set sandbox=location-sbx \
    --set injectRoutingKey=auto \
    --set URL="http://location.hotrod.svc:8081/location?locationID=1" \
    --set S1_VUS=10 --set S2_VUS=20 --set S3_VUS=5 \
    --wait &
done
wait

Each job runs on its own runner pod, and jobs beyond the number of idle pods wait in the queue. Every job is an independent k6 run with its own summary and threshold results, so the total load is the sum of the jobs. Read each job's output with signadot logs --job <job-name>, or download its summary with signadot artifact download --job <job-name> /tmp/k6-summary.json. To combine metrics across pods, send every run to a shared metrics backend with k6's --out option.

Clean up​

signadot sandbox delete location-sbx
signadot jobrunnergroup delete k6-perf-tests

To remove HotROD, run kubectl delete namespace hotrod.

Run it on every pull request​

In CI, the pipeline builds an image from the pull request and creates a sandbox that forks the changed Deployment with it, like location-sbx above. It then submits the k6 job with injectRoutingKey=auto and --attach. When a threshold is crossed, the job fails and so does the check. Running the baseline job in the same pipeline gives reviewers a reference measured at the same time, and both JSON summaries stay attached to their jobs. See CI/CD integrations for running Signadot jobs from GitHub Actions, GitLab, Bitbucket and Jenkins.

Run the same check as a Signadot Plan​

A Signadot Plan is a reusable, tagged validation workflow that a person or a coding agent reruns by name against any sandbox. The k6 action in Signadot's action catalog runs a k6 script on your cluster's Plan Runner Group, so there's no job runner group or job spec to maintain. It differs from the job above in three ways:

  • The step's routingContext gives the script SIGNADOT_ROUTING_KEY, and the script adds the routing headers itself.
  • The target_url input becomes __ENV.TARGET_URL, and options passes k6 flags such as --vus 20 --duration 1m.
  • The action exits with k6's exit code, so a crossed threshold fails the step and the plan. Its summary output holds k6's JSON summary, such as summary.metrics.http_req_duration['p(95)'].

You rarely write the plan by hand: the signadot-plan agent skill drafts, tests and tags one from a one-sentence request, and the plan spec reference covers the format. Plans need Signadot CLI v1.6.0 or later, Operator v1.3.1 or later, and a Plan Runner Group enabled on the cluster. For a worked example with video, see Catch performance regressions with k6 and Signadot Plans.

Frequently asked questions​

How do I set a p95 latency threshold in k6?​

Add a thresholds entry to the script's options: http_req_duration: ['p(95)<500'] requires the 95th percentile of request duration to stay under 500 milliseconds. Pair it with http_req_failed: ['rate<0.01'] so a run where requests fail quickly can't pass. When any threshold is crossed, k6 marks it in the end-of-test summary and exits with code 99, which fails the Signadot job that runs it.

What is the difference between k6 checks and thresholds?​

A check validates a single response, such as status is 200, and k6 reports the share of checks that passed, but failed checks don't change k6's exit code. A threshold is a pass/fail criterion on a metric across the whole run, such as p(95)<500 on http_req_duration, and a crossed threshold makes k6 exit with a non-zero code. To fail a run on checks, add a threshold on the checks metric, for example checks: ['rate>0.99'].

How do I run k6 load tests inside Kubernetes?​

Create a Signadot job runner group that uses the grafana/k6 image, then submit k6 scripts to it as Signadot jobs with signadot job submit. The runner pods live in your cluster, so k6 reaches services by their in-cluster DNS names and measures latency without a tunnel or ingress in the path. Setting the job's routingContext to a sandbox and trafficManager.injectRoutingKey to auto sends the load to the sandboxed version of a service instead of the baseline.

Can k6 load tests run distributed across several pods?​

Yes. Set scaling.manual.desiredPods on the job runner group to keep several k6 runner pods ready, then submit one job per pod at the same time; jobs beyond the number of idle pods wait in the queue. Each job is an independent k6 run with its own end-of-test summary, so the total load is the sum of the jobs. To combine metrics across pods, send each run to a shared metrics backend with k6's --out option.

How do I performance test a pull request without a dedicated environment?​

Deploy only the service the pull request changes as a Signadot sandbox in your existing cluster, sharing the baseline's other services and databases. Run the same k6 job twice, once without the sandbox's routing key to measure the baseline and once with it to measure the change, and compare the two summaries. Both runs use the same cluster and dependencies at the same time, so a difference between them points to the code change rather than the environment.

See also​