Skip to main content

Catch Security Bugs Before Merge with ZAP and Signadot Sandboxes

Static scanners read code. Some security bugs only show up when the code runs: a query that turns injectable once real input reaches it, or an error page that leaks database details. Catching those takes a running target, and in a microservices system that usually means a shared staging environment, after the change is merged, where a finding could come from any of the changes deployed together.

This tutorial runs ZAP, the most widely used open source dynamic application security testing (DAST) scanner, against a single change before merge. The changed service runs in a Signadot sandbox, and everything else it talks to comes from the cluster's baseline. ZAP runs in the cluster as a Signadot Job, which adds the sandbox's routing key to every request ZAP sends, so its attacks reach your change and nothing else.

You'll make a change to HotROD's location service that adds search by name, and introduces a SQL injection along with it. ZAP catches it. You fix it and scan again.

ZAP's requests carry the sandbox's routing key, so they reach the location service with your change, running on your machine. Requests without the key, like the frontend's, reach the baseline location service. Both versions use the cluster's MySQL.

The changed service runs on your machine, so you don't build or push an image, and a fix is one restart away from its next scan. The same Job also scans a change deployed in the cluster, which is how it runs as a pull request check.

Estimated time: 20 minutes.

What ZAP detects​

ZAP's rules come in two kinds. Passive rules only inspect the responses to ordinary requests, so they're safe to run against any sandbox. Active rules send attack payloads and watch how the application reacts.

CategoryExamplesRules
InjectionSQL injection, OS command injection, server-side template injection, XML external entities, CRLF injectionActive
Cross-site scriptingReflected and stored XSSActive
File accessPath traversal, remote file inclusionActive
Information disclosureDatabase errors, stack traces and debug messages in responses, server version numbers in headersPassive
Missing security headersContent Security Policy, anti-clickjacking, X-Content-Type-Options, Strict-Transport-SecurityPassive
Weak cookie settingsCookies without the HttpOnly, Secure or SameSite attributesPassive
Cross-origin misconfigurationOverly permissive CORS headersPassive

Each finding has a risk level (High, Medium, Low or Informational), the request that triggered it, and a suggested fix. ZAP's Docker image ships three packaged scans built on these rules:

ScanWhat it doesUse it for
Baseline (zap-baseline.py)Crawls the target and runs passive rules onlyWeb UIs, and any sandbox whose data isn't isolated
API (zap-api-scan.py)Imports an OpenAPI, SOAP or GraphQL definition and runs active rules against each operationAPIs. It skips browser-only checks such as XSS
Full (zap-full-scan.py)Crawls the target and runs all active rulesWeb apps, when the sandbox's data is isolated

This tutorial uses the API scan.

Before you begin​

You need:

  • A Kubernetes cluster with the Signadot Operator installed.
  • The Signadot CLI, logged in with signadot auth login. Creating a job runner group requires a Signadot admin.
  • kubectl pointed at that cluster, plus git, jq and Go 1.21 or later. Go downloads the toolchain version HotROD asks for.

This tutorial was tested with Signadot CLI v1.8.0, Operator 1.4.0 and ZAP 2.17.0.

1. Install HotROD​

Skip this step if HotROD already runs in the hotrod namespace. The files in this tutorial assume that namespace.

kubectl create namespace hotrod
kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'
kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m

2. Get the change​

The change is on HotROD's zap-location-search branch. Clone that branch and the examples repository side by side, then copy the Signadot files into HotROD:

git clone https://github.com/signadot/examples.git signadot-examples
git clone -b zap-location-search https://github.com/signadot/hotrod.git
cd hotrod
cp -R ../signadot-examples/zap-security-tutorial/security .signadot/

The change adds an optional name filter to GET /locations. Here's the core of the commit, in services/location/database.go:

query := "SELECT id, name, coordinates FROM locations"
if name != "" {
    query += fmt.Sprintf(" WHERE name LIKE '%%%s%%'", name)
}
rows, err := d.db.Query(query)

It returns the right rows for normal input, so it could pass a quick review.

The files you copied into .signadot/security/:

FileWhat it does
sandbox.yamlA sandbox that routes requests for the location service to port 8081 on your machine
zap-runner.yamlA job runner group that runs Jobs in ZAP's official image
zap-api-scan.yamlThe scan: a ZAP API scan of GET /locations, sent with the sandbox's routing key

3. Run the change in a sandbox​

sandbox.yaml is a local sandbox: requests that carry its routing key and are bound for the location service go to port 8081 on your machine.

name: location-search
spec:
  cluster: "@{cluster}"
  description: location service with name search, running on my machine
  local:
    - name: location
      from:
        kind: Deployment
        namespace: hotrod
        name: location
      mappings:
        - port: 8081
          toLocal: localhost:8081

Connect your machine to the cluster. signadot local connect asks for your password, because it updates /etc/hosts and local networking:

signadot local connect

Create the sandbox. Set CLUSTER to your cluster's name from signadot cluster list:

export CLUSTER=my-cluster
signadot sandbox apply -f .signadot/security/sandbox.yaml --set cluster=$CLUSTER

In a second terminal, in the hotrod directory, start the location service with your change. get-env loads the baseline's environment, so the service connects to the cluster's MySQL and Redis:

eval "$(signadot sandbox get-env location-search)"
go run ./cmd/hotrod location

Leave it running. Back in the first terminal, send the same request twice: once as a normal request, and once with the sandbox's routing key:

ROUTING_KEY=$(signadot sandbox get location-search -o json | jq -r .routingKey)
curl "http://location.hotrod:8081/locations?name=Coffee"
curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name=Coffee"

The first request reaches the baseline, which ignores name and returns all five locations. The second reaches the code on your machine and returns only Amazing Coffee Roasters. Both requests go to the same address. The routing key decides which version handles them.

4. Create the ZAP job runner group​

A job runner group keeps a pod running ZAP's image in your cluster, ready to run scan Jobs. zap-runner.yaml:

name: zap
spec:
  cluster: "@{cluster}"
  namespace: signadot-tests
  image: ghcr.io/zaproxy/zaproxy:stable
  jobTimeout: 15m
  scaling:
    manual:
      desiredPods: 1

Create its namespace, then the runner group:

kubectl create namespace signadot-tests
signadot jobrunnergroup apply -f .signadot/security/zap-runner.yaml --set cluster=$CLUSTER

The first time, the pod pulls ZAP's image, which takes a minute or two. Run signadot jobrunnergroup get zap until it shows Status: 1/1 pods ready.

5. Scan the sandbox​

zap-api-scan.yaml is the scan Job:

spec:
  namePrefix: zap-api-scan
  runnerGroup: zap
  # Ties the Job to the sandbox, and re-queues it if the sandbox isn't ready.
  routingContext:
    sandbox: "@{sandbox}"
  # Adds the sandbox's routing key to every plain HTTP request the Job sends.
  # ZAP targets the baseline service address, and Signadot routes its requests
  # to the sandboxed workload.
  trafficManager:
    injectRoutingKey: auto
  script: |
    #!/bin/bash
    set -euo pipefail
    mkdir -p /zap/wrk && cd /zap/wrk

    # Scan only what the change touches: GET /locations and its new
    # `name` parameter. ZAP sends attack payloads to every operation listed
    # here, so write operations are left out.
    cat > openapi.yaml <<'EOF'
    openapi: 3.0.3
    info:
      title: HotROD location service
      version: "1.0"
    servers:
      - url: @{target}
    paths:
      /locations:
        get:
          summary: List locations, optionally filtered by name
          parameters:
            - name: name
              in: query
              schema:
                type: string
                example: Coffee
          responses:
            "200":
              description: Matching locations
    EOF

    # -I: low-risk warnings (like missing headers) don't fail the scan.
    zap-api-scan.py -t /zap/wrk/openapi.yaml -f openapi -I \
      -r zap-report.html -J zap-report.json

    # Fail the Job if ZAP found anything High risk.
    HIGH=$(jq '[.site[].alerts[] | select(.riskcode == "3")] | length' zap-report.json)
    if [ "$HIGH" -gt 0 ]; then
      echo
      echo "ZAP found $HIGH High-risk issue(s):"
      jq -r '.site[].alerts[] | select(.riskcode == "3")
        | "  \(.name) [\(.pluginid)]: param \(.instances[0].param), attack \(.instances[0].attack)"' zap-report.json
      exit 1
    fi
    echo "No High-risk issues found."
  uploadArtifact:
    - path: /zap/wrk/zap-report.html
    - path: /zap/wrk/zap-report.json

It does three things:

  • It scans only what changed. The OpenAPI definition lists only GET /locations and its name parameter. ZAP's API scan sends attack payloads to each operation in the definition, so the scan covers exactly what changed and leaves out operations that write data.
  • It reaches the change through the routing key. routingContext ties the Job to the sandbox, and with injectRoutingKey: auto a proxy next to the Job adds the routing headers to its outbound plain HTTP traffic. ZAP needs no extra configuration, and its own header settings stay free for your app's credentials if the scan needs to log in.
  • It fails on High-risk findings only. Low-risk findings, like missing security headers, show up in the report but don't fail the Job.

Run the scan:

signadot job submit -f .signadot/security/zap-api-scan.yaml \
  --set sandbox=location-search \
  --set target=http://location.hotrod:8081 \
  --attach

target is the baseline service's address. ZAP never needs to know where your change runs. After about 30 seconds, the Job fails:

WARN-NEW: SQL Injection [40018] x 1
	http://location.hotrod:8081/locations?name=%27 (500 Internal Server Error)
...
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 4	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 116

ZAP found 1 High-risk issue(s):
  SQL Injection [40018]: param name, attack '
Error: exit status 1

ZAP sent a single quote as name, and the service answered with a server error. A harmless value didn't cause one. Send the same request yourself to see why:

curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name='"
Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''' at line 1

The quote ended the SQL string early. Whatever follows it in name runs as SQL, and the error message gives away the database. The full report has a description and a fix for each finding. Download it with the Job name from the output above:

signadot artifact download --job <job-name> /zap/wrk/zap-report.html

You can also open both reports from the Job's page in the Signadot dashboard.

6. Fix it and scan again​

The fix passes name to the database as a query parameter instead of pasting it into the SQL. It's one commit on the zap-location-search-fix branch:

query := "SELECT id, name, coordinates FROM locations"
var args []any
if name != "" {
    query += " WHERE name LIKE ?"
    args = append(args, "%"+name+"%")
}
rows, err := d.db.Query(query, args...)

Switch to that branch:

git checkout zap-location-search-fix

In the second terminal, stop the service with Ctrl+C and start it again:

go run ./cmd/hotrod location

Scan again:

signadot job submit -f .signadot/security/zap-api-scan.yaml \
  --set sandbox=location-search \
  --set target=http://location.hotrod:8081 \
  --attach

This time the Job passes:

FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 2	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 117
No High-risk issues found.

The single quote is now just a character to search for. It finds Rachel's Floral Designs:

curl -H "baggage: sd-routing-key=$ROUTING_KEY" "http://location.hotrod:8081/locations?name='"

The change is ready for review, and the passing scan is evidence that it's safe to merge.

Scan a change deployed in the cluster​

The sandbox decides where the change runs, so the Job works the same way when the change is deployed in the cluster. Build an image from your branch, then create a sandbox that forks the location Deployment with that image in place of local:

name: location-search
spec:
  cluster: "@{cluster}"
  forks:
    - forkOf:
        kind: Deployment
        namespace: hotrod
        name: location
      customizations:
        images:
          - image: "@{image}"

Then run the same signadot job submit command.

Run the scan as a pull request check​

In CI, the pipeline builds an image from the pull request and creates a sandbox that forks the service's Deployment with that image, like the one above. Then it submits the same Job with --attach. If the scan finds a High-risk issue, the Job fails and so does the PR check, and both reports stay attached to the Job for the reviewer. See CI/CD integrations for running Signadot Jobs from GitHub Actions, GitLab, Bitbucket and Jenkins.

A coding agent can run the same loop on its own changes before it opens a pull request. See Run it in the agent loop and in CI.

Keep active scans contained​

An API scan sends real attack payloads. The sandboxed service processes them, and its calls to shared services and databases are real. This tutorial keeps that safe by listing only a read-only operation in the OpenAPI definition. For your own services:

  • List only the operations the change touches in the scan's OpenAPI definition.
  • Before scanning operations that write data, give the sandbox its own database with a resource plugin, so injected rows never land in shared data.
  • To check a web UI without sending attacks, use ZAP's passive baseline scan, zap-baseline.py, in the same Job.

See Keep active scans contained for more.

Clean up​

Stop the location service in the second terminal with Ctrl+C. Then run:

signadot sandbox delete location-search
signadot jobrunnergroup delete zap
signadot local disconnect

To remove HotROD, run kubectl delete namespace hotrod.

Troubleshooting​

What you seeWhat to check
namespace "signadot-tests" does not exist when creating the runner groupRun kubectl create namespace signadot-tests first.
The Job stays queuedThe runner pod is still pulling ZAP's image. Check kubectl -n signadot-tests get pods.
curl: Could not resolve host: location.hotrodRun signadot local status, and signadot local connect if it isn't connected.
The scan passes before the fix, or the routed curl returns all five locationsRequests aren't reaching your machine. Check that go run ./cmd/hotrod location is running and that signadot sandbox get location-search shows the sandbox as ready.

Frequently asked questions​

How do I run a ZAP scan against a single microservice change in Kubernetes?​

Run the changed service in a Signadot sandbox and run ZAP as a Signadot Job in the cluster. The Job's routingContext ties it to the sandbox, and trafficManager.injectRoutingKey: auto adds the sandbox's routing key to every plain HTTP request ZAP sends. ZAP targets the baseline service's address, and Signadot routes its requests to the changed workload, so every finding belongs to that change.

Which ZAP scan should I run against a sandbox?​

Use the API scan (zap-api-scan.py) for APIs, with an OpenAPI definition that lists only the operations the change touches. Use the baseline scan (zap-baseline.py), which runs passive rules only, for web UIs or any sandbox whose data isn't isolated. Use the full scan (zap-full-scan.py) for web apps when the sandbox has its own data.

Is it safe to run an active ZAP scan against a sandbox?​

An active scan sends real attack payloads, and the sandboxed service's calls to shared services and databases are real. Keep it contained: list only the changed operations in the scan's OpenAPI definition, leave out operations that write data, and give the sandbox its own database with a resource plugin before scanning operations that do.

Can the ZAP scan run as a pull request check?​

Yes. In CI, create a sandbox that forks the service Deployment with the image built from the pull request, then submit the same Job with signadot job submit --attach. The Job fails when ZAP reports a High-risk finding, which fails the PR check, and the HTML and JSON reports stay attached to the Job for reviewers.

Can ZAP scan code running on my laptop?​

Yes. With a local sandbox and signadot local connect, requests that carry the sandbox routing key reach the service running on your machine. ZAP still runs in the cluster and targets the baseline address, so you fix the code, restart the service and scan again without building or pushing an image.

See also​