Catch Risky Runtime Behavior Before Merge with Falco and Signadot Sandboxes

Scanners test what an endpoint returns. Some problems only show up in what the code does while it runs: a new dependency that opens a connection to the internet when it's first used, a handler that passes input to a shell, code that reads a file it shouldn't. A diff review can miss these, and so can the unit tests, because nobody writes a test for behavior they don't expect.

This tutorial runs Falco, the open source runtime security engine for Kubernetes and a graduated CNCF project, against a single change before merge. Falco watches system calls on every node in the cluster and raises an alert when a container does something one of its rules describes. The changed service runs in a Signadot sandbox, and everything else it talks to comes from the cluster's baseline. Signadot labels every sandbox pod with its routing key, so you can scope Falco's rules to sandbox pods: they fire for your change and stay quiet for normal traffic.

You'll deploy a change to HotROD's route service that computes an ETA with a small helper package, the way a developer might reach for a third-party library. On first use, that helper runs a shell command that reaches a host outside the cluster: a dependency phoning home. A ride dispatch flows from frontend through Kafka and driver to route. Falco catches the shell and the outbound connection for the sandboxed route only, and a Signadot Job turns that into a pass/fail check. You fix the change and the same Job passes.

The check Job sends a dispatch with the sandbox's routing key to the baseline frontend. It flows through Kafka to driver, which calls the sandboxed route because the request carries the key. Falco, running on every node, watches the sandboxed route pod's system calls, and the Job reads the alerts Falco raised for this sandbox.

Estimated time: 20 minutes.

What Falco detects

Falco's rules describe what a process does, not what a service returns. These are the behaviors most worth catching in a sandbox:

Behavior	What it can point to	Rules
Outbound connections to addresses outside the cluster	A dependency phoning home, data exfiltration, server-side request forgery	A sandbox-scoped rule, installed in this tutorial
Shells started by the application	Command injection, or a handler or library shelling out	A sandbox-scoped rule, and Falco's default `Run shell untrusted`
Reads of sensitive files	Code reading credentials or private keys it has no reason to touch	Falco's default `Read sensitive file untrusted` and `Search Private Keys or Passwords`
New binaries run at runtime	A downloaded payload, or an executable written after the container started	Falco's default `Drop and execute new binary in container`
Calls to the Kubernetes API	Code that talks to the API server when it shouldn't	Falco's default `Contact K8S API Server From Container`

Falco's default rules apply to every pod in the cluster. Rules scoped to sandbox pods report behavior that's normal elsewhere but unexpected from a change, such as any connection that leaves the cluster.

Before you begin

You need:

A Kubernetes cluster with the Signadot Operator v1.4.0 or later installed.
The Signadot CLI, logged in with signadot auth login. Creating a job runner group requires a Signadot admin.
kubectl pointed at that cluster, plus Helm and git.
A node whose kernel exposes BTF (/sys/kernel/btf/vmlinux), so Falco's modern eBPF driver works. Most recent distributions, and Docker Desktop's Kubernetes, have it.

This tutorial was tested with Signadot CLI v1.8.0, Operator 1.4.0 and Falco 0.45.0 (chart 9.2.0) on a Docker Desktop cluster.

1. Install HotROD

Skip this step if HotROD already runs in the hotrod namespace. The files in this tutorial assume that namespace.

kubectl create namespace hotrod
kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'
kubectl -n hotrod wait --for=condition=available deployment --all --timeout=5m

2. Get the tutorial files

git clone https://github.com/signadot/examples.git signadot-examples
cd signadot-examples/falco-security-tutorial

The change itself lives on HotROD's falco-route-eta branch and is already built into the public image signadot/hotrod:falco-route-eta, so you don't need to clone or build HotROD. Here's the core of the commit, in services/route/server.go:

// Estimate the ETA from the trip's endpoints using the trafficeta helper,
// instead of returning a random number.
eta := trafficeta.Estimate(req.From, req.To)

trafficeta.Estimate looks like an ordinary helper. The behavior that matters is one function call deep, in services/route/trafficeta/trafficeta.go, and only happens at runtime:

// fetchTrafficFactor simulates the dependency contacting its backend. It runs a
// shell that reaches an outside host...
func fetchTrafficFactor() {
	cmd := exec.Command("sh", "-c", "wget -q -T 2 -O - "+phoneHomeURL+" || true")
	_ = cmd.Run()
}

The ETA comes out right, so the change could pass a review and its unit tests. The files in security/:

File	What it does
`falco-values.yaml`	Helm values that install Falco and add two rules scoped to sandbox pods
`sandbox.yaml`	A sandbox that forks `route` with your image
`falco-reader-rbac.yaml`	A service account that can read Falco's alert logs
`falco-runner.yaml`	A job runner group that runs the check Job, using that service account
`falco-check.yaml`	The Job: send one dispatch, then pass or fail on Falco's alerts for the sandbox

3. Install Falco with sandbox-scoped rules

Falco sees system calls, not HTTP headers, so the routing key doesn't reach it the way it reaches a scanner. Instead, you write rules that only match sandbox pods. The Signadot Operator labels every forked pod with signadot.com/sandbox-routing-key, so no extra setup is needed. The rules in security/falco-values.yaml reuse macros and lists from Falco's default ruleset (outbound, spawned_process, container, shell_binaries) and fire only for pods that carry that label:

driver:
  kind: modern_ebpf

falco:
  json_output: true

customRules:
  signadot_rules.yaml: |-
    - macro: signadot_sandbox_pod
      condition: (k8s.pod.label[signadot.com/sandbox-routing-key] != "")

    - rule: Sandbox Outbound Connection Outside Cluster
      desc: A sandboxed workload connected to an address outside the cluster.
      condition: outbound and container and signadot_sandbox_pod
      output: >
        Sandbox pod connected outside the cluster
        (routing_key=%k8s.pod.label[signadot.com/sandbox-routing-key]
        pod=%k8s.pod.name ns=%k8s.ns.name connection=%fd.name
        proc=%proc.name cmdline=%proc.cmdline
        container=%container.name image=%container.image.repository)
      priority: WARNING
      tags: [signadot, network]

    - rule: Sandbox Shell Spawned
      desc: A process in a sandboxed workload started a shell.
      condition: >
        spawned_process and container and signadot_sandbox_pod
        and proc.name in (shell_binaries)
      output: >
        Shell spawned in sandbox pod
        (routing_key=%k8s.pod.label[signadot.com/sandbox-routing-key]
        pod=%k8s.pod.name ns=%k8s.ns.name shell=%proc.name parent=%proc.pname
        cmdline=%proc.cmdline container=%container.name
        image=%container.image.repository)
      priority: WARNING
      tags: [signadot, process]

json_output: true makes Falco write each alert as a JSON line, which the check Job reads later. Install Falco, or upgrade an existing release:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm upgrade --install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  -f security/falco-values.yaml --wait

Wait until the Falco pod is 2/2 Running:

kubectl -n falco rollout status ds/falco

Baseline pods don't carry the signadot.com/sandbox-routing-key label, so these rules stay quiet for normal traffic and fire only for the code under test. Falco's own default rules still apply to every pod, as usual.

4. Run the change in a sandbox

security/sandbox.yaml forks the route Deployment with your image. Nothing else is needed: the Operator labels the fork's pods with the sandbox's routing key, which the rules above already match.

name: "@{sandbox}"
spec:
  cluster: "@{cluster}"
  description: route service running the trafficeta change, for the Falco tutorial
  forks:
  - forkOf:
      kind: Deployment
      namespace: hotrod
      name: route
    customizations:
      images:
      - image: "@{image}"

Create the sandbox. Set CLUSTER to your cluster's name from signadot cluster list:

export CLUSTER=my-cluster
signadot sandbox apply -f security/sandbox.yaml \
  --set cluster=$CLUSTER \
  --set sandbox=falco-route-eta \
  --set image=signadot/hotrod:falco-route-eta

Confirm the fork's pod came up with the routing-key label and your image:

kubectl -n hotrod get pods -l signadot.com/sandbox-routing-key \
  -o custom-columns=POD:.metadata.name,IMAGE:'.spec.containers[?(@.name=="hotrod")].image'

5. Set up the check Job

The Job sends one request and then reads Falco's alerts, so its runner needs read access to Falco's logs. security/falco-reader-rbac.yaml creates a falco-reader service account in signadot-tests and a Role, scoped to the falco namespace, that lets it read Falco's pods and their logs and nothing else:

kubectl create namespace signadot-tests
kubectl apply -f security/falco-reader-rbac.yaml

security/falco-runner.yaml is a job runner group that runs Jobs in an image with kubectl, jq and curl, as that service account:

name: falco-check
spec:
  cluster: "@{cluster}"
  namespace: signadot-tests
  image: alpine/k8s:1.31.1
  jobTimeout: 10m
  podTemplate:
    spec:
      serviceAccountName: falco-reader
  scaling:
    manual:
      desiredPods: 1

signadot jobrunnergroup apply -f security/falco-runner.yaml --set cluster=$CLUSTER

The first time, the pod pulls its image, which takes a minute or two. Run signadot jobrunnergroup get falco-check until it shows Status: 1/1 pods ready.

6. Run the check

security/falco-check.yaml is the check Job:

spec:
  namePrefix: falco-check
  runnerGroup: falco-check
  # Ties the Job to the sandbox, and re-queues it if the sandbox isn't ready.
  routingContext:
    sandbox: "@{sandbox}"
  # Adds the sandbox's routing key to the plain HTTP request below, so the
  # dispatch is handled by the sandboxed route service.
  trafficManager:
    injectRoutingKey: auto
  script: |
    #!/bin/bash
    set -euo pipefail

    # Signadot sets this to the sandbox's routing key for every Job. The operator
    # also stamps it onto the forked pods as the signadot.com/sandbox-routing-key
    # label, which is how the Falco rules and the filter below pick out this
    # sandbox's alerts.
    RK="$SIGNADOT_ROUTING_KEY"

    # Only count alerts generated from here on, so a re-run after a fix doesn't
    # pick up alerts from an earlier run of the same sandbox (the routing key is
    # the same across runs). Falco timestamps every alert in evt.time (epoch ns).
    START_EPOCH=$(date +%s)

    echo "Exercising sandbox '@{sandbox}' (routing key: $RK)"

    # Send one ride dispatch. injectRoutingKey adds the routing header, so the
    # request flows frontend -> kafka -> driver -> the sandboxed route service.
    curl -s -o /dev/null -w "dispatch HTTP %{http_code}\n" \
      -X POST "@{target}/dispatch" \
      -H "Content-Type: application/json" \
      -d '{"SessionID":123456,"RequestID":1,"PickupLocationID":123,"DropoffLocationID":567}'

    # The dispatch is handled asynchronously through Kafka, so give it a moment
    # to reach the route service.
    echo "Waiting for the request to reach the route service..."
    sleep 20

    # Read Falco's alerts and keep only the ones for this sandbox's pods (matched
    # by routing key) that happened during this run.
    echo "Reading Falco alerts for routing key '$RK'..."
    kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco \
      --since=5m --max-log-requests=50 --prefix=false 2>/dev/null \
      | grep '^{' \
      | jq -c --arg rk "$RK" --argjson start "$START_EPOCH" \
          'select(.output_fields["k8s.pod.label[signadot.com/sandbox-routing-key]"] == $rk
                  and ((.output_fields["evt.time"] / 1000000000) | floor) >= $start)
            | {rule, priority, output}' \
      > /tmp/falco-alerts.jsonl || true

    COUNT=$(wc -l < /tmp/falco-alerts.jsonl | tr -d ' ')
    if [ "$COUNT" -gt 0 ]; then
      echo
      echo "Falco raised $COUNT alert(s) for this sandbox:"
      jq -r '"  [\(.priority)] \(.rule)"' /tmp/falco-alerts.jsonl | sort | uniq -c
      echo
      echo "Details:"
      jq -r '.output' /tmp/falco-alerts.jsonl
      exit 1
    fi
    echo "No Falco alerts for this sandbox. The change is clean."
  uploadArtifact:
    - path: /tmp/falco-alerts.jsonl

routingContext ties the Job to the sandbox, and trafficManager.injectRoutingKey: auto adds the sandbox's routing key to the plain HTTP request it sends. The script sends one ride dispatch to the baseline frontend, waits for the request to flow through Kafka to the route service, and keeps only the Falco alerts from this run whose signadot.com/sandbox-routing-key label matches this sandbox's routing key. If there are any, the Job fails.

signadot job submit -f security/falco-check.yaml \
  --set sandbox=falco-route-eta \
  --set target=http://frontend.hotrod:8080 \
  --attach

target is the baseline frontend's address. The routing key, not the address, decides that the sandboxed route handles the request. After about 30 seconds, the Job fails:

Exercising sandbox 'falco-route-eta' (routing key: k6qxtz2h7df7u)
dispatch HTTP 200
Waiting for the request to reach the route service...
Reading Falco alerts for routing key 'k6qxtz2h7df7u'...

Falco raised 4 alert(s) for this sandbox:
      3   [Warning] Sandbox Outbound Connection Outside Cluster
      1   [Warning] Sandbox Shell Spawned

Details:
... Shell spawned in sandbox pod (routing_key=k6qxtz2h7df7u ... shell=sh parent=hotrod
    cmdline=sh -c wget -q -T 2 -O - https://example.com/traffic/v1/factor || true ...)
... Sandbox pod connected outside the cluster (routing_key=k6qxtz2h7df7u ...
    connection=10.244.0.62:34008->104.20.23.154:443 proc=wget ...)
Error: exit status 1

The sandboxed route spawned a shell and connected to a host outside the cluster while it handled the dispatch: behavior the diff didn't show and no test checked for. The list of alerts is also uploaded as a Job artifact (/tmp/falco-alerts.jsonl), which you can open from the Job's page in the Signadot dashboard.

7. Fix it and check again

The fix drops the phone-home and computes the ETA in process. It's one commit on the falco-route-eta-fix branch, built into signadot/hotrod:falco-route-eta-fix. Point the sandbox at the fixed image:

signadot sandbox apply -f security/sandbox.yaml \
  --set cluster=$CLUSTER \
  --set sandbox=falco-route-eta \
  --set image=signadot/hotrod:falco-route-eta-fix

Run the same check:

signadot job submit -f security/falco-check.yaml \
  --set sandbox=falco-route-eta \
  --set target=http://frontend.hotrod:8080 \
  --attach

This time the request still flows all the way to route, but nothing fires, and the Job passes:

Exercising sandbox 'falco-route-eta' (routing key: k6qxtz2h7df7u)
dispatch HTTP 200
Waiting for the request to reach the route service...
Reading Falco alerts for routing key 'k6qxtz2h7df7u'...
No Falco alerts for this sandbox. The change is clean.

The change is ready for review, and the passing check is evidence that it behaves at runtime.

Run the check during development and in CI

Falco watches processes on the cluster's nodes, so the change has to run in the cluster. A local sandbox, where the service runs on your machine, is out of Falco's view. To check a change before you open a pull request, build an image from your branch, update the sandbox with it using signadot sandbox apply, and submit the check Job, as in steps 4 and 6.

In CI, the pipeline builds an image from the pull request and creates a sandbox that forks the service's Deployment with that image. That is the same sandbox the check needs, so the pipeline submits the same Job with --attach. If Falco reports anything for the sandbox, the PR check fails, and the alerts stay attached to the Job for the reviewer. See CI/CD integrations for running Signadot Jobs from GitHub Actions, GitLab, Bitbucket and Jenkins.

A coding agent can run the same Job after its tests, read the alerts, and fix the change or explain the behavior in its pull request. See Run it in the agent loop and in CI.

Tune the rules for your services

This check only observes: it sends one ordinary dispatch and reads Falco's alerts. It doesn't send attack payloads, so it's safe to run against a sandbox whose calls to baseline services and shared databases are real.

The sandboxed workload runs the service's existing code along with the change, so the first run can show behavior the service already had, such as calls to a payment provider or an entrypoint script that starts a shell. Add exceptions for that behavior, so that each alert that remains is new:

Exclude known external destinations with and not fd.sip in (...) or and not fd.snet in (...).
Exclude expected processes with and not proc.pname in (...).
Pod labels apply to every container in the pod, including service mesh sidecars. To limit a rule to your application, add and container.name = <your-container>.
Falco's outbound macro ignores private (RFC 1918) addresses. If your cluster uses other ranges for pods or services, such as 100.64.0.0/10, exclude them as well.

Falco only sees the code paths that run, so the Job's traffic matters. It can send a few representative requests, run your existing tests with a testing framework, or run a ZAP scan, whose requests also reach error paths that functional tests skip.

Clean up

signadot sandbox delete falco-route-eta
signadot jobrunnergroup delete falco-check
kubectl delete -f security/falco-reader-rbac.yaml
helm uninstall falco -n falco

To remove HotROD, run kubectl delete namespace hotrod.

Troubleshooting

What you see	What to check
The Falco pod is `CrashLoopBackOff` with a driver error	Your kernel may not support the modern eBPF driver. Check that `/sys/kernel/btf/vmlinux` exists on the node. See Falco's driver docs.
The rules never fire, even on the vulnerable image	Confirm the fork pod carries the label: `kubectl -n hotrod get pod -l signadot.com/sandbox-routing-key`. If Falco shows `<NA>` for the label, your Falco build may not read pod labels from the container runtime; enable the k8smeta plugin.
The Job fails with `forbidden` reading Falco logs	`kubectl apply -f security/falco-reader-rbac.yaml` wasn't applied, or the runner group isn't using the `falco-reader` service account.
The Job stays queued	The runner pod is still pulling its image, or the node is low on disk. Check `kubectl -n signadot-tests get pods`.
The check passes on the vulnerable image	The dispatch may not have reached the sandboxed `route`. Confirm the fork pod is ready and that `location` IDs `123` and `567` exist (they're part of HotROD's seed data).

Frequently asked questions

How do I scope Falco rules to a single sandbox?

The Signadot Operator (v1.4.0 or later) labels every forked pod with signadot.com/sandbox-routing-key, set to the sandbox routing key. Baseline pods do not have the label. A Falco macro that matches k8s.pod.label[signadot.com/sandbox-routing-key] != "" limits a rule to sandbox pods, and comparing the label value with a routing key picks out one sandbox.

How does a Signadot Job turn Falco alerts into a pass/fail check?

The Job sends traffic through the sandbox, then reads Falco's JSON alerts from its pod logs and keeps only the alerts from this run whose routing-key label matches the Job's SIGNADOT_ROUTING_KEY. Any matching alert fails the Job, and the alerts are uploaded as a Job artifact.

Can Falco check a service running on my machine?

No. Falco watches system calls on the cluster's nodes, so the change has to run in the cluster. Build an image from your branch and create a sandbox that forks the service Deployment with that image. A local sandbox, where the service runs on your machine, is out of Falco's view.

Do Falco's default rules change when sandboxes are present?

No. Falco's default rules still apply to every pod in the cluster. Rules scoped to sandbox pods add checks for behavior that is normal elsewhere but unexpected from a change, such as any connection that leaves the cluster.

Why does the check need to send traffic to the sandbox?

Falco only sees code paths that run. The check exercises the changed service with requests that carry the sandbox routing key. In your own pipeline, run your existing tests or a ZAP scan through the sandbox, so the changed code paths actually execute.

What Falco detects​

Before you begin​

1. Install HotROD​

2. Get the tutorial files​

3. Install Falco with sandbox-scoped rules​

4. Run the change in a sandbox​

5. Set up the check Job​

6. Run the check​

7. Fix it and check again​

Run the check during development and in CI​

Tune the rules for your services​

Clean up​

Troubleshooting​

Frequently asked questions​

How do I scope Falco rules to a single sandbox?​

How does a Signadot Job turn Falco alerts into a pass/fail check?​

Can Falco check a service running on my machine?​

Do Falco's default rules change when sandboxes are present?​

Why does the check need to send traffic to the sandbox?​

See also​