Falco

Overview

Falco is the open source runtime security engine for Kubernetes, and a graduated CNCF project. It watches system calls on every node and raises an alert when a container does something one of its rules describes.

Scanners test what an endpoint returns. Falco shows what the code does while it runs, and some problems only show up there: a new dependency that connects to the internet the first time it's used, a handler that passes input to a shell, code that reads a file it shouldn't. A diff review can miss these, and so can unit tests, because nobody writes a test for behavior they don't expect. Run Falco against a sandbox and you see that behavior for a single change, before it merges.

What Falco detects

Falco's rules describe what a process does, not what a service returns. These are the behaviors most worth catching in a sandbox:

Behavior	What it can point to	Rules
Outbound connections to addresses outside the cluster	A dependency phoning home, data exfiltration, server-side request forgery	A sandbox-scoped rule, shown below
Shells started by the application	Command injection, or a handler or library shelling out	A sandbox-scoped rule, and Falco's default Run shell untrusted
Reads of sensitive files	Code reading credentials or private keys it has no reason to touch	Falco's default Read sensitive file untrusted and Search Private Keys or Passwords
New binaries run at runtime	A downloaded payload, or an executable written after the container started	Falco's default Drop and execute new binary in container
Calls to the Kubernetes API	Code that talks to the API server when it shouldn't	Falco's default Contact K8S API Server From Container

Falco's default rules apply to every pod in the cluster. Rules scoped to sandbox pods report behavior that's normal elsewhere but unexpected from a change, such as any connection that leaves the cluster.

How Jobs and sandboxes run Falco checks

Four pieces fit together:

1. A sandbox runs the change in the cluster, and its pods are labeled. The sandbox forks the changed service's Deployment with an image built from the change. The Signadot Operator (v1.4.0 or later) labels every forked pod with signadot.com/sandbox-routing-key, set to the sandbox's routing key. Baseline pods don't have this label.

2. Falco runs on every node, with rules scoped to sandbox pods. Falco reads pod labels from the container runtime, so a rule can match sandbox pods directly:

   - macro: signadot_sandbox_pod
     condition: (k8s.pod.label[signadot.com/sandbox-routing-key] != "")

   Rules that use this macro fire for the code under test and stay quiet for normal traffic through the baseline. Install Falco with Helm, with JSON output turned on, so a Job can read its alerts.

3. A Job exercises the change and reads the alerts. The Job is tied to the sandbox with routingContext, and trafficManager.injectRoutingKey: auto adds the routing key to the requests it sends, so they reach the change. Once the traffic has flowed through, the Job reads Falco's alerts. It keeps only the alerts from this run whose routing-key label matches the Job's SIGNADOT_ROUTING_KEY. The Job's runner group uses a service account that can read Falco's pod logs in Falco's namespace, and nothing else.

4. The Job's result is the verdict. Any matching alert fails the Job. The alerts are uploaded as a Job artifact, which you can open from the Job's page in the Signadot dashboard.

Falco only sees the code paths that run, so the Job's traffic matters. It can send a few representative requests, run your existing tests, or run a ZAP scan, whose requests also reach error paths that functional tests skip.

Unlike an active scan, this check only observes. It sends ordinary requests and reads alerts, so it's safe to run against a sandbox whose calls to shared services and databases are real.

Check during development

Falco watches processes on the cluster's nodes, so the change has to run in the cluster. A local sandbox, where the service runs on your machine, is out of Falco's view. To check a change before you open a pull request:

1. Build an image from your branch.
2. Create or update a sandbox that forks the service with that image, using signadot sandbox apply.
3. Submit the check Job with signadot job submit ... --attach.
4. If the Job fails, read the alerts in its output, fix the change, update the sandbox with the new image, and run the Job again.

A coding agent can run the same loop on its own changes, and fix the change or explain the behavior in its PR. See Run it in the agent loop and in CI.

Check every pull request

In CI, the pipeline builds an image from the pull request and creates a sandbox that forks the service's Deployment with that image. That is the same sandbox the check needs, so the pipeline submits the same Job with --attach. If Falco reports anything for the sandbox, the PR check fails, and the alerts stay attached to the Job for the reviewer. See CI/CD integrations for running Signadot Jobs from GitHub Actions, GitLab, Bitbucket and Jenkins.

Tune the rules

The sandboxed workload runs the service's existing code along with the change. The first run can show behavior the service already had, such as calls to a payment provider or an entrypoint script that starts a shell. Add exceptions for that behavior, so that each alert that remains is new:

• Exclude known external destinations with and not fd.sip in (...) or and not fd.snet in (...).
• Exclude expected processes with and not proc.pname in (...).
• Pod labels apply to every container in the pod, including service mesh sidecars. To limit a rule to your application, add and container.name = <your-container>.
• Falco's outbound macro ignores private (RFC 1918) addresses. If your cluster uses other ranges for pods or services, such as 100.64.0.0/10, exclude them as well.

Walk through it with the tutorial

The Falco and Signadot sandboxes tutorial puts this together on HotROD, Signadot's demo app:

• A change to HotROD's route service computes an ETA with a small helper package. The first time it's used, the helper runs a shell command that reaches a host outside the cluster: a dependency phoning home. The ETA is correct, so the change could pass review and its unit tests.
• Falco is installed with Helm, with two sandbox-scoped rules: one for outbound connections outside the cluster and one for spawned shells.
• A check Job sends one ride dispatch through frontend, Kafka and driver to the sandboxed route service. Falco catches the shell and the outbound connection for the sandbox only, and the Job fails.
• After the sandbox is updated with the fixed image, the same Job passes.

The change and the fix are published as images, so you don't need to build anything. The tutorial's Helm values, RBAC, runner group, sandbox and Job specs are in the same directory, ready to copy into your own repository.