Skip to main content

Kafka

Kafka works with Signadot sandboxes without giving each sandbox its own broker or its own copy of the topic. Most changes near a Kafka hop need no isolation machinery at all. The one case that does, sandboxing a consumer, has two proven patterns, and this guide walks through both, including the failure modes. The recommended pattern is message-level routing.

Kafka needs its own thinking because of how Signadot routing works. Signadot routes synchronous requests (HTTP, gRPC) to sandboxes at the network layer: a request carries a routing key in its headers, and the routing layer delivers it to the sandboxed workload instead of the baseline one. A broker breaks that chain. Nothing calls a consumer; it pulls records from Kafka on its own schedule, so there is no request for the routing layer to redirect. The message itself has to carry the routing context, and the consumers themselves have to act on it.

New to Kafka? A two-minute refresher

The rest of this guide assumes a handful of Kafka terms. If they are already familiar, skip this section; nothing in it is Signadot-specific.

  • Topic: a named stream that producers publish records to. A producer does not send a record to anyone in particular. It appends the record to the topic without knowing who will read it, or whether anyone ever will.
  • Partition: a topic is split into partitions, and each partition is an append-only log. Records are added at the tail and never reordered, and the partition a record lands in is decided at publish time.
  • Offset: a consumer's position in a partition is a single number, the offset, and Kafka remembers it per consumer group. "Where do I resume?" and "where does a brand-new reader start?" are both questions about offsets.
  • Consumer group: consumers join a group rather than subscribing alone, and Kafka splits the topic's partitions across the group's members, rebalancing whenever a member joins or leaves. Within a group, each record is delivered to exactly one member: whichever one holds its partition. Across groups there is no such exclusivity. Every group receives every record, at its own pace, with its own offsets.
  • Headers: alongside its value, a record carries headers, arbitrary key-value metadata that the broker never reads and partitioning ignores. Tracing context already travels this way, and so does the sandbox routing key in the rest of this guide.

Step through the same ideas as a diagram:

Step through: topics and partitions, consumer groups, per-group delivery, and headers.

The two properties doing the work in the rest of this guide: headers travel with the record untouched, and delivery is exclusive within a consumer group but not across groups.

How Routing Keys Travel Through Kafka​

For synchronous hops, the sandbox routing key travels in request headers (baggage / tracestate, key sd-routing-key; see Header-Based Routing). Across a Kafka hop it travels in the record headers: the producer copies the key from the incoming request into the record, and the consumer reads it back out and propagates it onto its own downstream requests and any records it publishes.

If your services use OpenTelemetry instrumentation (or an APM agent that propagates baggage), the instrumented Kafka client typically injects and extracts baggage on record headers already. It is the same mechanism that lets a distributed trace survive a queue, and it does not require a tracing backend.

The Three Cases​

"Does this work with Kafka?" is really three questions, with different answers depending on where your change sits relative to the topic. Step through the diagram:

Step through: the three cases, and which of them need anything at all.
  1. Downstream of the topic: the producer and consumer are both baseline, and your fork sits somewhere after the queue. The key rides through the record headers, comes back out in the consumer, and normal request routing takes over on the next synchronous call. This works out of the box.
  2. A sandboxed producer: it publishes exactly the way the baseline does, to the same topic and the same partitions, with the key in the record headers. The broker never learns the sandbox exists. This also works out of the box.
  3. A sandboxed consumer: nothing calls a consumer, so nothing routes to it. The message has to reach the right copy, and only the right copy. This case needs one of the two patterns below.

Why the Naive Approach Fails​

The obvious way to test a consumer change is to deploy the branch and change nothing else: same topic, same consumer group id. This fails, and it fails silently. Step through what happens:

Step through: fork a consumer into the baseline's group, and partition assignment ignores the routing key.

Kafka delivers each message to exactly one member of each consumer group. The moment the sandboxed consumer joins the baseline's group, Kafka rebalances and splits the partitions between them, and which member gets a message is decided by partition, not by routing key. The damage goes in both directions:

  • Your test messages mostly land on baseline-held partitions, so your own traffic rarely reaches your sandbox.
  • Everyone else's ordinary traffic lands on the sandbox-held partitions a share of the time, so unreviewed branch code silently processes work that was never meant for it.
caution

A shared consumer group is not isolation. There are no errors and no dropped messages, just some fraction of the work silently done by the wrong version. Never let a sandboxed consumer join the baseline's consumer group.

Keep the one shared topic, and separate traffic per message instead of with infrastructure. Two moves: give the sandboxed consumer its own consumer group, and make both consumers, baseline included, decide per message using the Routes API.

Step through: a group per sandbox, every message seen by both, and one shared per-message check.

1. A Consumer Group per Sandbox​

The sandboxed consumer uses a new, unique group id, for example the baseline group id suffixed with the sandbox routing key. It is no longer competing for partitions with the baseline group: both groups now receive a copy of every message, and the decision about who processes what moves into the consumers, where the routing key is visible.

A brand-new group has no committed offsets, so where it starts is decided by auto.offset.reset. The consumer default of latest is what you want: the sandbox should see new messages, not replay history.

note

Watch out for services that deliberately set auto.offset.reset=earliest (also the Kafka Streams default). In a sandbox, earliest means the new group replays the topic's entire retained history through your branch code on startup.

2. Both Sides Check Each Message​

Both consumers ask the same question of every message: is this routing key live, and does it route to me?

  • The sandboxed consumer processes a message only if its routing key matches the sandbox (or a RouteGroup the sandbox belongs to), and skips everything else.
  • The baseline consumer processes everything except messages whose key belongs to a live sandbox. It has to actively decline work it would otherwise have done, which is why the check must land in the baseline too.

The live set of routing keys comes from the Signadot Routes API, available as gRPC and REST APIs with Go library code. With the Go SDK, a consumer keeps a watched view of the routing rules for its baseline workload:

import "github.com/signadot/routesapi/go-routesapi/watched"

routing, err := watched.BaselineWatchedFromEnv(nil)

and applies one check per message, which works unchanged whether the workload is running as baseline or in a sandbox:

func ShouldProcess(routing watched.BaselineWatched, routingKey string) bool {
	if sb := os.Getenv("SIGNADOT_SANDBOX_NAME"); sb != "" {
		// In a sandbox: process only keys that route to this sandbox.
		return routing.RoutesTo(routingKey, sb)
	}
	// In the baseline: process only keys no sandbox has claimed.
	return routing.Get(routingKey) == nil
}

SIGNADOT_SANDBOX_NAME is set automatically in sandboxed workloads by the sandbox downward API. The baseline workload identity (SIGNADOT_BASELINE_KIND, SIGNADOT_BASELINE_NAME, SIGNADOT_BASELINE_NAMESPACE) is configuration you set on the workload itself, for example as environment variables in its Deployment spec. SIGNADOT_ROUTESERVER is optional: it defaults to the Route Server address of a standard Signadot Operator installation.

3. Put It in a Library​

The full cost of the pattern is the group id, the offset-reset setting, the per-message check, and propagating the key onto anything the consumer publishes downstream. It is the same few lines in every consumer, baseline included, which is why it belongs in a shared platform library rather than in each service. With the library in place, consumers become forkable by upgrading a dependency. See the working demo for a complete implementation.

Topic-Based Isolation​

The alternative is to give the sandbox its own topic. A Resource Plugin provisions the topic when the sandbox is created and tears it down with the sandbox, and the sandboxed consumer is pointed at the new topic through configuration (an environment variable, if the topic name already comes from one).

Step through: the sandbox topic starts empty until the producer is co-forked into the sandbox.

The appeal is real: no consumer code changes at all. No key check, no filtering, no library, and the group id can stay the same because a different topic has different partitions.

But the baseline producer still publishes where it always did. The routing key on the message changes nothing, because nothing is reading it, so the sandbox topic stays empty. The fix is to co-fork the producer into the sandbox: a second copy that runs the unchanged baseline image, with the sandbox's topic name in its configuration. A keyed request then routes to that producer copy, which publishes to the sandbox topic, and the sandboxed consumer picks it up.

That is also the limitation: you cannot sandbox the consumer on its own. You fork the producer and the consumer together, and you have to know every producer on the topic. If a fifth service also publishes there and you forget it, part of the flow silently never reaches the sandbox.

Choosing Between Them​

Message-level routing
one topic · a group per sandbox		Topic-based isolation
a topic per sandbox · config only
What has to change in the app?Every consumer: group id, offset reset, and a check per message.Nothing, if the topic name already comes from the environment.
What has to be forked?Only the workloads you actually changed.Those, plus every producer on the topic, co-forked with sandbox config.
What do you have to know?Nothing. Every decision is per message.The producer graph for every topic you isolate, kept current.
How does it fail?Silently, if a consumer is missing the check: stolen or duplicated work.Silently, if a producer is missed: no messages arrive at all.
What does a sandbox cost?A consumer group. Nothing to provision.A topic per sandbox, provisioned and reclaimed. Brokers have limits.
Where does it fit?The default. Keeps scaling with services and teams.When consumer code is off limits, or the topic has one producer and one team.
tip

Default to message-level routing. Reach for topic-based isolation when you cannot touch the consumer's code, and in that case map the topic's producers first.

What a Forkable Consumer Needs​

For message-level routing, a consumer is sandbox-ready when it has:

  • A consumer group of its own, starting at the latest offset. Never the baseline's group id, and auto.offset.reset pinned to latest.
  • Cooperation on both sides. Baseline and sandbox read the live routing keys from the Routes API, so exactly one of them processes each message.
  • One library per language, not a change per service. Group id, the key check, and key propagation downstream, written once.

Sandboxed producers and anything downstream of the topic need none of this.

Resources​