Skip to main content

Redis Streams Isolation with Consumer Groups

Redis Streams works with Signadot sandboxes without giving each sandbox its own Redis or its own copy of the stream. Once the routing key travels inside each stream entry, most changes near a stream need no isolation machinery at all. The one case that does, sandboxing a consumer, has two patterns, and this guide walks through both, including the failure modes. The recommended pattern is message-level routing, and the Redis Streams tutorial runs it on a real cluster.

In short
  • Never let a sandboxed consumer join the baseline's consumer group. Redis splits a group's entries between its consumers by timing, not by routing key.
  • Give each sandbox its own group, <group>-<routing key>, created at $ so it sees new entries only.
  • Carry the routing key inside each entry, as traceparent and baggage fields. Stream entries have no headers, and the OpenTelemetry Redis instrumentation does not do this for you.
  • Check every entry on both sides, and acknowledge every entry. The baseline and the sandbox ask the Routes API whether a key is theirs, and XACK what they skip as well as what they process.

Redis Streams needs its own thinking because of how Signadot routing works. Signadot routes synchronous requests (HTTP, gRPC) to sandboxes at the network layer: a request carries a routing key in its headers, and the routing layer delivers it to the sandboxed workload instead of the baseline one. A stream breaks that chain. Nothing calls a consumer; it reads entries with XREADGROUP on its own schedule, so there is no request for the routing layer to redirect. The entry itself has to carry the routing context, and the consumers themselves have to act on it.

New to Redis Streams? A two-minute refresher

The rest of this guide assumes a handful of Redis Streams terms. If they are already familiar, skip this section; nothing in it is Signadot-specific.

  • Stream: a Redis key that holds an append-only log of entries. A producer appends an entry with XADD without knowing who will read it, or whether anyone ever will.
  • Entry ID: every entry gets an ID, by default <milliseconds>-<sequence> (for example 1792725846853-0), and IDs only ever grow. A reader's position in a stream is an entry ID.
  • Fields: an entry is a flat set of field-value pairs, like a small hash. That is all it is. There are no headers or attributes kept apart from the payload.
  • Consumer group: readers share a stream through a named group, created with XGROUP CREATE. Consumers in the group read with XREADGROUP, and each new entry goes to exactly one consumer in the group: whichever one asks next. Across groups there is no such exclusivity. Every group sees every entry, at its own pace.
  • Last-delivered ID: a group's position in the stream. XREADGROUP with the special ID > returns entries after it, and moves it forward.
  • Pending entries list (PEL): each group tracks every entry it has delivered but not yet acknowledged, with the consumer holding it, how long it has been idle, and how many times it has been delivered.
  • XACK: acknowledges one entry and removes it from the group's PEL. Until then the entry stays pending, and can be read again by the same consumer or claimed by another one.

If you know Kafka, four differences matter for this guide:

  • No headers. A Kafka record carries headers next to its value; a stream entry carries only fields. Tracing context and the routing key have to travel as reserved fields inside the entry.
  • No partitions. Kafka assigns partitions to group members, so which member gets a record depends on its partition. Within a Redis consumer group, an entry goes to whichever consumer calls XREADGROUP next.
  • An explicit start position. There is no auto.offset.reset. Where a new group starts is the ID you pass to XGROUP CREATE: $ for entries appended from now on, 0 for everything still in the stream.
  • Acknowledgements per entry. Kafka commits one offset per partition. Redis acknowledges entries one at a time, and an entry nobody acknowledges stays in the PEL indefinitely.

The two properties doing the work in the rest of this guide: fields travel with the entry untouched, and delivery is exclusive within a consumer group but not across groups.

How Routing Keys Travel Through Redis Streams​

For synchronous hops, the sandbox routing key travels in request headers (baggage / tracestate, key sd-routing-key; see Header-Based Routing). Across a stream it travels in the entry fields: the producer copies its current context into the entry when it calls XADD, and the consumer reads it back out and makes it current while it handles the entry, so its downstream requests, and any entries it appends, carry the key too.

With OpenTelemetry, each half is one call on the standard propagator. On the producer, propagate.inject writes the W3C traceparent and baggage fields, and the baggage holds sd-routing-key:

Producer: add the context to the entry
from opentelemetry import propagate

carrier = {}
propagate.inject(carrier)  # {"traceparent": "...", "baggage": "sd-routing-key=..."}
redis_client.xadd(stream, {**fields, **carrier})

On the consumer, propagate.extract turns those fields back into a context:

Consumer: read the context back out
from opentelemetry import baggage, context, propagate

ctx = propagate.extract(fields)
routing_key = baggage.get_baggage("sd-routing-key", ctx)
token = context.attach(ctx)  # outbound calls made by the handler carry the key
try:
    handle(entry_id, fields)
finally:
    context.detach(token)
Redis needs explicit propagation

This is where Redis differs from Kafka. Instrumented Kafka clients typically inject and extract context on record headers by themselves. opentelemetry-instrumentation-redis only creates spans for Redis commands: it does not write context into stream entries or read it back out. Without the two steps above, the routing key stops at the producer, even in the cases below that need nothing else.

The Three Cases​

"Does this work with Redis Streams?" is really three questions, with different answers depending on where your change sits relative to the stream:

  1. Downstream of the stream: the producer and consumer are both baseline, and your fork sits somewhere after the stream. The key rides through the entry fields, comes back out in the consumer, and normal request routing takes over on the consumer's next synchronous call. Beyond copying context in and out of the entry, this needs nothing.
  2. A sandboxed producer: it appends exactly the way the baseline does, to the same stream, with the key in the entry fields. Redis never learns the sandbox exists. This also needs nothing beyond context propagation.
  3. A sandboxed consumer: nothing calls a consumer, so nothing routes to it. The entry has to reach the right copy, and only the right copy. This case needs one of the two patterns below.

The Redis Streams tutorial runs all three cases on a real cluster.

Why the Naive Approach Fails​

The obvious way to test a consumer change is to deploy the branch and change nothing else: same stream, same consumer group. This fails, and it fails silently.

Within a consumer group, Redis hands each entry to whichever consumer calls XREADGROUP next. The routing key is in the entry, but Redis never looks at it. The moment the sandboxed consumer starts reading through the baseline's group, the two split the stream between them by timing, and the damage goes in both directions:

  • Your test entries mostly land on the baseline consumer, so your own traffic rarely reaches your sandbox.
  • Everyone else's ordinary entries land on the sandbox a share of the time, so unreviewed branch code silently processes work that was never meant for it, and acknowledges it. An acknowledged entry leaves the PEL, so nothing retries it and the baseline never sees it.

Adding a routing-key check to the sandboxed consumer does not fix this. An entry it skips was still delivered to it and not to the baseline, so the work is lost if the sandbox acknowledges it, and stuck in the PEL if it does not.

A shared consumer group is not isolation

There are no errors, just some fraction of the entries silently handled, and acknowledged, by the wrong version. Never let a sandboxed consumer read through the baseline's consumer group.

Keep the one shared stream, and separate traffic per entry instead of with infrastructure. Two moves: give the sandboxed consumer its own consumer group, and make both consumers, baseline included, decide per entry using the Routes API.

1. A Consumer Group per Sandbox​

The sandboxed consumer reads through a new, unique group, for example the baseline group name suffixed with the sandbox routing key: <group>-<routing key>. The routing key is available inside the sandboxed workload as SIGNADOT_SANDBOX_ROUTING_KEY, set by the sandbox downward API. The sandbox is no longer competing with the baseline for entries: both groups now receive every entry, and the decision about who processes what moves into the consumers, where the routing key is visible.

A brand-new group starts wherever XGROUP CREATE says. Create the sandbox's group at $, so that it sees only entries appended from then on:

Redis command
XGROUP CREATE orders order-workers-<routing key> $ MKSTREAM

MKSTREAM creates the stream if it does not exist yet. When the sandbox's pod restarts, the group already exists and XGROUP CREATE fails with BUSYGROUP: treat that as success, and the consumer resumes from the group's last-delivered ID.

Never start a sandbox's group at 0

In a sandbox, 0 means the new group replays the stream's entire retained history through your branch code on startup. 0 is the right start for the baseline's group, which is created once, so that entries appended before the baseline's first start are not missed.

2. Both Sides Check Each Entry​

Both consumers ask the same question of every entry: is this routing key live, and does it route to me?

  • The sandboxed consumer processes an entry only if its routing key matches the sandbox (or a RouteGroup the sandbox belongs to), and skips everything else.
  • The baseline consumer processes everything except entries whose key belongs to a live sandbox of the same workload. It has to actively decline work it would otherwise have done, which is why the check must land in the baseline too. An entry with no key, or with a key for a sandbox of some other service, or for a sandbox that has been deleted, stays with the baseline.

The live set of routing keys comes from the Signadot Routes API, available as gRPC and REST APIs. The REST endpoint returns the routing rules for one baseline workload, and each rule pairs a routingKey with its destinationSandbox:

Routes API (REST)
GET http://routeserver.signadot.svc:7778/api/v1/workloads/routing-rules
    ?baselineKind=Deployment&baselineNamespace=<namespace>&baselineName=<name>

From those rules a consumer keeps two sets, refreshed in the background (the reference implementation polls every 5 seconds): sandbox_keys, every routing key in the rules, and own_keys, the keys whose destination is this sandbox. The check per entry then works unchanged whether the workload is running as baseline or in a sandbox:

The per-entry check
def should_process(routing_key):
    if sandbox_name:  # SIGNADOT_SANDBOX_NAME is set: running in a sandbox
        # Process only keys that route to this sandbox.
        return routing_key is not None and (
            routing_key == sandbox_routing_key or routing_key in own_keys)
    # In the baseline: process only keys no sandbox has claimed.
    return routing_key not in sandbox_keys

SIGNADOT_SANDBOX_NAME and SIGNADOT_SANDBOX_ROUTING_KEY are set automatically in sandboxed workloads. The baseline workload identity (SIGNADOT_BASELINE_KIND, SIGNADOT_BASELINE_NAME, SIGNADOT_BASELINE_NAMESPACE) is configuration you set on the workload itself, for example as environment variables in its Deployment spec, and a sandbox's fork inherits it. In Go, the watched package keeps the routing rules current for you; the Kafka guide shows it.

3. Acknowledge Every Entry​

Redis adds a rule that Kafka does not need. Every group receives every entry, so each consumer skips most of what it reads: the sandbox skips the baseline's traffic, and the baseline skips the sandbox's. A skipped entry is still a delivered entry. If the consumer does not XACK it, the entry stays in that group's PEL for good, and the PEL grows with every entry meant for someone else.

So XACK every entry, processed or skipped. The reference implementation also acknowledges an entry whose handler failed, which keeps the PEL empty at the cost of dropping that entry; see Known Limits.

4. Put It in a Library​

The full cost of the pattern is the group name, the start ID, the per-entry check, the acknowledgement, and copying context in and out of entry fields. It is the same code in every producer and consumer, baseline included, which is why it belongs in a shared platform library rather than in each service. With the library in place, application code stays plain Redis client code, and consumers become forkable by upgrading a dependency.

The examples repo has a reference implementation for Python on redis-py, platform_redis:

  • platform_redis.Redis is a drop-in redis.Redis whose xadd adds traceparent and baggage to the entry.
  • platform_redis.StreamConsumer picks the group, creates it at the right start ID, polls the Routes API, checks and acknowledges every entry, makes the entry's context current while the handler runs, and recovers entries left pending by restarted or replaced pods.

The application only supplies a handler:

Application code
redis_client = platform_redis.Redis.from_url(REDIS_URL, decode_responses=True)
consumer = platform_redis.StreamConsumer(
    redis_client, stream=STREAM, group=CONSUMER_GROUP, name=socket.gethostname())
consumer.run(handle)

Routing is on in sandboxes and in workloads that set SIGNADOT_BASELINE_NAME. Elsewhere, such as a cluster without Signadot, StreamConsumer processes every entry.

Stream-per-Sandbox Isolation​

The alternative is to give the sandbox its own stream. A stream is just a Redis key, and the first XADD (or XGROUP CREATE ... MKSTREAM) creates it, so there is nothing to provision ahead of time and no broker limit on how many you have. A Resource Plugin still earns its place: it picks a stream name for the sandbox, hands it to the sandbox's workloads through its outputs, and deletes the key, groups and all, when the sandbox is deleted. The sandboxed consumer is pointed at the new stream through configuration (an environment variable, if the stream name already comes from one).

The appeal is real: no consumer code changes for routing. No key check, no filtering, no library, and the group name can stay the same because a different stream has its own groups. (Context still has to travel in the entry fields if something downstream of the consumer is forked too.)

But the baseline producer still appends where it always did. The routing key on the entry changes nothing, because nothing is reading it, so the sandbox stream stays empty. The fix is to co-fork the producer into the sandbox: a second copy that runs the unchanged baseline image, with the sandbox's stream name in its configuration. A keyed request then routes to that producer copy, which appends to the sandbox stream, and the sandboxed consumer picks it up.

That is also the limitation: you cannot sandbox the consumer on its own. You fork the producer and the consumer together, and you have to know every producer on the stream. If a fifth service also appends there and you forget it, part of the flow silently never reaches the sandbox.

Choosing Between Them​

Message-level routing
one stream · a group per sandbox		Stream-per-sandbox
a stream per sandbox · config only
What has to change in the app?Producers: context in the entry fields. Consumers: group name, start ID, and a check and an XACK per entry.Nothing, if the stream name already comes from the environment.
What has to be forked?Only the workloads you actually changed.Those, plus every producer on the stream, co-forked with sandbox config.
What do you have to know?Nothing. Every decision is per entry.The producer graph for every stream you isolate, kept current.
How does it fail?Silently, if a consumer is missing the check: stolen or duplicated work. If it is missing the XACK: a PEL that only grows.Silently, if a producer is missed: no entries arrive at all.
What does a sandbox cost?A consumer group, which outlives the sandbox until you destroy it.A stream key, named and deleted by a resource plugin.
Where does it fit?The default. Keeps scaling with services and teams.When consumer code is off limits, or the stream has one producer and one team.
Default to message-level routing

Reach for a stream per sandbox when you cannot touch the consumer's code, and in that case map the stream's producers first.

What a Forkable Redis Streams Consumer Needs​

For message-level routing, a consumer is sandbox-ready when it has:

  • A consumer group of its own, created at $. Never the baseline's group, and never 0 in a sandbox.
  • Context copied in and out of entry fields. Producers add traceparent and baggage on XADD. Consumers extract them, make them current while the handler runs, and strip them before the application sees the fields.
  • Cooperation on both sides. Baseline and sandbox read the live routing keys from the Routes API, so exactly one of them processes each entry.
  • An XACK for every entry. Processed or skipped, so no group's PEL fills up with entries meant for someone else.
  • One library per language, not a change per service. Group name, start ID, the key check, the acknowledgement, and context propagation, written once.

Sandboxed producers and anything downstream of the stream need only the context propagation.

Operational Notes​

Leftover Consumer Groups​

Redis does not delete a consumer group when its consumers go away, and Signadot does not know the group exists. After a sandbox is deleted, its group <group>-<routing key> stays on the stream. It holds no copy of the data, so it costs little, but XINFO GROUPS keeps listing it, with a lag that grows with every new entry. Destroy it when you delete the sandbox, for example in the same script or CI job:

Redis command
XGROUP DESTROY orders order-workers-<routing key>

Dead Consumers​

A consumer is created by the first XREADGROUP that names it, and Redis never removes it on its own. If each pod uses its hostname as its consumer name, as the reference implementation does, every replaced pod leaves a consumer behind in its group, along with any entries it read but did not acknowledge. XINFO CONSUMERS <stream> <group> lists them with their pending counts and idle times. Once a dead consumer has nothing pending, remove it:

Redis command
XGROUP DELCONSUMER orders order-workers <consumer name>

Deleting a consumer that still has pending entries discards those entries: nothing can claim them afterwards. Recover them first.

Recovering Pending Entries​

An entry stays pending until it is acknowledged, so a consumer that stops between reading an entry and acknowledging it leaves the entry in the PEL. Two reads bring it back:

  • At startup, read your own pending entries first. XREADGROUP with the ID 0 instead of > returns entries already delivered to this consumer name and not yet acknowledged. This covers a process that restarts under the same name, such as a container restart in the same pod.
  • Periodically, claim other consumers' idle entries. XAUTOCLAIM moves entries that another consumer has held for longer than a minimum idle time to the calling consumer and returns them. This covers a pod that was replaced and will never come back under its old name. The reference implementation runs it every minute with a 60-second idle threshold.

Apply the same routing-key check and the same XACK to recovered entries as to new ones. Reading pending entries with 0 can also return an entry with no fields, if the entry was trimmed from the stream while it was pending: acknowledge it and move on.

Trimming Ignores Consumer Groups​

A stream grows until you trim it, usually with MAXLEN on XADD or with XTRIM. By default, trimming removes the oldest entries whether or not every group has read or acknowledged them. So a leftover sandbox group never stops the stream from being trimmed, and a group that falls far enough behind can miss entries that were trimmed before it read them.

Since Redis 8.2, XADD and XTRIM also accept an ACKED option, which trims only entries that every group has acknowledged. With ACKED, a leftover sandbox group holds back trimming until you destroy it, so cleaning up sandbox groups stops being optional.

Redis Pub/Sub Is Different​

Redis also has Pub/Sub (PUBLISH and SUBSCRIBE), which works differently. Messages are fire-and-forget: nothing is stored or acknowledged, and every subscriber to a channel receives every message. There is no consumer group to get wrong, because a sandboxed subscriber already receives its own copy of everything, so filtering alone is enough: baseline and sandbox subscribers apply the same per-message check and drop what is not theirs. A Pub/Sub message is a single string, so the routing context has to go inside the payload, for example as traceparent and baggage fields in a JSON envelope.

Known Limits of the Reference Implementation​

  • A failed handler drops its entry. StreamConsumer acknowledges every entry, including one whose handler raised an exception, so the failure is logged and the entry is not retried. If you need retries, keep acknowledging skipped entries, leave failed ones pending for the reclaim loop to retry, and move entries whose delivery count passes a limit to a dead-letter stream.
  • Routing changes take up to one poll to reach the baseline. A sandbox knows its own routing key from the moment it starts, but the baseline learns about it on its next Routes API poll, up to 5 seconds later. In that window an entry carrying the new key can be processed by both. In the same way, for up to one poll after a sandbox is deleted, the baseline still skips its key, so an entry sent with that key may be processed by neither. Give a new sandbox a few seconds after it is ready before you send test traffic.
  • Consumers wait for the Routes API. A consumer reads nothing until its first Routes API answer, so if the Route Server is unreachable at startup it stays not ready. After that, it keeps the last rules it fetched whenever a refresh fails.
  • decode_responses=True is required. StreamConsumer compares field names and routing keys as strings, and refuses a client created without it.
  • Pipelined XADD carries no context. Only platform_redis.Redis.xadd adds the context fields. pipeline().xadd(...) goes through redis-py's own pipeline class, so entries appended that way have no routing key and go to the baseline.

Frequently Asked Questions​

Can a sandboxed consumer share the baseline's Redis consumer group?​

No. Within a consumer group, Redis delivers each entry to exactly one consumer, whichever one calls XREADGROUP next, and it never looks at the routing key. A sandboxed consumer in the baseline's group silently takes a share of everyone's entries and acknowledges them, while most of its own test entries go to the baseline. Give each sandbox a consumer group of its own instead.

Does OpenTelemetry propagate context through Redis Streams automatically?​

No. opentelemetry-instrumentation-redis creates spans for Redis commands, but it does not write trace context or baggage into stream entries. The producer has to inject traceparent and baggage into the entry's fields when it calls XADD, and the consumer has to extract them when it reads the entry. Instrumented Kafka clients typically do this on record headers, but Redis stream entries have no headers.

Should a sandbox's consumer group start at $ or 0?​

At $. A group created at $ sees only entries appended after it was created, which is what a sandbox needs. A group created at 0 starts with everything still in the stream, so a sandbox would replay the stream's retained history through branch code. 0 is the right start for the baseline's group, which is created once, so that entries appended before the baseline first starts are not missed.

Do consumers need to acknowledge the entries they skip?​

Yes. Every consumer group receives every entry, so each consumer skips most of what it reads. An entry that is delivered but never acknowledged with XACK stays in the group's pending entries list (PEL) indefinitely, so the baseline and sandboxed consumers both acknowledge every entry, processed or skipped.

What happens to a sandbox's consumer group when the sandbox is deleted?​

It stays on the stream. Redis does not delete a consumer group when its consumers go away, and Signadot does not know the group exists, so destroy it with XGROUP DESTROY when you delete the sandbox. It holds no copy of the data, but XINFO GROUPS keeps listing it, and with the ACKED trimming option of Redis 8.2 and later it holds back trimming.

Does the same approach work with Redis Pub/Sub?​

Partly. Redis Pub/Sub delivers every message to every subscriber and stores nothing, so there is no consumer group to separate: a sandboxed subscriber already receives its own copy, and a per-message routing-key check in each subscriber is enough. A Pub/Sub message is a single string, so the routing context has to travel inside the payload, for example in a JSON envelope.

Resources​