Skip to main content

Set up Message Queue Isolation

Message queues introduce asynchronous communication between microservices, typically via a message queue broker or service. Since communication is indirect, one might consider including the broker and producers in sandboxes when working on a message queue consumer microservice, to ensure the sandbox consumer does not interfere with the baseline by consuming messages intended for the baseline.

However, this is often impractical in terms of setup time and resource usage. Signadot provides more effective alternatives for isolating message queue traffic in sandboxes.

General Approaches to Message Queue Isolation​

1. Message-Level Routing (Routing API)​

Many message queues (e.g., Kafka) support multiple consumers. In these scenarios, each consumer can receive all messages. To isolate sandboxes, you can:

  1. Decorate messages with a routing key in the header or message attribute.
  2. Send all messages to all consumers (e.g., Kafka consumer groups).
  3. Configure each consumer to process or ignore messages based on the routing key.

This approach is lightweight, flexible, and works with RouteGroups. It allows sharing the message queue broker between the baseline and sandboxes without interference.

Signadot provides gRPC and REST APIs, and library code to support this methodology. See this working demo.

2. Topic/Queue-Based Isolation​

Alternatively, use ResourcePlugins to create dedicated topics or queues for each sandbox. This requires only topic/queue name configuration for producers and consumers, but is less flexible than message-level routing.

System-Specific Guidance​

Kafka​

Kafka supports both message-level routing and topic-based isolation. The recommended approach is to use message headers to propagate a routing key (e.g., the sandbox or RouteGroup key).

Consumer Group Strategy:

  • Sandboxed consumers should create new, unique ConsumerGroups. This ensures that they receive a copy of all messages received by the baseline consumers, rather than competing for messages within the same group.
  • Each sandboxed consumer processes only messages with routing keys that match its own routing key or the routing key of any RouteGroups it is part of.
  • All other messages are processed by the baseline consumers (in the baseline ConsumerGroup).

Alternatively, use ResourcePlugins to provision sandbox-specific topics and configure the publishers and consumers to publish and consume from the new/temporary topics.

Summary:

  • Use message headers for routing key propagation.
  • Sandboxed consumers use unique ConsumerGroups to receive all messages.
  • Consumers filter messages by routing key.
  • Optionally, create sandbox-specific topics for full isolation.

RabbitMQ​

RabbitMQ isolation uses routing keys in message headers for selective message consumption, similar to Kafka and Pub/Sub, but adapted for RabbitMQ's queue-based model.

Solution Approach:

  • Publisher: The Signadot wrapper library extracts routing keys (e.g., from OpenTelemetry baggage) and injects them into RabbitMQ message headers during publishing. This works transparently with existing exchanges and routing patterns.
  • Consumer: The wrapper intercepts consume calls, connects to the Signadot Routes API to determine valid routing keys for the sandbox, and filters messages before passing them to business logic. Isolation is achieved by using a separate queue for each sandbox, not by rejecting messages.

Queue Management:

  • The sandboxed consumer always creates its own ephemeral queue in code. The wrapper replicates bindings from the baseline queue so the sandbox queue receives the same message types. The queue is automatically cleaned up when the sandboxed consumer shuts down.

Summary:

  • Use routing keys in headers for message-level isolation.
  • Signadot wrappers automate queue creation and message filtering.
  • Each sandboxed consumer creates and manages its own ephemeral queue in code.

Google Cloud Pub/Sub​

GCP Pub/Sub isolation is achieved by having each sandboxed consumer create its own subscription. This ensures the sandboxed consumer receives a copy of all messages published to the topic. The consumer then filters messages based on a routing key in the message attributes, processing only those intended for its sandbox or RouteGroup.

Alternatively, use ResourcePlugins to create sandbox-specific topics if needed.

Summary:

  • Each sandboxed consumer creates its own subscription to receive all messages.
  • Use message attributes for routing key propagation.
  • Consumers filter messages by attribute.
  • Optionally, create sandbox-specific topics/subscriptions.

AWS SQS/SNS​

For SQS, you can use either dedicated queues per sandbox (via ResourcePlugins) or a shared queue with message-level routing. For the latter, the consumer logic is as follows:

  1. Receive: Consumer polls and receives a message.
  2. Inspect: Consumer checks message attributes for the routing key.
  3. Decision:
    • If the message matches the sandbox's routing key, process and delete it.
    • If not, immediately call the SQS ChangeMessageVisibility API for that message with VisibilityTimeout=0 to make it instantly visible again for other consumers.

This approach avoids the need for separate queues for each consumer and enables efficient message isolation.

SQS with SNS:

  • If SQS is used together with SNS, a more efficient approach is to create a separate SQS queue for each sandboxed consumer and subscribe it to the same SNS topic. This ensures the sandboxed consumer receives all messages published to the topic, but does require creating a new queue for each sandboxed consumer.

Summary:

  • Use message attributes for routing key propagation.
  • Consumers filter and process only relevant messages.
  • Use ChangeMessageVisibility to return unmatched messages to the queue.
  • If using SNS, another option is to create a separate queue per sandboxed consumer for efficient isolation.

NATS​

Solution Approach:

  • Publisher: The Signadot wrapper extracts routing keys from OpenTelemetry baggage headers and injects them into NATS message headers during publishing. Messages are published to the same subjects as before—no routing changes needed.
  • Subscriber: The wrapper intercepts subscription handlers, connects to the Signadot Routes API to determine valid routing keys, and filters messages before passing them to the customer's message handlers. Both baseline and sandbox subscribers use the same subjects.

NATS Architecture Advantages:

  • Subject-Based Natural Fanout: NATS delivers a copy of each message to all subscribers on a subject. Both baseline and sandbox subscribers receive every message published to their subscribed subjects.

Summary:

  • Use message headers for routing key propagation and filtering.
  • Signadot wrappers automate message filtering for sandbox isolation.
  • No subject or routing changes required; NATS's fanout model ensures all subscribers receive all messages.