Message Queues
Message Queues in Sandboxes
Message queues introduce asynchronous communication between microservices which is realized by a message queue broker or service. Since communication between microservices is not direct, but rather passes via such a broker, one may be led to include the broker and producers in sandboxes whenever working on a message queue consumer microservice, as doing so would ensure that the sandbox consumer does not interfere with the baseline by consuming messages intended for the baseline.
However, this is not very practical either in terms of set up time or resources used and Signadot provides 2 more effective alternatives.
Using The Routing API
Many message queues, such as Kafka, support having multiple consumers of messages. In this "multi-fanout" scenario, each consumer consumes all messages.
When working with sandboxes involving message queues, one can dispatch messages conditionally to one consumer or another by:
- Decorating messages with a routing key in the header.
- Sending all messages to all consumers, such as Kafka consumer groups.
- Configuring each consumer to process or to not process each message as a function of the routing key of the message.
This approach is light weight and provides a great deal of flexibility. It works with RouteGroups, it allows independently testing consumers and producers in Sandboxes, and it allows sharing the message queue broker between the baseline and different sandboxes without interference.
Starting in v0.15.0, the Signadot Operator provides gRPC and REST APIs, and library code supporting this methodology. Additionally, a full working demo is available.
Topic Based Isolation
Another effective mechanism for working with message queues in Sandboxes is to use ResourcePlugins to create dedicated topics for each Sandbox. This approach may require a bit less set up, and is less invasive because the only configuration needed for producers and consumers is topic names. However, it is less flexible.