Testing SQS-Based Microservices with Signadot Sandboxes

You can test a new version of an Amazon SQS consumer on the same queue as the stable one, without a queue per developer. Every consumer version polls the shared queue and reads a routing key from each message's baggage attribute. A consumer that receives a message meant for another version calls ChangeMessageVisibility with a timeout of 0 so another consumer can receive it at once, and the version the message is meant for processes and deletes it. This tutorial runs that pattern on Kubernetes with a Python demo, forks the consumer into a Signadot sandbox, then adds SNS fan-out with a forked producer and a route group.

The same model works for Kafka, Google Pub/Sub and RabbitMQ; this page covers what is specific to SQS. Full source: signadot/examples/SQS-Based-Microservices-with-Signadot-main.

How it works

In a shared environment, SQS hands each message to whichever consumer polls first, so a developer's test message can land on the stable consumer, and other people's messages can land on the version under test. A routing key on every message, and one rule every consumer applies to it, fixes that:

1. A request carries the routing key in its W3C baggage header, as sd-routing-key=<key>. OpenTelemetry instrumentation passes the header from service to service and copies it into a baggage message attribute on the message the producer sends.
2. The baseline consumer and every sandboxed consumer poll the same queue.
3. Whichever consumer receives a message checks whether its routing key is one of its own. If not, it sets the message's visibility timeout to 0, which makes the message visible again immediately, and polls again. If so, it processes the message and deletes it.

Here a message carries the routing key K of a sandbox that forks the consumer, and the baseline consumer happens to receive it first:

The receive loop

The consumer in apps/consumer/app.py asks for the baggage attribute by name (FILTER_ATTRIBUTE_NAME), because SQS returns message attributes only when you request them:

apps/consumer/app.py

response = sqs_client.receive_message(
    QueueUrl=sqs_queue_url,
    MaxNumberOfMessages=1,
    MessageAttributeNames=[FILTER_ATTRIBUTE_NAME],
    VisibilityTimeout=60,
    WaitTimeSeconds=20,
)

if "Messages" in response:
    message = response["Messages"][0]
    message_body_dict = json.loads(message["Body"])
    message_attributes = message.get("MessageAttributes", {})

    routing_key = extract_routing_key_from_baggage(message_attributes)

    if not router_api.should_process(routing_key):
        # This message is not for this consumer instance. Make it immediately visible again for other consumers.
        sqs_client.change_message_visibility(
            QueueUrl=sqs_queue_url,
            ReceiptHandle=message["ReceiptHandle"],
            VisibilityTimeout=0,
        )
        continue

    # ... process the message
    sqs_client.delete_message(
        QueueUrl=sqs_queue_url, ReceiptHandle=message["ReceiptHandle"]
    )

extract_routing_key_from_baggage in modules/otel/baggage.py parses the attribute with OpenTelemetry's W3C baggage propagator and returns its sd-routing-key entry. VisibilityTimeout=60 hides a received message from other consumers while this one decides, and the release ends that early. WaitTimeSeconds=20 turns on long polling: an idle consumer waits up to 20 seconds for a message instead of returning empty-handed.

How a consumer knows which routing keys are its own

Each consumer asks the Signadot Operator's Routes API which routing keys apply to its workload, and caches the answer. RoutesAPIClient in modules/pull_router/router_api.py refreshes the cache every five seconds, on a background thread, from the route server's REST endpoint:

GET http://routeserver.signadot.svc:7778/api/v1/workloads/routing-rules
    ?baselineKind=Deployment&baselineNamespace=aws-sqs-app&baselineName=consumer-deployment
    [&destinationSandboxName=<sandbox name>]

A sandboxed consumer adds destinationSandboxName, so it gets only the keys routed to its own sandbox: the sandbox's own key and the key of every route group that contains it. The baseline leaves it out and gets every key claimed by any sandbox that forks consumer-deployment. The decision then fits in a few lines:

modules/pull_router/router_api.py

def should_process(self, routing_key: Optional[str]) -> bool:
    current_cached_keys = self._routing_keys_cache

    if self.sandbox_name: # This is a sandboxed workload
        if routing_key is None:
            return False
        should = routing_key in current_cached_keys
        # ...
        return should
    else: # This is a baseline workload
        if routing_key is None:
            return True
        should = routing_key not in current_cached_keys
        # ...
        return should

A sandboxed consumer processes only its own keys. The baseline processes messages with no key, and messages whose key no consumer sandbox claims, such as the key of a sandbox that forks only the producer. Every message has exactly one owner.

The client finds the route server through ROUTES_API_ROUTE_SERVER_ADDR, set in k8s/consumer.yaml, and identifies its workload with BASELINE_NAMESPACE, BASELINE_NAME and SIGNADOT_SANDBOX_NAME. That last one is empty in the baseline; the Operator sets it to the sandbox's name in forked pods.

How the routing key gets into the message

The producer never sets message attributes itself. main.py starts the producer and the frontend under opentelemetry-instrument: the FastAPI and httpx instrumentations carry the baggage header from the browser to the producer, and the boto3 SQS and botocore instrumentations inject it into the message attributes of every SendMessage and SNS Publish. The producer's send_message and publish calls stay plain.

SQS allows at most 10 message attributes per message, and the instrumentation adds traceparent and baggage only while there is room, so leave space for them in your own messages. If a consumer calls other services or sends messages of its own, it should pass the baggage on too, so the routing key reaches the next hop. See context propagation.

SQS alone, SNS fan-out, or a queue per sandbox

SNS is not required. Everything above works with SQS alone. The demo's SNS scenario publishes to the sns-signadot topic, which delivers into the same shared queue, so the consumers behave exactly as with plain SQS; only the producer changes. The subscription uses raw message delivery, which passes SNS message attributes through as SQS message attributes. Without it, baggage would arrive inside the JSON envelope in the message body, and the consumer would not find it.

SNS also allows a different design: give each sandboxed consumer its own queue, subscribed to the topic. Every consumer version then receives its own copy of every message, much like a Kafka consumer group per sandbox, and deletes the copies that are not its own instead of releasing them.

	Shared queue (SQS alone, or SNS into one queue)	Queue per sandbox (SNS fan-out)
AWS resources per sandbox	None	One queue and one SNS subscription
Who receives a message	One consumer at a time, until the owner does	Every consumer version, from its own queue
A consumer that is not the owner	Releases it with ChangeMessageVisibility 0	Deletes its own copy
Producers	Send to the queue or publish to the topic	Must publish to the topic
Extra requests	A receive and a release each time the wrong consumer gets it first	Each queue receives and deletes every message
FIFO queues	A message held by the wrong consumer holds up its message group	Each queue orders its own copy
Dead-letter queues	Releases count toward maxReceiveCount	Other versions do not affect it
In the demo	Steps 4 to 7	Not included

For that design, the amazon-sqs resource plugin creates a queue with each sandbox and deletes it with the sandbox (see resource plugins); the forked consumer reads the queue's name from the resource's output. Two code changes complete the pattern: consumers delete copies that are not theirs instead of releasing them (a released copy would only come back to the same consumer), and producers publish to the topic instead of sending to a queue. The plugin does not manage SNS subscriptions, so subscribe the queue at startup, as the demo's main.py does, and remove the subscription when the sandbox goes away.

Tutorial: test an SQS consumer and an SNS producer in sandboxes

The demo runs four workloads in the aws-sqs-app namespace:

• Frontend: a web UI for sending messages and seeing which version of each service handled them.
• Producer: sends each message to the sqs-signadot queue, or publishes it to the sns-signadot topic when SNS_FANOUT_PUBLISH is set.
• Consumer: polls the queue and processes or releases each message.
• Redis: stores the event log the frontend shows.

The producer and consumer pods run the Signadot DevMesh sidecar (sidecar.signadot.com/inject: "true"), which routes requests carrying a sandbox's routing key to that sandbox's fork.

Estimated time: 30 minutes.

Before you begin

You need:

• A Kubernetes cluster with the Signadot Operator v0.15.0 or later installed. The consumer uses the Operator's Routes API. The demo targets Minikube (minikube start --driver=docker), but any cluster works if you push the image to a registry it can pull from.
• The Signadot CLI, logged in with signadot auth login.
• kubectl, Docker, git and jq, plus the AWS CLI to check and clean up the AWS resources.
• An AWS account where you can create an IAM user and an access key, or LocalStack (see step 1).
• The Signadot Chrome extension to choose which sandbox your browser's requests go to, or curl.

Step 1: Clone the demo and add AWS credentials

git clone https://github.com/signadot/examples.git
cd examples/SQS-Based-Microservices-with-Signadot-main

The producer and consumer call SQS and SNS with an IAM user's access key. The demo's README attaches the AWS managed policies AmazonSQSFullAccess and AmazonSNSFullAccess. For a narrower policy, these are the actions the demo calls:

Action	Called by	For
sqs:CreateQueue, sns:CreateTopic	Producer and consumer, at startup	Create sqs-signadot and sns-signadot if they don't exist
sqs:GetQueueAttributes, sqs:SetQueueAttributes	Producer and consumer, at startup	Read the queue ARN and set the policy that lets the topic send to the queue
sns:Subscribe	Producer and consumer, at startup	Subscribe the queue to the topic
sqs:SendMessage	Producer	Send to the queue
sns:Publish	Producer, when SNS_FANOUT_PUBLISH is set	Publish to the topic
sqs:ReceiveMessage	Consumer	Poll the queue
sqs:ChangeMessageVisibility	Consumer	Release a message meant for another version
sqs:DeleteMessage	Consumer	Remove a message after processing it

Base64-encode the access key ID and secret access key:

echo -n 'YOUR_AWS_ACCESS_KEY_ID' | base64
echo -n 'YOUR_AWS_SECRET_ACCESS_KEY' | base64

Put the two values in k8s/secrets.yaml, and don't commit the edited file:

k8s/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: aws-sqs-app
type: Opaque
data:
  AWS_ACCESS_KEY_ID: < YOUR_BASE64_ENCODED_ACCESS_KEY >
  AWS_SECRET_ACCESS_KEY: < YOUR_BASE64_ENCODED_SECRET_KEY >

The region, queue name and topic name are in k8s/configmap.yaml: us-east-1, sqs-signadot and sns-signadot.

Using LocalStack instead of AWS. LocalStack emulates SQS and SNS. The demo has no LocalStack setting of its own, but boto3 reads the standard AWS_ENDPOINT_URL environment variable, so you can run LocalStack where the pods can reach it and add AWS_ENDPOINT_URL: "http://<localstack-host>:4566" to k8s/configmap.yaml. LocalStack accepts any credentials, so any base64-encoded values work in the Secret.

Step 2: Build the image

The frontend, producer and consumer all run from one image, sqs-signadot:latest. On Minikube, build it inside Minikube's Docker daemon, so the cluster can use it without a registry:

eval $(minikube docker-env)
docker build -t sqs-signadot:latest .

On another cluster, push the image to a registry your cluster can pull from, and set image: in k8s/frontend.yaml, k8s/producer.yaml and k8s/consumer.yaml to match. If you build on an Apple silicon Mac for an x86 cluster, add --platform linux/amd64 to docker build.

Step 3: Deploy the demo and send a baseline message

Create the namespace and apply every manifest in k8s/: the Secret, the ConfigMap, Redis, and the three services.

kubectl create namespace aws-sqs-app
kubectl apply -f k8s/
kubectl -n aws-sqs-app wait --for=condition=available deployment --all --timeout=5m

On startup, the producer and consumer each create the queue and topic if they don't exist, set a queue policy that lets the topic send to the queue, and subscribe the queue to the topic with raw message delivery. Check the result with the AWS CLI:

aws sqs get-queue-url --queue-name sqs-signadot --region us-east-1
aws sns list-subscriptions --region us-east-1 \
  --query "Subscriptions[?ends_with(TopicArn, ':sns-signadot')].[Protocol,Endpoint]" --output text

The second command should print sqs and the ARN of the sqs-signadot queue.

Forward the frontend to your machine:

kubectl -n aws-sqs-app port-forward service/frontend-service 8080:8080

Open http://localhost:8080, enter a message body and click Publish. With no sandboxes yet, the message carries no routing key, and the baseline consumer processes it. The frontend's log lists the newest entry first:

Request Scope > msgID: 4941-1, msgBody: "test", routingKey: ""
21:24:37.013  consumer-deployment (baseline)  Consumed message from sqs queue.
21:24:36.024  producer-deployment (baseline)  Sending produce request to SQS queue.
21:24:35.937  frontend-deployment (baseline)  Sending produce request to producer API.
21:24:36.178  browser                         Sending publish request to frontend API.

Step 4: Fork the consumer into a sandbox

The sandbox spec forks consumer-deployment and changes nothing else; the routing logic is already in the consumer. The name's spelling, sqs-counsumer-sandbox, is the repository's. The label puts the sandbox in the route group you create in step 7.

sandbox/sqs_sandbox.yaml

apiVersion: signadot.com/v1
kind: Sandbox
name: sqs-counsumer-sandbox
spec:
  labels:
    app: "sns-sqs-fanout-sandbox"
  description: Isolated sandbox environments to enable sqs messgae routing
  cluster: "@{cluster}"
  forks:
  - forkOf:
      kind: Deployment
      name: consumer-deployment
      namespace: aws-sqs-app

Set CLUSTER to your cluster's name from signadot cluster list, and create the sandbox:

export CLUSTER=my-cluster
signadot sandbox apply -f sandbox/sqs_sandbox.yaml --set cluster=$CLUSTER

The Operator sets SIGNADOT_SANDBOX_NAME in the forked consumer, so its Routes API client asks for the keys routed to sqs-counsumer-sandbox. Within five seconds the baseline consumer has the new key in its cache too:

kubectl -n aws-sqs-app logs deploy/consumer-deployment -c consumer | grep "Routing keys updated" | tail -1

... - modules.pull_router.router_api - INFO - RoutesAPIClient: Routing keys updated: ['<sandbox routing key>']

Step 5: Send messages with and without the routing key

Select sqs-counsumer-sandbox in the Signadot Chrome extension and publish another message. The baseline producer sends it with the sandbox's routing key, and the sandboxed consumer processes it:

Request Scope > msgID: 4941-3, msgBody: "test", routingKey: 809bblns7gu1q
21:29:16.680  consumer-deployment (sandbox=sqs-counsumer-sandbox)  Consumed message from sqs queue.
21:29:15.742  producer-deployment (baseline)                       Sending produce request to SQS queue.
21:29:15.718  frontend-deployment (baseline)                       Sending produce request to producer API.
21:29:15.961  browser                                              Sending publish request to frontend API.

To do the same from a terminal, put the routing key in the baggage header yourself:

SBX_KEY=$(signadot sandbox get sqs-counsumer-sandbox -o json | jq -r .routingKey)
curl -s -X POST http://localhost:8080/api/produce \
  -H 'Content-Type: application/json' \
  -H "baggage: sd-routing-key=$SBX_KEY" \
  -d '{"id": "cli-1", "body": "for the sandbox"}'

Then read both consumers' decisions in their logs:

SBX_DEPLOY=$(kubectl -n aws-sqs-app get deploy -o name | grep sqs-counsumer-sandbox)
kubectl -n aws-sqs-app logs $SBX_DEPLOY -c consumer | grep -E "Processing message|Skipping message"
kubectl -n aws-sqs-app logs deploy/consumer-deployment -c consumer | grep -E "Processing message|Skipping message"

The sandboxed consumer logs the message it processed. If the baseline consumer received the message first, its log also shows the release:

... - apps.consumer.app - INFO - Processing message: {'id': 'cli-1', 'body': 'for the sandbox'}
... - apps.consumer.app - INFO - Skipping message with routing_key: '<sandbox routing key>'. Releasing back to queue.

Now clear the sandbox in the extension (or drop the baggage header from the curl command) and send again. The baseline consumer processes the message. If the sandboxed consumer receives it first, it logs Skipping message with routing_key: 'None' and releases it.

Step 6: Publish through SNS from a forked producer

A producer change can be tested against the stable consumer. This sandbox forks the producer and sets SNS_FANOUT_PUBLISH, so the fork publishes to the topic instead of sending to the queue:

sandbox/sns_sandbox.yaml

apiVersion: signadot.com/v1
kind: Sandbox
name: sns-sqs-fanout-sandbox
spec:
  labels:
    app: "sns-sqs-fanout-sandbox"
  description: Isolated sandbox environments to enable sns to sqs fanout routing
  cluster: "@{cluster}"
  forks:
  - forkOf:
      kind: Deployment
      name: producer-deployment
      namespace: aws-sqs-app
    customizations:
      env:
        - name: SNS_FANOUT_PUBLISH
          value: "true"

signadot sandbox apply -f sandbox/sns_sandbox.yaml --set cluster=$CLUSTER

Select sns-sqs-fanout-sandbox in the extension and publish. The DevMesh sidecar routes the request to the forked producer, which publishes to the topic with the producer sandbox's routing key, and the topic delivers the message into the shared queue. No consumer sandbox claims that key, so the baseline consumer processes the message:

Request Scope > msgID: 4941-4, msgBody: "test", routingKey: kz9ffuv61yl9b
21:30:42.849  consumer-deployment (baseline)                        Consumed message from sqs queue.
21:30:41.848  producer-deployment (sandbox=sns-sqs-fanout-sandbox)  Sending produce request to SNS topic.
21:30:41.812  frontend-deployment (baseline)                        Sending produce request to producer API.
21:30:42.056  browser                                               Sending publish request to frontend API.

Step 7: Test the producer and consumer together with a route group

Real changes often span a producer and a consumer. Both sandboxes carry the label app: sns-sqs-fanout-sandbox, which this route group matches, so the group gives them one shared routing key:

sandbox/sns-sqs-router-grp.yaml

name: sns-sqs-router-grp
spec:
  cluster: "@{cluster}"
  description: "route group for testing multiple sandboxes together"
  match:
    any:
    - label:
        key: app
        value: sns-sqs-fanout-sandbox

signadot routegroup apply -f sandbox/sns-sqs-router-grp.yaml --set cluster=$CLUSTER

Select sns-sqs-router-grp in the extension and publish. The request reaches the forked producer, which publishes to the topic with the route group's routing key. The consumer sandbox claims that key as a member of the group, so the forked consumer processes the message, while the baseline handles every other message:

Request Scope > msgID: 4941-5, msgBody: "test", routingKey: s3w3znxvw1960
21:32:25.519  consumer-deployment (sandbox=sqs-counsumer-sandbox)   Consumed message from sqs queue.
21:32:24.260  producer-deployment (sandbox=sns-sqs-fanout-sandbox)  Sending produce request to SNS topic.
21:32:24.234  frontend-deployment (baseline)                        Sending produce request to producer API.
21:32:24.438  browser                                               Sending publish request to frontend API.

Clean up

Delete the route group, the sandboxes and the namespace:

signadot routegroup delete sns-sqs-router-grp
signadot sandbox delete sns-sqs-fanout-sandbox
signadot sandbox delete sqs-counsumer-sandbox
kubectl delete namespace aws-sqs-app

The queue and topic outlive the namespace. Delete them too; deleting a topic also deletes its subscriptions:

QUEUE_URL=$(aws sqs get-queue-url --queue-name sqs-signadot --region us-east-1 --query QueueUrl --output text)
TOPIC_ARN=$(aws sns list-topics --region us-east-1 \
  --query "Topics[?ends_with(TopicArn, ':sns-signadot')].TopicArn" --output text)
aws sqs delete-queue --queue-url "$QUEUE_URL" --region us-east-1
aws sns delete-topic --topic-arn "$TOPIC_ARN" --region us-east-1

If you created the IAM user only for this tutorial, delete its access key as well.

Frequently asked questions

How do you test SQS consumers in Kubernetes without a queue per developer?

Run the version under test as a sandboxed consumer on the same queue as the baseline consumer. Each message carries a routing key in its baggage message attribute, and every consumer checks that key against Signadot's Routes API: a consumer that receives a message meant for another version calls ChangeMessageVisibility with a timeout of 0, and the version the message is meant for processes and deletes it. No queue, topic or copy of the environment is created per developer.

Do you need SNS to isolate SQS consumers?

No. With SQS alone, every consumer version polls the shared queue and releases messages that are not its own, so the right consumer can receive them. SNS adds a second option: subscribe a separate queue per sandboxed consumer to a topic, so every version gets its own copy and deletes the copies it does not want, at the cost of creating a queue and a subscription per sandbox.

Do released messages count toward a dead-letter queue's maxReceiveCount?

Yes. SQS increments a message's ApproximateReceiveCount on every receive, including receives that end in a release, and moves the message to the dead-letter queue once the count exceeds the redrive policy's maxReceiveCount. On a shared queue a message can be received and released a few times before its owner gets it, so set maxReceiveCount comfortably above the number of consumer versions that poll the queue.

Does the shared-queue pattern work with SQS FIFO queues?

It works, but releases slow ordered delivery. While a message from a FIFO message group is in flight, SQS returns no other messages from that group, so a message held by the wrong consumer holds up its group until it is released, and a sandbox message at the head of a group holds up the baseline messages behind it until the sandboxed consumer takes it. If sandbox and baseline messages share message groups, a queue per sandbox avoids that coupling.

How do you test SQS locally?

Run the whole demo on Minikube on your laptop, as this tutorial does. It talks to real SQS and SNS by default; to stay off AWS, run LocalStack, which emulates SQS and SNS, and set the AWS_ENDPOINT_URL environment variable, which boto3 reads, in the demo's ConfigMap. The routing keys travel in SQS message attributes either way, so the Signadot side does not change.

How much do the extra receives cost?

Every SQS API action is billed as a request, so each release adds one ReceiveMessage and one ChangeMessageVisibility to the usual send, receive and delete. Idle consumers also make requests: with long polling at the 20-second maximum, as in this demo, an idle consumer makes about 130,000 receive requests a month, far fewer than with short polling. Deleting sandboxes you no longer use stops their consumers polling.

See also

• AWS SQS/SNS isolation guide: the consumer logic in brief, and the queue-per-sandbox option
• Testing Kafka-based microservices: isolation strategies compared, and the same routing-key contract with consumer groups
• Google Pub/Sub tutorial: a subscription per sandbox, the Pub/Sub counterpart of a queue per sandbox
• RabbitMQ tutorial: baseline and sandboxed consumers on one broker
• Context propagation: getting the routing key through every service hop
• Resource plugins: provision per-sandbox resources, such as a queue, with each sandbox
• Route groups: test changes that span several sandboxes together