Test Sidekiq Jobs in Kubernetes with Signadot Sandboxes
A sandbox runs the version of a service you changed next to the baseline
in a shared Kubernetes cluster, such as staging, and a routing key decides which
version handles a request. Sidekiq breaks that in two ways. Nobody calls a
Sidekiq worker: it pulls jobs from Redis. And a Sidekiq queue gives each job to exactly one worker
(BRPOP), so there is no consumer group that would let every version see every job.
This tutorial shows how baseline and sandboxed Sidekiq workers share one Redis anyway. A web
service (Sinatra) enqueues ProcessMessageJob. The job runs on a worker Deployment and enqueues
NotifyJob on a second queue, and NotifyJob calls a notifier service. You fork each service
into a sandbox and check where each job runs, including scheduled jobs, retries, and jobs left
behind by a deleted sandbox.
Full source: signadot/examples/sidekiq-tutorial
How Sidekiq jobs reach the right worker
The request reaches web with the routing key in its baggage header, and web pushes a job that
carries the key to the shared default queue. The baseline worker fetches the job. If a sandbox
claims the job's routing key, the baseline moves the job to that sandbox's queue, where the
sandboxed worker runs it. Otherwise the baseline runs it. Three rules make this work:
- Each sandboxed worker reads its own queues, such as
default@sidekiq-worker. - The baseline worker is the only reader of the shared queues. It runs each job, unless a sandbox claims the job's routing key. Then it moves the job to that sandbox's queue.
- The routing key travels inside each job as OpenTelemetry baggage.
That gives each job a predictable home:
| Job | Runs on |
|---|---|
| No routing key | Baseline worker |
| Key of the worker sandbox, or of a route group that includes it | Sandboxed worker |
| Key of the notifier or web sandbox | Baseline worker, and the key still reaches that sandbox |
| Enqueued by another job | Same version as the job that enqueued it |
Scheduled with perform_in | Decided when the job comes due |
| Retry of a failed sandbox job | Back to the sandbox, through the shared queue |
| Left in the queue of a deleted sandbox | Baseline worker, after a 30-second grace period |
The Signadot-specific code is a small gem,
signadot_sidekiq,
of the kind a platform team ships. It uses Sidekiq's own extension points, server middleware and
startup hooks, because the routing decision happens where jobs are fetched. The job code stays
plain Sidekiq, and services that only enqueue jobs need nothing beyond OpenTelemetry.
Before you begin
- A Kubernetes cluster with the Signadot Operator installed.
- The Signadot CLI, logged in with
signadot auth login. kubectlpointed at that cluster, andjq.- Docker, and a container registry your cluster can pull from.
1. Build the image
The three services run from one image. Build it and push it to your registry:
git clone https://github.com/signadot/examples.git
cd examples/sidekiq-tutorial
export IMAGE=docker.io/your-user/sidekiq-demo:latest # your registry
docker build -t $IMAGE .
docker push $IMAGE
docker.io/your-user/... is a placeholder. When you build images of your own services this way,
push them to a private repository and give the cluster pull access to it, for example with an
image pull secret.
If your cluster's nodes have a different CPU architecture than your machine (for example an Apple
silicon Mac and an x86 cluster), add --platform linux/amd64 to docker build. On a local cluster
that runs images straight from your Docker daemon, you can skip the push.
2. Deploy the demo
Point the manifests at your image, then deploy Redis and the three services into the
sidekiq-demo namespace:
sed -i.orig "s|image: sidekiq-demo:latest|image: $IMAGE|" k8s/*.yaml
kubectl create namespace sidekiq-demo
kubectl apply -f k8s/
kubectl -n sidekiq-demo wait --for=condition=available deploy --all --timeout=180s
Connect your machine to the cluster, so that cluster service names resolve locally (this asks for your password):
export CLUSTER=my-cluster # your cluster's name, from: signadot cluster list
signadot local connect --cluster=$CLUSTER
3. Look at the app code
The app code is plain Sidekiq
The jobs know nothing about sandboxes:
class ProcessMessageJob
include Sidekiq::Job
def perform(id, body, fail_once = false)
# ...
body = body.upcase if PROCESSOR_VERSION == "v2"
NotifyJob.perform_async(id, body)
end
end
class NotifyJob
include Sidekiq::Job
sidekiq_options queue: "notifications"
# ...
end
The worker starts like any Sidekiq process, with
sidekiq -r ./app/worker.rb -q default -q notifications
(k8s/worker.yaml).
The one Signadot line is in the server configuration. In a Rails app it goes in
config/initializers/sidekiq.rb:
Sidekiq.configure_server do |config|
SignadotSidekiq.configure(config)
end
Outside Signadot, where neither SIGNADOT_SANDBOX_NAME nor SIGNADOT_BASELINE_NAME is set, that
call changes nothing.
OpenTelemetry carries the routing key into each job
app/otel.rb turns on
the standard OpenTelemetry instrumentation for Sinatra, Sidekiq and Net::HTTP. No exporter is
needed. Together they carry the W3C baggage header, which holds sd-routing-key, through every
hop, with no propagation code in the app:
- The Sinatra instrumentation reads it from the incoming request.
- The Sidekiq instrumentation's client middleware writes it into each job when the job is pushed, including a push from inside another job.
- Its server middleware makes it current again while the job runs.
- The
Net::HTTPinstrumentation adds it to the job's call to the notifier.
A job in Redis carries the key next to its arguments:
{"class":"ProcessMessageJob","queue":"default","args":["m6","hello m6",false],
"baggage":"sd-routing-key=2t6sgxtcffl9d","traceparent":"00-aad54fc8…-01", …}
For other languages and frameworks, see Header-Based Routing.
The gem then routes each job from that key. How signadot_sidekiq works,
after the walkthrough, goes through its code.
4. Enqueue a job without a routing key
Enqueue a job through the web service:
WEB=http://web.sidekiq-demo.svc:4000
curl -X POST $WEB/api/messages -H 'Content-Type: application/json' \
-d '{"id": "m1", "body": "hello m1"}'
Every service writes what it did to an event log that the web service serves. Define a helper that shows which version of each service handled a message:
trail() {
curl -s $WEB/api/events | jq -r --arg id "$1" '.[] | select(.messageId == $id)
| "\(.time[11:19]) \(.service) (\(if .sandbox == "" then "baseline" else .sandbox end)): \(.text)"'
}
trail m1
07:38:45 web (baseline): Enqueuing ProcessMessageJob
07:38:45 worker (baseline): ProcessMessageJob ran with processor v1: "hello m1"
07:38:45 worker (baseline): NotifyJob ran
07:38:45 notifier (baseline): Sent notification: "hello m1"
5. Create the sandboxes
Each sandbox forks one service. The worker sandbox also sets PROCESSOR_VERSION=v2, the "change"
under test, which uppercases message bodies:
name: sidekiq-worker
spec:
description: "worker sandbox for the Sidekiq demo (processor v2)"
cluster: "@{cluster}"
labels:
demo: sidekiq
forks:
- forkOf:
kind: Deployment
namespace: sidekiq-demo
name: worker
customizations:
env:
# The change under test: v2 uppercases message bodies.
- name: PROCESSOR_VERSION
value: v2
Apply the three sandboxes and a route group that bundles them under one routing key:
signadot sandbox apply -f signadot/sandboxes/worker.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/notifier.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/web.yaml --set cluster=$CLUSTER
signadot routegroup apply -f signadot/routegroups/sidekiq-demo.yaml --set cluster=$CLUSTER
Save the routing keys:
WORKER_KEY=$(signadot sandbox get sidekiq-worker -o json | jq -r .routingKey)
NOTIFIER_KEY=$(signadot sandbox get sidekiq-notifier -o json | jq -r .routingKey)
WEB_KEY=$(signadot sandbox get sidekiq-web -o json | jq -r .routingKey)
GROUP_KEY=$(signadot routegroup get sidekiq-demo -o json | jq -r .routingKey)
The sandboxed worker reads its own queues:
kubectl -n sidekiq-demo logs -l signadot.com/sandbox-routing-key=$WORKER_KEY | grep "reading queues"
INFO … sandbox sidekiq-worker: reading queues default@sidekiq-worker, notifications@sidekiq-worker
6. Enqueue jobs with routing keys
Define a helper that enqueues with a routing key in the baggage header, as a preview URL or the
Signadot browser extension would. A third argument adds fields to the request:
enqueue() {
curl -s -o /dev/null -X POST $WEB/api/messages -H 'Content-Type: application/json' \
-H "baggage: sd-routing-key=$2" -d "{\"id\": \"$1\", \"body\": \"hello $1\"${3:+, $3}}"
}
Enqueue one job per routing key:
enqueue m2 $WORKER_KEY
enqueue m3 $NOTIFIER_KEY
enqueue m4 $WEB_KEY
enqueue m5 $GROUP_KEY
sleep 2; for id in m2 m3 m4 m5; do trail $id; echo; done
07:39:13 web (baseline): Enqueuing ProcessMessageJob
07:39:13 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M2"
07:39:13 worker (sidekiq-worker): NotifyJob ran
07:39:13 notifier (baseline): Sent notification: "HELLO M2"
07:39:15 web (baseline): Enqueuing ProcessMessageJob
07:39:15 worker (baseline): ProcessMessageJob ran with processor v1: "hello m3"
07:39:15 worker (baseline): NotifyJob ran
07:39:15 notifier (sidekiq-notifier): Sent notification: "hello m3"
07:39:17 web (sidekiq-web): Enqueuing ProcessMessageJob
07:39:17 worker (baseline): ProcessMessageJob ran with processor v1: "hello m4"
07:39:17 worker (baseline): NotifyJob ran
07:39:17 notifier (baseline): Sent notification: "hello m4"
07:39:19 web (sidekiq-web): Enqueuing ProcessMessageJob
07:39:19 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M5"
07:39:19 worker (sidekiq-worker): NotifyJob ran
07:39:19 notifier (sidekiq-notifier): Sent notification: "HELLO M5"
Each job reached exactly the services its routing key selects:
| Routing key | web | worker (both jobs) | notifier |
|---|---|---|---|
| none (m1) | baseline | baseline (v1) | baseline |
| worker sandbox (m2) | baseline | sandbox (v2) | baseline |
| notifier sandbox (m3) | baseline | baseline (v1) | sandbox |
| web sandbox (m4) | sandbox | baseline (v1) | baseline |
| route group (m5) | sandbox | sandbox (v2) | sandbox |
The notifier sandbox (m3) shows that the routing key survives the queue: the baseline worker ran
both jobs, and its call to the notifier still landed on the sandbox, after two Redis hops. In m2
and m5, NotifyJob ran in the sandbox too, because a job enqueued by another job carries the same
routing key.
The baseline worker's log shows each hand-off:
kubectl -n sidekiq-demo logs deploy/worker | grep handed
INFO … class=ProcessMessageJob: handed ProcessMessageJob ca29a7c3… (routing key 2t6sgxtcffl9d) to default@sidekiq-worker
INFO … class=NotifyJob: handed NotifyJob 6311f98d… (routing key 2t6sgxtcffl9d) to notifications@sidekiq-worker
INFO … class=ProcessMessageJob: handed ProcessMessageJob 9e886289… (routing key 34rqkcc3s4jpw) to default@sidekiq-worker
INFO … class=NotifyJob: handed NotifyJob 6fa4dbcd… (routing key 34rqkcc3s4jpw) to notifications@sidekiq-worker
Your routing keys and job IDs will differ.
7. Test scheduled jobs and retries
Schedule a job 10 seconds out with perform_in, and enqueue one whose first attempt fails:
enqueue m6 $WORKER_KEY '"delay": 10'
enqueue m7 $WORKER_KEY '"failOnce": true'
sleep 30; trail m6; echo; trail m7
07:39:28 web (baseline): Scheduling ProcessMessageJob in 10s
07:39:43 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M6"
07:39:43 worker (sidekiq-worker): NotifyJob ran
07:39:43 notifier (baseline): Sent notification: "HELLO M6"
07:39:28 web (baseline): Enqueuing ProcessMessageJob
07:39:28 worker (sidekiq-worker): ProcessMessageJob failed on purpose; Sidekiq will retry it
07:39:34 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M7"
07:39:34 worker (sidekiq-worker): NotifyJob ran
07:39:34 notifier (baseline): Sent notification: "HELLO M7"
Both ran in the sandbox. When each came due (the scheduled job after 10 seconds, the retry after
its 5-second delay), Sidekiq pushed it to the shared default queue, and the baseline handed it to
the sandbox. Had the sandbox been deleted by then, the baseline would have run it.
8. Delete a sandbox while its jobs wait
Tell the sandboxed worker to stop taking new jobs, with Sidekiq's quiet signal (TSTP), so the
next job waits in its queue:
kubectl -n sidekiq-demo exec $(kubectl -n sidekiq-demo get pods -o name \
-l signadot.com/sandbox-routing-key=$WORKER_KEY) -- sh -c 'kill -TSTP 1'
enqueue m8 $WORKER_KEY; sleep 2
kubectl -n sidekiq-demo exec deploy/redis -- redis-cli LLEN queue:default@sidekiq-worker
1
Delete the sandbox. About 30 seconds after the sandbox disappears from the routing rules, the baseline worker moves the job back to the shared queue and runs it:
signadot sandbox delete sidekiq-worker
sleep 60; trail m8
07:40:58 web (baseline): Enqueuing ProcessMessageJob
07:41:41 worker (baseline): ProcessMessageJob ran with processor v1: "hello m8"
07:41:41 worker (baseline): NotifyJob ran
07:41:41 notifier (baseline): Sent notification: "hello m8"
The baseline worker's log shows the sweep:
kubectl -n sidekiq-demo logs deploy/worker | grep "is gone"
INFO … sandbox sidekiq-worker is gone: moved 1 job(s) from default@sidekiq-worker to default
INFO … sandbox sidekiq-worker is gone: moved 0 job(s) from notifications@sidekiq-worker to notifications
9. Clean up
Delete the route group and the remaining sandboxes, then the demo, and disconnect:
signadot routegroup delete sidekiq-demo
signadot sandbox delete sidekiq-notifier
signadot sandbox delete sidekiq-web
kubectl delete namespace sidekiq-demo
signadot local disconnect
With a Redis you keep, there is nothing else to remove: the baseline worker deletes the queues of deleted sandboxes on its own.
How signadot_sidekiq works
The gem does one of two things, depending on where the worker runs.
In a sandbox, where the Signadot Operator sets SIGNADOT_SANDBOX_NAME, it swaps each queue the
worker was started with for the sandbox's own, keeping weights and order:
config.capsules.each_value do |capsule|
capsule.queues = capsule.weights.map { |queue, weight| [sandbox_queue(queue, sandbox), weight] }
end
Nothing has to create these queues. A Sidekiq queue is a Redis list, which appears when the first job is pushed to it.
In the baseline, where SIGNADOT_BASELINE_NAME is set in k8s/worker.yaml, it adds a server
middleware that runs before every job:
def call(_job_instance, job, queue)
routing_key = SignadotSidekiq.routing_key(job)
sandbox = routing_key && @rules.sandbox_for(routing_key)
return yield unless sandbox
target = SignadotSidekiq.sandbox_queue(queue, sandbox)
Sidekiq::Client.push(job.merge("queue" => target, "retry_queue" => job["retry_queue"] || queue))
end
The rest of the gem covers what happens around that:
- Which keys are claimed.
RoutingRulesasks the Signadot Routes API every 5 seconds which sandboxes of theworkerDeployment claim which routing keys. A sandbox claims its own key and the keys of the route groups it is in. The baseline runs no job until it has read the rules once. - Retries.
retry_queuemakes Sidekiq retry a failed sandbox job on the shared queue, so the baseline decides again where the retry runs. - Scheduled jobs. Sidekiq pushes a
perform_injob to the shared queue when it comes due, so the decision happens then, not when the job was scheduled. - Deleted sandboxes.
Sweepermoves jobs left in a deleted sandbox's queues back to the shared queue, 30 seconds after the sandbox disappears from the routing rules.
The baseline worker tells the Routes API which workload it is through three environment variables:
env:
- name: SIGNADOT_BASELINE_KIND
value: Deployment
- name: SIGNADOT_BASELINE_NAME
value: worker
- name: SIGNADOT_BASELINE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
Use it in your own app
To apply the pattern to your own Sidekiq services:
- Propagate the routing key. Enable the OpenTelemetry Sidekiq instrumentation in every service that enqueues or runs jobs, next to the instrumentation for your web framework and HTTP client.
- Add the gem to your workers. Copy
signadot_sidekiqinto your codebase and callSignadotSidekiq.configure(config)insideSidekiq.configure_server. It requires Sidekiq 7.1 or later. - Identify the baseline. Set
SIGNADOT_BASELINE_KIND,SIGNADOT_BASELINE_NAMEandSIGNADOT_BASELINE_NAMESPACEon the baseline worker's Deployment. Sandboxed workers need no configuration: the Signadot Operator setsSIGNADOT_SANDBOX_NAMEin their pods.
Frequently asked questions
How do you test Sidekiq jobs in a shared Kubernetes cluster without a separate Redis per environment?
Give each sandboxed Sidekiq worker its own queues, such as default@my-sandbox, and let the baseline worker stay the only reader of the shared queues. The routing key travels inside each job as OpenTelemetry baggage. The baseline worker runs a job itself unless a sandbox claims the job's routing key, in which case it moves the job to that sandbox's queue. Baseline and sandboxed workers then share one Redis.
Can sandboxed Sidekiq workers read the shared queue and skip jobs that are not theirs?
No. Sidekiq fetches jobs with BRPOP, which removes the job from Redis and gives it to exactly one worker. A worker that skipped a job would lose it. That is why each sandboxed worker reads its own queues and the baseline worker moves jobs to them.
Does every service that enqueues Sidekiq jobs need the signadot_sidekiq gem?
No. Only Sidekiq worker processes load it. A service that enqueues jobs needs only the OpenTelemetry Sidekiq instrumentation, whose client middleware writes the routing key into each job.
Do scheduled jobs and retries run in the right sandbox?
Yes. Sidekiq pushes a perform_in job to the shared queue when it comes due, so the baseline worker decides at that moment where it runs. A failed sandbox job is retried on the shared queue as well, through the job's retry_queue, and the baseline hands it back to the sandbox if the sandbox still exists.
What happens to jobs left in the queue of a deleted sandbox?
The baseline worker runs them. About 30 seconds after a sandbox disappears from the routing rules, a sweeper in the baseline worker moves the jobs left in that sandbox's queues back to the shared queues and removes the sandbox queues from Redis.
Do baseline Sidekiq workers restart when sandboxes come and go?
No. The baseline worker reads the routing rules from the Signadot Routes API every 5 seconds, so it learns about new and deleted sandboxes while it runs.
How do I use this in a Rails app?
Call SignadotSidekiq.configure(config) inside Sidekiq.configure_server in config/initializers/sidekiq.rb, and enable the OpenTelemetry Sidekiq instrumentation. The gem uses only Sidekiq's server middleware and lifecycle hooks, nothing from Sinatra or Rails. It requires Sidekiq 7.1 or later.
See also
- Testing Redis Streams: selective consumption on Redis consumer groups
- Testing Temporal workers: the same idea for Temporal task queues
- Message Queue Isolation: the general approaches, by broker
- Header-Based Routing
- Route groups
- Sandbox specification