Skip to main content

Test Sidekiq Jobs in Kubernetes with Signadot Sandboxes

A sandbox runs the version of a service you changed next to the baseline in a shared Kubernetes cluster, such as staging, and a routing key decides which version handles a request. Sidekiq breaks that in two ways. Nobody calls a Sidekiq worker: it pulls jobs from Redis. And a Sidekiq queue gives each job to exactly one worker (BRPOP), so there is no consumer group that would let every version see every job.

This tutorial shows how baseline and sandboxed Sidekiq workers share one Redis anyway. A web service (Sinatra) enqueues ProcessMessageJob. The job runs on a worker Deployment and enqueues NotifyJob on a second queue, and NotifyJob calls a notifier service. You fork each service into a sandbox and check where each job runs, including scheduled jobs, retries, and jobs left behind by a deleted sandbox.

Full source: signadot/examples/sidekiq-tutorial

How Sidekiq jobs reach the right worker​

The request reaches web with the routing key in its baggage header, and web pushes a job that carries the key to the shared default queue. The baseline worker fetches the job. If a sandbox claims the job's routing key, the baseline moves the job to that sandbox's queue, where the sandboxed worker runs it. Otherwise the baseline runs it. Three rules make this work:

  1. Each sandboxed worker reads its own queues, such as default@sidekiq-worker.
  2. The baseline worker is the only reader of the shared queues. It runs each job, unless a sandbox claims the job's routing key. Then it moves the job to that sandbox's queue.
  3. The routing key travels inside each job as OpenTelemetry baggage.

That gives each job a predictable home:

JobRuns on
No routing keyBaseline worker
Key of the worker sandbox, or of a route group that includes itSandboxed worker
Key of the notifier or web sandboxBaseline worker, and the key still reaches that sandbox
Enqueued by another jobSame version as the job that enqueued it
Scheduled with perform_inDecided when the job comes due
Retry of a failed sandbox jobBack to the sandbox, through the shared queue
Left in the queue of a deleted sandboxBaseline worker, after a 30-second grace period

The Signadot-specific code is a small gem, signadot_sidekiq, of the kind a platform team ships. It uses Sidekiq's own extension points, server middleware and startup hooks, because the routing decision happens where jobs are fetched. The job code stays plain Sidekiq, and services that only enqueue jobs need nothing beyond OpenTelemetry.

Before you begin​

Prerequisites
  • A Kubernetes cluster with the Signadot Operator installed.
  • The Signadot CLI, logged in with signadot auth login.
  • kubectl pointed at that cluster, and jq.
  • Docker, and a container registry your cluster can pull from.

1. Build the image​

The three services run from one image. Build it and push it to your registry:

git clone https://github.com/signadot/examples.git
cd examples/sidekiq-tutorial

export IMAGE=docker.io/your-user/sidekiq-demo:latest # your registry
docker build -t $IMAGE .
docker push $IMAGE
Any registry your cluster can pull from works

docker.io/your-user/... is a placeholder. When you build images of your own services this way, push them to a private repository and give the cluster pull access to it, for example with an image pull secret.

If your cluster's nodes have a different CPU architecture than your machine (for example an Apple silicon Mac and an x86 cluster), add --platform linux/amd64 to docker build. On a local cluster that runs images straight from your Docker daemon, you can skip the push.

2. Deploy the demo​

Point the manifests at your image, then deploy Redis and the three services into the sidekiq-demo namespace:

sed -i.orig "s|image: sidekiq-demo:latest|image: $IMAGE|" k8s/*.yaml

kubectl create namespace sidekiq-demo
kubectl apply -f k8s/
kubectl -n sidekiq-demo wait --for=condition=available deploy --all --timeout=180s

Connect your machine to the cluster, so that cluster service names resolve locally (this asks for your password):

export CLUSTER=my-cluster  # your cluster's name, from: signadot cluster list
signadot local connect --cluster=$CLUSTER

3. Look at the app code​

The app code is plain Sidekiq​

The jobs know nothing about sandboxes:

app/jobs.rb
class ProcessMessageJob
include Sidekiq::Job

def perform(id, body, fail_once = false)
# ...
body = body.upcase if PROCESSOR_VERSION == "v2"
NotifyJob.perform_async(id, body)
end
end

class NotifyJob
include Sidekiq::Job
sidekiq_options queue: "notifications"
# ...
end

The worker starts like any Sidekiq process, with sidekiq -r ./app/worker.rb -q default -q notifications (k8s/worker.yaml). The one Signadot line is in the server configuration. In a Rails app it goes in config/initializers/sidekiq.rb:

app/worker.rb
Sidekiq.configure_server do |config|
SignadotSidekiq.configure(config)
end

Outside Signadot, where neither SIGNADOT_SANDBOX_NAME nor SIGNADOT_BASELINE_NAME is set, that call changes nothing.

OpenTelemetry carries the routing key into each job​

app/otel.rb turns on the standard OpenTelemetry instrumentation for Sinatra, Sidekiq and Net::HTTP. No exporter is needed. Together they carry the W3C baggage header, which holds sd-routing-key, through every hop, with no propagation code in the app:

  • The Sinatra instrumentation reads it from the incoming request.
  • The Sidekiq instrumentation's client middleware writes it into each job when the job is pushed, including a push from inside another job.
  • Its server middleware makes it current again while the job runs.
  • The Net::HTTP instrumentation adds it to the job's call to the notifier.

A job in Redis carries the key next to its arguments:

{"class":"ProcessMessageJob","queue":"default","args":["m6","hello m6",false],
"baggage":"sd-routing-key=2t6sgxtcffl9d","traceparent":"00-aad54fc8…-01", …}

For other languages and frameworks, see Header-Based Routing.

The gem then routes each job from that key. How signadot_sidekiq works, after the walkthrough, goes through its code.

4. Enqueue a job without a routing key​

Enqueue a job through the web service:

WEB=http://web.sidekiq-demo.svc:4000
curl -X POST $WEB/api/messages -H 'Content-Type: application/json' \
-d '{"id": "m1", "body": "hello m1"}'

Every service writes what it did to an event log that the web service serves. Define a helper that shows which version of each service handled a message:

trail() {
curl -s $WEB/api/events | jq -r --arg id "$1" '.[] | select(.messageId == $id)
| "\(.time[11:19]) \(.service) (\(if .sandbox == "" then "baseline" else .sandbox end)): \(.text)"'
}
trail m1
Output
07:38:45 web (baseline): Enqueuing ProcessMessageJob
07:38:45 worker (baseline): ProcessMessageJob ran with processor v1: "hello m1"
07:38:45 worker (baseline): NotifyJob ran
07:38:45 notifier (baseline): Sent notification: "hello m1"

5. Create the sandboxes​

Each sandbox forks one service. The worker sandbox also sets PROCESSOR_VERSION=v2, the "change" under test, which uppercases message bodies:

signadot/sandboxes/worker.yaml
name: sidekiq-worker
spec:
description: "worker sandbox for the Sidekiq demo (processor v2)"
cluster: "@{cluster}"
labels:
demo: sidekiq
forks:
- forkOf:
kind: Deployment
namespace: sidekiq-demo
name: worker
customizations:
env:
# The change under test: v2 uppercases message bodies.
- name: PROCESSOR_VERSION
value: v2

Apply the three sandboxes and a route group that bundles them under one routing key:

signadot sandbox apply -f signadot/sandboxes/worker.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/notifier.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/web.yaml --set cluster=$CLUSTER
signadot routegroup apply -f signadot/routegroups/sidekiq-demo.yaml --set cluster=$CLUSTER

Save the routing keys:

WORKER_KEY=$(signadot sandbox get sidekiq-worker -o json | jq -r .routingKey)
NOTIFIER_KEY=$(signadot sandbox get sidekiq-notifier -o json | jq -r .routingKey)
WEB_KEY=$(signadot sandbox get sidekiq-web -o json | jq -r .routingKey)
GROUP_KEY=$(signadot routegroup get sidekiq-demo -o json | jq -r .routingKey)

The sandboxed worker reads its own queues:

kubectl -n sidekiq-demo logs -l signadot.com/sandbox-routing-key=$WORKER_KEY | grep "reading queues"
Output
INFO  … sandbox sidekiq-worker: reading queues default@sidekiq-worker, notifications@sidekiq-worker

6. Enqueue jobs with routing keys​

Define a helper that enqueues with a routing key in the baggage header, as a preview URL or the Signadot browser extension would. A third argument adds fields to the request:

enqueue() {
curl -s -o /dev/null -X POST $WEB/api/messages -H 'Content-Type: application/json' \
-H "baggage: sd-routing-key=$2" -d "{\"id\": \"$1\", \"body\": \"hello $1\"${3:+, $3}}"
}

Enqueue one job per routing key:

enqueue m2 $WORKER_KEY
enqueue m3 $NOTIFIER_KEY
enqueue m4 $WEB_KEY
enqueue m5 $GROUP_KEY
sleep 2; for id in m2 m3 m4 m5; do trail $id; echo; done
Output
07:39:13 web (baseline): Enqueuing ProcessMessageJob
07:39:13 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M2"
07:39:13 worker (sidekiq-worker): NotifyJob ran
07:39:13 notifier (baseline): Sent notification: "HELLO M2"

07:39:15 web (baseline): Enqueuing ProcessMessageJob
07:39:15 worker (baseline): ProcessMessageJob ran with processor v1: "hello m3"
07:39:15 worker (baseline): NotifyJob ran
07:39:15 notifier (sidekiq-notifier): Sent notification: "hello m3"

07:39:17 web (sidekiq-web): Enqueuing ProcessMessageJob
07:39:17 worker (baseline): ProcessMessageJob ran with processor v1: "hello m4"
07:39:17 worker (baseline): NotifyJob ran
07:39:17 notifier (baseline): Sent notification: "hello m4"

07:39:19 web (sidekiq-web): Enqueuing ProcessMessageJob
07:39:19 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M5"
07:39:19 worker (sidekiq-worker): NotifyJob ran
07:39:19 notifier (sidekiq-notifier): Sent notification: "HELLO M5"

Each job reached exactly the services its routing key selects:

Routing keywebworker (both jobs)notifier
none (m1)baselinebaseline (v1)baseline
worker sandbox (m2)baselinesandbox (v2)baseline
notifier sandbox (m3)baselinebaseline (v1)sandbox
web sandbox (m4)sandboxbaseline (v1)baseline
route group (m5)sandboxsandbox (v2)sandbox

The notifier sandbox (m3) shows that the routing key survives the queue: the baseline worker ran both jobs, and its call to the notifier still landed on the sandbox, after two Redis hops. In m2 and m5, NotifyJob ran in the sandbox too, because a job enqueued by another job carries the same routing key.

The baseline worker's log shows each hand-off:

kubectl -n sidekiq-demo logs deploy/worker | grep handed
Output
INFO  … class=ProcessMessageJob: handed ProcessMessageJob ca29a7c3… (routing key 2t6sgxtcffl9d) to default@sidekiq-worker
INFO … class=NotifyJob: handed NotifyJob 6311f98d… (routing key 2t6sgxtcffl9d) to notifications@sidekiq-worker
INFO … class=ProcessMessageJob: handed ProcessMessageJob 9e886289… (routing key 34rqkcc3s4jpw) to default@sidekiq-worker
INFO … class=NotifyJob: handed NotifyJob 6fa4dbcd… (routing key 34rqkcc3s4jpw) to notifications@sidekiq-worker

Your routing keys and job IDs will differ.

7. Test scheduled jobs and retries​

Schedule a job 10 seconds out with perform_in, and enqueue one whose first attempt fails:

enqueue m6 $WORKER_KEY '"delay": 10'
enqueue m7 $WORKER_KEY '"failOnce": true'
sleep 30; trail m6; echo; trail m7
Output
07:39:28 web (baseline): Scheduling ProcessMessageJob in 10s
07:39:43 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M6"
07:39:43 worker (sidekiq-worker): NotifyJob ran
07:39:43 notifier (baseline): Sent notification: "HELLO M6"

07:39:28 web (baseline): Enqueuing ProcessMessageJob
07:39:28 worker (sidekiq-worker): ProcessMessageJob failed on purpose; Sidekiq will retry it
07:39:34 worker (sidekiq-worker): ProcessMessageJob ran with processor v2: "HELLO M7"
07:39:34 worker (sidekiq-worker): NotifyJob ran
07:39:34 notifier (baseline): Sent notification: "HELLO M7"

Both ran in the sandbox. When each came due (the scheduled job after 10 seconds, the retry after its 5-second delay), Sidekiq pushed it to the shared default queue, and the baseline handed it to the sandbox. Had the sandbox been deleted by then, the baseline would have run it.

8. Delete a sandbox while its jobs wait​

Tell the sandboxed worker to stop taking new jobs, with Sidekiq's quiet signal (TSTP), so the next job waits in its queue:

kubectl -n sidekiq-demo exec $(kubectl -n sidekiq-demo get pods -o name \
-l signadot.com/sandbox-routing-key=$WORKER_KEY) -- sh -c 'kill -TSTP 1'

enqueue m8 $WORKER_KEY; sleep 2
kubectl -n sidekiq-demo exec deploy/redis -- redis-cli LLEN queue:default@sidekiq-worker
Output
1

Delete the sandbox. About 30 seconds after the sandbox disappears from the routing rules, the baseline worker moves the job back to the shared queue and runs it:

signadot sandbox delete sidekiq-worker
sleep 60; trail m8
Output
07:40:58 web (baseline): Enqueuing ProcessMessageJob
07:41:41 worker (baseline): ProcessMessageJob ran with processor v1: "hello m8"
07:41:41 worker (baseline): NotifyJob ran
07:41:41 notifier (baseline): Sent notification: "hello m8"

The baseline worker's log shows the sweep:

kubectl -n sidekiq-demo logs deploy/worker | grep "is gone"
Output
INFO  … sandbox sidekiq-worker is gone: moved 1 job(s) from default@sidekiq-worker to default
INFO … sandbox sidekiq-worker is gone: moved 0 job(s) from notifications@sidekiq-worker to notifications

9. Clean up​

Delete the route group and the remaining sandboxes, then the demo, and disconnect:

signadot routegroup delete sidekiq-demo
signadot sandbox delete sidekiq-notifier
signadot sandbox delete sidekiq-web

kubectl delete namespace sidekiq-demo
signadot local disconnect

With a Redis you keep, there is nothing else to remove: the baseline worker deletes the queues of deleted sandboxes on its own.

How signadot_sidekiq works​

The gem does one of two things, depending on where the worker runs.

In a sandbox, where the Signadot Operator sets SIGNADOT_SANDBOX_NAME, it swaps each queue the worker was started with for the sandbox's own, keeping weights and order:

signadot_sidekiq/lib/signadot_sidekiq.rb
config.capsules.each_value do |capsule|
capsule.queues = capsule.weights.map { |queue, weight| [sandbox_queue(queue, sandbox), weight] }
end

Nothing has to create these queues. A Sidekiq queue is a Redis list, which appears when the first job is pushed to it.

In the baseline, where SIGNADOT_BASELINE_NAME is set in k8s/worker.yaml, it adds a server middleware that runs before every job:

signadot_sidekiq/lib/signadot_sidekiq/router.rb
def call(_job_instance, job, queue)
routing_key = SignadotSidekiq.routing_key(job)
sandbox = routing_key && @rules.sandbox_for(routing_key)
return yield unless sandbox

target = SignadotSidekiq.sandbox_queue(queue, sandbox)
Sidekiq::Client.push(job.merge("queue" => target, "retry_queue" => job["retry_queue"] || queue))
end

The rest of the gem covers what happens around that:

  • Which keys are claimed. RoutingRules asks the Signadot Routes API every 5 seconds which sandboxes of the worker Deployment claim which routing keys. A sandbox claims its own key and the keys of the route groups it is in. The baseline runs no job until it has read the rules once.
  • Retries. retry_queue makes Sidekiq retry a failed sandbox job on the shared queue, so the baseline decides again where the retry runs.
  • Scheduled jobs. Sidekiq pushes a perform_in job to the shared queue when it comes due, so the decision happens then, not when the job was scheduled.
  • Deleted sandboxes. Sweeper moves jobs left in a deleted sandbox's queues back to the shared queue, 30 seconds after the sandbox disappears from the routing rules.

The baseline worker tells the Routes API which workload it is through three environment variables:

k8s/worker.yaml
env:
- name: SIGNADOT_BASELINE_KIND
value: Deployment
- name: SIGNADOT_BASELINE_NAME
value: worker
- name: SIGNADOT_BASELINE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace

Use it in your own app​

To apply the pattern to your own Sidekiq services:

  1. Propagate the routing key. Enable the OpenTelemetry Sidekiq instrumentation in every service that enqueues or runs jobs, next to the instrumentation for your web framework and HTTP client.
  2. Add the gem to your workers. Copy signadot_sidekiq into your codebase and call SignadotSidekiq.configure(config) inside Sidekiq.configure_server. It requires Sidekiq 7.1 or later.
  3. Identify the baseline. Set SIGNADOT_BASELINE_KIND, SIGNADOT_BASELINE_NAME and SIGNADOT_BASELINE_NAMESPACE on the baseline worker's Deployment. Sandboxed workers need no configuration: the Signadot Operator sets SIGNADOT_SANDBOX_NAME in their pods.

Frequently asked questions​

How do you test Sidekiq jobs in a shared Kubernetes cluster without a separate Redis per environment?​

Give each sandboxed Sidekiq worker its own queues, such as default@my-sandbox, and let the baseline worker stay the only reader of the shared queues. The routing key travels inside each job as OpenTelemetry baggage. The baseline worker runs a job itself unless a sandbox claims the job's routing key, in which case it moves the job to that sandbox's queue. Baseline and sandboxed workers then share one Redis.

Can sandboxed Sidekiq workers read the shared queue and skip jobs that are not theirs?​

No. Sidekiq fetches jobs with BRPOP, which removes the job from Redis and gives it to exactly one worker. A worker that skipped a job would lose it. That is why each sandboxed worker reads its own queues and the baseline worker moves jobs to them.

Does every service that enqueues Sidekiq jobs need the signadot_sidekiq gem?​

No. Only Sidekiq worker processes load it. A service that enqueues jobs needs only the OpenTelemetry Sidekiq instrumentation, whose client middleware writes the routing key into each job.

Do scheduled jobs and retries run in the right sandbox?​

Yes. Sidekiq pushes a perform_in job to the shared queue when it comes due, so the baseline worker decides at that moment where it runs. A failed sandbox job is retried on the shared queue as well, through the job's retry_queue, and the baseline hands it back to the sandbox if the sandbox still exists.

What happens to jobs left in the queue of a deleted sandbox?​

The baseline worker runs them. About 30 seconds after a sandbox disappears from the routing rules, a sweeper in the baseline worker moves the jobs left in that sandbox's queues back to the shared queues and removes the sandbox queues from Redis.

Do baseline Sidekiq workers restart when sandboxes come and go?​

No. The baseline worker reads the routing rules from the Signadot Routes API every 5 seconds, so it learns about new and deleted sandboxes while it runs.

How do I use this in a Rails app?​

Call SignadotSidekiq.configure(config) inside Sidekiq.configure_server in config/initializers/sidekiq.rb, and enable the OpenTelemetry Sidekiq instrumentation. The gem uses only Sidekiq's server middleware and lifecycle hooks, nothing from Sinatra or Rails. It requires Sidekiq 7.1 or later.

See also​