Test Debezium Change Data Capture (CDC) Pipelines with Signadot Sandboxes
Sandboxes route a request to the right version of a service by its routing key, which travels in request headers. Change data capture breaks that chain twice. The service that handles the request only writes a row, and Debezium publishes the event later from the database's write-ahead log, without ever seeing the request.
This tutorial shows how to get the routing key through anyway. An orders service writes orders to
Postgres, Debezium publishes each change to Kafka, and a fulfillment consumer reads the changes
and calls a notifier. You fork each service into a sandbox and watch every change reach the right
version of each service. Postgres, Debezium and Kafka stay shared: you never fork them.
Full source: signadot/examples/debezium-cdc-tutorial
How the routing key gets through the database
The request reaches orders with the routing key in its baggage header. The key is stored on the
row, comes out of Postgres with the change, and reaches the consumers as Kafka headers. It makes
three hops:
- Into the database. The
ordersservice sets the request's OpenTelemetry context (traceparentandbaggage) as transaction-local Postgres settings. A trigger copies them into two columns,otel_traceparentandotel_baggage, on every insert and update. - Out of the database. Debezium captures the two columns with the rest of the row, and a
built-in Kafka Connect transform moves them into the Kafka headers
traceparentandbaggage. - Out of Kafka. The baseline consumer reads through the consumer group
fulfillment, and a sandbox reads through its own group,fulfillment-<routing key>. Each one asks the Signadot Routes API which keys belong to sandboxes. The baseline skips those keys, and a sandbox processes only its own.
All of this lives in
platform_cdc/,
a small library of the kind a platform team would ship. The app code is plain SQLAlchemy, Flask and
requests.
Before you begin
- A Kubernetes cluster with the Signadot Operator installed.
- The Signadot CLI, logged in with
signadot auth login. kubectlpointed at that cluster, andjq.- Docker, and a container registry your cluster can pull from.
1. Build the image
The three services run from one image. Build it and push it to your registry:
git clone https://github.com/signadot/examples.git
cd examples/debezium-cdc-tutorial
export IMAGE=docker.io/your-user/debezium-cdc-demo:latest # your registry
docker build -t $IMAGE .
docker push $IMAGE
docker.io/your-user/... is a placeholder. When you build images of your own services this way,
push them to a private repository and give the cluster pull access to it, for example with an
image pull secret.
If your cluster's nodes have a different CPU architecture than your machine (for example an Apple
silicon Mac and an x86 cluster), add --platform linux/amd64 to docker build. On a local cluster
that runs images straight from your Docker daemon, you can skip the push.
2. Deploy the demo
Point the manifests at your image, then deploy Postgres, Kafka, Debezium (Kafka Connect) and the
three services into the debezium-demo namespace. The migrate Job creates the schema:
sed -i.orig "s|image: debezium-cdc-demo:latest|image: $IMAGE|" k8s/*.yaml
kubectl create namespace debezium-demo
kubectl apply -f k8s/
kubectl -n debezium-demo wait --for=condition=complete job/migrate --timeout=300s
kubectl -n debezium-demo wait --for=condition=available deploy --all --timeout=300s
Connect your machine to the cluster, so that cluster service names resolve locally (this asks for your password):
export CLUSTER=my-cluster # your cluster's name, from: signadot cluster list
signadot local connect --cluster=$CLUSTER
Register the Debezium connector for the orders table through the Kafka Connect REST API:
curl -X PUT http://debezium.debezium-demo.svc:8083/connectors/orders/config \
-H 'Content-Type: application/json' -d @debezium/orders-connector.json
curl -s http://debezium.debezium-demo.svc:8083/connectors/orders/status | jq .tasks
The task's state is RUNNING after a few seconds.
3. Look at the code
Three small pieces carry the routing key. The orders service creates its engine with
platform_cdc.create_engine, a drop-in sqlalchemy.create_engine, and writes rows as usual
(platform_cdc/db.py has the engine and the trigger):
engine = platform_cdc.create_engine(DATABASE_URL)
...
with engine.begin() as conn:
conn.execute(text("INSERT INTO orders (id, item, status) VALUES (:id, :item, 'new')"),
{"id": order["id"], "item": order["item"]})
The migration tracks each captured table once. That adds the two context columns and the trigger that fills them:
platform_cdc.track_table(conn, "orders")
The connector
(debezium/orders-connector.json)
moves the columns into Kafka headers, the same headers a service that published the event itself
would have set:
"transforms": "unwrap,context",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.context.type": "org.apache.kafka.connect.transforms.HeaderFrom$Value",
"transforms.context.fields": "otel_traceparent,otel_baggage",
"transforms.context.headers": "traceparent,baggage",
"transforms.context.operation": "move",
The fulfillment service hands a handler to
platform_cdc.ChangeConsumer, which picks the consumer
group, checks each event's routing key, and makes the event's context current while the handler
runs, so the call to the notifier carries the key:
consumer = platform_cdc.ChangeConsumer(
KAFKA_BROKERS, topic=ORDERS_TOPIC, group=CONSUMER_GROUP,
on_ready=lambda: pathlib.Path("/tmp/ready").touch())
consumer.run(handle)
4. Create an order without a routing key
ORDERS=http://orders.debezium-demo.svc:4000
curl -X POST $ORDERS/api/orders -H 'Content-Type: application/json' \
-d '{"id": "o1", "item": "book"}'
Every service writes what it did to an event log that the orders service serves. Define a helper
that shows which workload handled each step of an order. The first change event takes a few
seconds, while Debezium creates the topic:
trail() {
curl -s $ORDERS/api/events | jq -r --arg id "$1" '.events[]
| select(.context.orderID == $id)
| "\(.context.baselineWorkload) (\(.context.sandboxName | if . == "" then "baseline" else . end)): \(.logEntry)"'
}
trail o1
orders (baseline): Inserted order into Postgres
fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o1 (book) is new"
notifier (baseline): Sent notification: "order o1 (book) is new"
5. Create the sandboxes
Each sandbox forks one service. The fulfillment sandbox also sets FULFILLMENT_VERSION=v2, the
"change" under test, which uppercases item names:
name: debezium-fulfillment
spec:
description: "fulfillment sandbox for the Debezium CDC demo (fulfillment v2)"
cluster: "@{cluster}"
labels:
demo: debezium-cdc
forks:
- forkOf:
kind: Deployment
namespace: debezium-demo
name: fulfillment
customizations:
env:
# The change under test: v2 uppercases item names.
- name: FULFILLMENT_VERSION
value: v2
Apply the three sandboxes and a route group that bundles them under one routing key:
signadot sandbox apply -f signadot/sandboxes/fulfillment.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/notifier.yaml --set cluster=$CLUSTER
signadot sandbox apply -f signadot/sandboxes/orders.yaml --set cluster=$CLUSTER
signadot routegroup apply -f signadot/routegroups/debezium-demo.yaml --set cluster=$CLUSTER
Save the routing keys:
FULFILLMENT_KEY=$(signadot sandbox get debezium-fulfillment -o json | jq -r .routingKey)
NOTIFIER_KEY=$(signadot sandbox get debezium-notifier -o json | jq -r .routingKey)
ORDERS_KEY=$(signadot sandbox get debezium-orders -o json | jq -r .routingKey)
GROUP_KEY=$(signadot routegroup get debezium-demo -o json | jq -r .routingKey)
6. Create orders with routing keys
Define a helper that creates an order with a routing key in the baggage header, as a preview URL
or the Signadot browser extension would:
order() {
curl -s -o /dev/null -X POST $ORDERS/api/orders -H 'Content-Type: application/json' \
-H "baggage: sd-routing-key=$2" -d "{\"id\": \"$1\", \"item\": \"book\"}"
}
Create one order per routing key:
order o2 $FULFILLMENT_KEY
order o3 $NOTIFIER_KEY
order o4 $ORDERS_KEY
order o5 $GROUP_KEY
sleep 2; for id in o2 o3 o4 o5; do trail $id; echo; done
orders (baseline): Inserted order into Postgres
fulfillment (debezium-fulfillment): Consumed change event (op c) with fulfillment v2: "order o2 (BOOK) is new"
notifier (baseline): Sent notification: "order o2 (BOOK) is new"
orders (baseline): Inserted order into Postgres
fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o3 (book) is new"
notifier (debezium-notifier): Sent notification: "order o3 (book) is new"
orders (debezium-orders): Inserted order into Postgres
fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o4 (book) is new"
notifier (baseline): Sent notification: "order o4 (book) is new"
orders (debezium-orders): Inserted order into Postgres
fulfillment (debezium-fulfillment): Consumed change event (op c) with fulfillment v2: "order o5 (BOOK) is new"
notifier (debezium-notifier): Sent notification: "order o5 (BOOK) is new"
Each change reached exactly the services its routing key selects:
| Routing key | orders | fulfillment | notifier |
|---|---|---|---|
| none (o1) | baseline | baseline (v1) | baseline |
| fulfillment sandbox (o2) | baseline | sandbox (v2) | baseline |
| notifier sandbox (o3) | baseline | baseline (v1) | sandbox |
| orders sandbox (o4) | sandbox | baseline (v1) | baseline |
| route group (o5) | sandbox | sandbox (v2) | sandbox |
The notifier sandbox (o3) is the clearest proof: the baseline consumer's call landed on the sandbox, so the routing key made it through Postgres, Debezium and Kafka.
The key belongs to the write, not the row
Order o2 was created through the fulfillment sandbox. Ship it with a request that has no routing
key. The trigger clears the stored key, and the baseline handles this change:
curl -s -o /dev/null -X POST $ORDERS/api/orders/o2/ship; sleep 2; trail o2 | tail -3
orders (baseline): Updated order in Postgres
fulfillment (baseline): Consumed change event (op u) with fulfillment v1: "order o2 (book) is shipped"
notifier (baseline): Sent notification: "order o2 (book) is shipped"
7. See the key at each hop
The rows hold the context of the request that last wrote them:
kubectl -n debezium-demo exec deploy/postgres -- \
psql -U postgres -d demo -c "SELECT id, status, otel_baggage FROM orders ORDER BY id"
id | status | otel_baggage
----+---------+------------------------------
o1 | new |
o2 | shipped |
o3 | new | sd-routing-key=u5h40ssk4ykjd
o4 | new | sd-routing-key=77nkpp3435chn
o5 | new | sd-routing-key=mlgky8fm7lb1n
In Kafka, the context is in the headers and no longer in the event's value:
kubectl -n debezium-demo exec deploy/kafka -- /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 --topic demo.public.orders --from-beginning \
--timeout-ms 5000 --property print.headers=true 2>/dev/null | sed 's/__debezium[^,]*,//g'
...
traceparent:00-de9f...,baggage:sd-routing-key=mlgky8fm7lb1n {"id":"o5","item":"book","status":"new","__deleted":"false","__op":"c"}
traceparent:00-c9f7...,baggage:null {"id":"o2","item":"book","status":"shipped","__deleted":"false","__op":"u"}
The baseline consumer's log shows each decision:
kubectl -n debezium-demo logs deploy/fulfillment | grep -E "processing|skipped"
... processing demo.public.orders[0]@0 (routing key None)
... skipped demo.public.orders[0]@1 (routing key 22gurfl1k6v6s)
... processing demo.public.orders[0]@2 (routing key u5h40ssk4ykjd)
... processing demo.public.orders[0]@3 (routing key 77nkpp3435chn)
... skipped demo.public.orders[0]@4 (routing key mlgky8fm7lb1n)
... processing demo.public.orders[0]@5 (routing key None)
Your routing keys will differ.
8. Clean up
Delete the route group and the sandboxes:
signadot routegroup delete debezium-demo
signadot sandbox delete debezium-fulfillment
signadot sandbox delete debezium-notifier
signadot sandbox delete debezium-orders
Kafka keeps a consumer group's offsets after its consumer goes away, so the fulfillment sandbox's group is still there. With a Kafka you keep, delete it yourself:
kubectl -n debezium-demo exec deploy/kafka -- /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 --delete --group fulfillment-$FULFILLMENT_KEY
Then remove the demo and disconnect:
kubectl delete namespace debezium-demo
signadot local disconnect
Things to know
- Writes without a context go to the baseline. A write that doesn't come from a request, or
comes from a client that doesn't use the platform engine (a
psqlsession, a batch job), clears the two columns, so its change event has no routing key. - Deletes go to the baseline. A deleted row has nowhere to store the context of the request that deleted it. If sandboxes need to see deletes, use a soft delete (an update), or write the event to an outbox table that is tracked the same way.
- Sandboxes share the database. A sandboxed
ordersservice writes to the baseline's table. The stored routing key tells you which rows came from which sandbox. To give a sandbox its own database instead, see Using sandboxes with databases.
Frequently asked questions
How do you test a Debezium CDC pipeline without a separate database and Kafka per environment?
Store the routing key with each row, let Debezium move it into Kafka headers, and have consumers filter on it. A Postgres trigger stamps the OpenTelemetry baggage of the writing request on the row, a Kafka Connect transform turns those columns into the baggage and traceparent headers, and each consumer checks the key against the Signadot Routes API. Sandboxes then share one Postgres, one Debezium connector and one Kafka topic with the baseline.
Why does change data capture need more than header propagation?
Because the service that handles the request never publishes the event. It only writes a row. Debezium publishes the event later from the write-ahead log, and it has never seen the request or its headers. The routing key has to be written into the database with the row to reach the change event.
Does the application code have to change?
Very little. The service creates its SQLAlchemy engine with platform_cdc.create_engine, a migration calls platform_cdc.track_table once per captured table, and consumers pass a handler to platform_cdc.ChangeConsumer. The SQL the app runs never mentions the routing key, and the Debezium connector needs six lines of transform configuration.
What happens to deletes and to writes made outside a request?
Both go to the baseline. A deleted row has nowhere to store the context of the request that deleted it, and a write from psql or a batch job has no context, so the trigger clears the columns. If sandboxes need to see deletes, use a soft delete or an outbox table that is tracked the same way.
Does this work with the outbox pattern?
Yes. Track the outbox table with platform_cdc.track_table instead of the business table. Each outbox row then carries the routing key of the request that wrote it, and Debezium moves it into the Kafka headers in the same way.
See also
- Testing Kafka microservices: selective consumption when services publish to Kafka themselves
- Kafka integration guide
- Message Queue Isolation: the other brokers
- Header-Based Routing
- Sandbox specification