Testing Database Schema Changes with a MariaDB Resource Plugin

A schema change is hard to test on a shared database: add a column, and every service, developer and test that uses the table sees it at once. The safer pattern gives the change its own temporary database. In this tutorial, a Signadot resource plugin creates a MariaDB instance for each sandbox, you add a description column to the locations table of the HotROD demo's location service, and you test the change end to end while the shared database keeps its old schema.

Sources: the MariaDB resource plugin and the HotROD demo app.

How it works

A resource plugin ties a piece of infrastructure to a sandbox's lifetime. When you create a sandbox that requests a database from the mariadb plugin:

1. The plugin's create workflow runs provision.sh in a pod in the signadot namespace. It installs a MariaDB Helm release named after the sandbox and returns the host, port and root password as outputs.
2. The sandbox forks the location Deployment with your new image and sets the fork's database environment variables from those outputs.
3. Requests that carry the sandbox's routing key are routed to the fork, which uses the temporary database. All other requests reach the baseline and the shared database.

When you delete the sandbox, the delete workflow runs deprovision.sh, which uninstalls the release. This diagram shows the two paths a request can take:

The work splits in two. A platform engineer installs the plugin once per cluster (Step 1), and developers then use it for every schema change (Steps 2 to 6).

Before you begin

You need:

• A Kubernetes cluster with the Signadot Operator installed.
• HotROD running in the hotrod namespace. Without a service mesh, install it with:

  kubectl create ns hotrod
  kubectl -n hotrod apply -k 'https://github.com/signadot/hotrod/k8s/overlays/prod/devmesh'

  For Istio, use the istio overlay instead, as in the quickstart.
• The Signadot CLI, logged in with signadot auth login.
• kubectl with access to the signadot namespace, plus git and jq.
• Go, Docker, and a container registry your cluster can pull from.
• The Signadot Chrome extension for Step 5, and a mysql or mariadb client for Step 6.

Step 1: Install the MariaDB resource plugin

A plugin is registered once per cluster; after that, any sandbox can request a database by naming it. Clone the plugins repository:

git clone https://github.com/signadot/plugins.git
cd plugins/mariadb

plugin.yaml declares where the scripts run, the input a sandbox must provide, and the outputs the plugin returns:

plugins/mariadb/plugin.yaml

name: mariadb
spec:
  description: |
    Provision a MariaDB instance

    Sandbox should provide input 'dbname' for the name of the database.
    Plugin provisioner provides outputs 'host', 'port', 'root-password' for 
    an empty database instance tied to the lifetime of the sandbox.

  runner:
    image: dtzar/helm-kubectl
    namespace: signadot
    podTemplateOverlay: |
      spec:
        serviceAccountName: sd-mariadb

  create:
  - name: provision
    inputs:
    - name: dbname
      valueFromSandbox: true
      as:
        env: DBNAME

    outputs:
    - name: host
      valueFromPath: /tmp/host
    - name: port
      valueFromPath: /tmp/port
    - name: root-password
      valueFromPath: /tmp/root-password

    script: "@{embed: ./plugin/provision.sh}"

  delete:
    - name: deprovision
      script: "@{embed: ./plugin/deprovision.sh}"

A sandbox refers to each output by step and name: provision.host, provision.port and provision.root-password. The create step's script installs the Bitnami MariaDB chart, without a persistent volume, and writes those three files:

plugins/mariadb/plugin/provision.sh

export NAMESPACE=signadot
RELEASE_NAME="signadot-${SIGNADOT_RESOURCE_NAME,,}-${SIGNADOT_SANDBOX_ID}"
echo "Installing Helm release: ${RELEASE_NAME}"
helm -n ${NAMESPACE} install "${RELEASE_NAME}" bitnami/mariadb \
  --version 24.0.3 --wait --timeout 5m0s \
  --set fullnameOverride="${RELEASE_NAME}-mariadb" \
  --set primary.persistence.enabled=false \
  --set serviceAccount.create=false \
  --set auth.database="${DBNAME}"

# Get the generated password, based on instructions from the Helm chart.
MYSQL_ROOT_PASSWORD=$(kubectl -n ${NAMESPACE} get secret "${RELEASE_NAME}-mariadb" -o jsonpath="{.data.mariadb-root-password}" | base64 -d)

# Populate the outputs
DBHOST="${RELEASE_NAME}-mariadb.${NAMESPACE}.svc"
echo -n "${DBHOST}" > /tmp/host
echo -n 3306 > /tmp/port
echo -n "${MYSQL_ROOT_PASSWORD}" > /tmp/root-password

The scripts run as the sd-mariadb service account, which k8s/mariadb-init.yaml creates with a Role and RoleBinding limited to the signadot namespace:

API group	Resources	Verbs
core	pods	get, list, watch
core	configmaps, secrets, services, serviceaccounts	get, list, watch, create, delete, patch, update
apps	statefulsets	get, list, watch, create, delete, patch, update
networking.k8s.io	networkpolicies	get, list, watch, create, delete, patch, update
policy	poddisruptionbudgets	get, list, watch, create, delete, patch, update

Create the service account and its permissions, then register the plugin with Signadot:

kubectl -n signadot create -f k8s/mariadb-init.yaml
signadot resourceplugin apply -f plugin.yaml

The resource plugin spec documents every field used here.

Step 2: Add the description column

The schema change lives in HotROD's location service. Clone HotROD:

git clone https://github.com/signadot/hotrod.git
cd hotrod

The service creates its own table: when a query fails because locations does not exist, it runs tableSchema and inserts the seed rows. A new, empty database therefore gets the new schema on first use, with no migration step. Add the field to the Location struct:

services/location/interface.go

type Location struct {
	ID          int64  `json:"id"`
	Name        string `json:"name"`
	Coordinates string `json:"coordinates"`
	Description string `json:"description"` // new field
}

In services/location/database.go, add the column to tableSchema and a description to each entry of seed:

services/location/database.go

 const tableSchema = `
 CREATE TABLE IF NOT EXISTS locations
 (
     id bigint unsigned NOT NULL AUTO_INCREMENT,
     name varchar(255) NOT NULL,
     coordinates varchar(255) NOT NULL,
+    description varchar(255),
 ...
 var seed = []Location{
 	{
 		ID:          1,
 		Name:        "My Home",
 		Coordinates: "231,773",
+		Description: "Cozy two-bedroom apartment",
 	},

The other four seed entries get descriptions the same way. Then read and write the column in every query on the table:

Function	New SQL	Also add
List	SELECT id, name, coordinates, description FROM locations	&c.Description to rows.Scan
Get	SELECT id, name, coordinates, description FROM locations WHERE id = ?	&c.Description to row.Scan
Create	INSERT INTO locations SET name = ?, coordinates = ?, description = ?	location.Description to both d.db.Exec calls
Update	UPDATE locations SET name = ?, coordinates = ?, description = ? WHERE id = ?	location.Description, before location.ID, to both d.db.Exec calls
setupDB	INSERT INTO locations (id, name, coordinates, description) VALUES (?, ?, ?, ?)	c.Description to stmt.Exec

Don't skip setupDB: it inserts the seed rows, and without the change they get no description. Check that the service compiles with go build ./services/location/.

Step 3: Build the image and create the sandbox

The sandbox runs your code from an image. HotROD's Dockerfile copies a prebuilt binary, so build the binary first. These commands follow the repository's make build target, minus the step that rebuilds the React frontend, whose built assets are committed:

export IMAGE=<your-registry>/hotrod:schema-change
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o dist/linux/amd64/bin/hotrod ./cmd/hotrod
docker build --platform linux/amd64 -t $IMAGE .
docker push $IMAGE

Use a private repository for your own services

<your-registry> is a placeholder. When you build images of your own services this way, push them to a private repository and give the cluster pull access to it, for example with an image pull secret.

For a cluster with arm64 nodes, use arm64 in both places. Save this sandbox spec as sandbox.yaml:

sandbox.yaml

name: hotrod-schema-change
spec:
  cluster: "@{cluster}"
  resources:
  - name: locationdb
    plugin: mariadb
    params:
      dbname: location
  forks:
  - forkOf:
      kind: Deployment
      namespace: hotrod
      name: location
    customizations:
      images:
      - container: hotrod
        image: "@{image}"
      env:
      - name: MYSQL_ADDR
        valueFrom:
          resource:
            name: locationdb
            outputKey: provision.host
      - name: MYSQL_PASS
        valueFrom:
          resource:
            name: locationdb
            outputKey: provision.root-password
  - forkOf:
      kind: Deployment
      namespace: hotrod
      name: frontend
    customizations:
      images:
      - container: hotrod
        image: "@{image}"
  defaultRouteGroup:
    endpoints:
    - name: frontend
      target: http://frontend.hotrod.svc:8080

The spec does three things:

• Requests a database. The locationdb resource asks the mariadb plugin for a database named location, the name the service uses by default (MYSQL_DBNAME).
• Points the location fork at it. HotROD reads its database address from MYSQL_ADDR and its password from MYSQL_PASS. MYSQL_ADDR gets the plugin's host; with no port in the address, the MySQL driver uses 3306, the port the plugin's MariaDB listens on. The user stays root, the service's default.
• Forks the frontend too. The frontend decodes the location service's responses into the same Location struct, so with the baseline image it would drop the new field. It needs no database.

Create the sandbox, with your cluster name from signadot cluster list:

export CLUSTER=<your-cluster-name>
signadot sandbox apply -f sandbox.yaml --set cluster=$CLUSTER --set image=$IMAGE

The sandbox becomes ready once the plugin's provision step finishes, which includes waiting up to five minutes for MariaDB to start. If it fails, signadot logs --sandbox hotrod-schema-change --resource locationdb shows the script's output; see testing and debugging plugins.

Step 4: Test the change with curl

The location service is not exposed outside the cluster, so call it through signadot local connect, which makes cluster service names reachable from your machine. Leave it running for the rest of the tutorial:

signadot local connect --cluster=$CLUSTER

Requests reach the sandbox only when they carry its routing key, so look it up:

export ROUTING_KEY=$(signadot sandbox get hotrod-schema-change -o json | jq -r .routingKey)

Create a location with a description:

curl -s -X POST http://location.hotrod.svc:8081/location \
  -H "Content-Type: application/json" \
  -H "baggage: sd-routing-key=$ROUTING_KEY" \
  -d '{"name": "Central Park", "coordinates": "400,500", "description": "A large public park in New York City"}' | jq

{
  "id": 732,
  "name": "Central Park",
  "coordinates": "400,500",
  "description": "A large public park in New York City"
}

The first query on the empty database created the table with the new column and inserted the seed rows, so the new row gets the next ID after the seed's highest, 731. List every location:

curl -s http://location.hotrod.svc:8081/locations \
  -H "baggage: sd-routing-key=$ROUTING_KEY" | jq -c '.[]'

{"id":1,"name":"My Home","coordinates":"231,773","description":"Cozy two-bedroom apartment"}
{"id":123,"name":"Rachel's Floral Designs","coordinates":"115,277","description":"Flower shop specializing in arrangements"}
{"id":392,"name":"Trom Chocolatier","coordinates":"577,322","description":"Artisan chocolate and confectionery"}
{"id":567,"name":"Amazing Coffee Roasters","coordinates":"211,653","description":"Local coffee roaster with a variety of blends"}
{"id":731,"name":"Japanese Desserts","coordinates":"728,326","description":"Traditional Japanese sweets and pastries"}
{"id":732,"name":"Central Park","coordinates":"400,500","description":"A large public park in New York City"}

Now send the same request without the header. It reaches the baseline service and the shared database, which have no description and no Central Park:

curl -s http://location.hotrod.svc:8081/locations | jq -c '.[0]'

{"id":1,"name":"My Home","coordinates":"231,773"}

Step 5: Preview the change in the frontend

A reviewer can check the same change through the app's UI, with no headers to set by hand. The Chrome extension adds the routing key to every request the browser sends:

1. Install the Signadot Chrome extension and sign in to your Signadot account.
2. Open the extension, select the hotrod-schema-change sandbox, and turn routing on.
3. With signadot local connect still running, open http://frontend.hotrod.svc:8080.

The forked frontend serves the page, and its calls to the location service reach the location fork. The UI does not display descriptions, so look at the data it loads: in your browser's developer tools, open the Network tab, reload, and select the splash request. Its response lists the locations from the temporary database, each with a description. The same check from a terminal:

curl -s http://frontend.hotrod.svc:8080/splash \
  -H "baggage: sd-routing-key=$ROUTING_KEY" | jq -c '.Locations[0]'

{"id":1,"name":"My Home","coordinates":"231,773","description":"Cozy two-bedroom apartment"}

The sandbox's frontend endpoint also has a preview URL, shown in the dashboard, that routes to the sandbox without the extension. The developer access guide covers the headers the extension sets and the other ways to reach a sandbox.

Step 6: Inspect the temporary database

To confirm the schema directly, connect to the sandbox's MariaDB. The provision step logs the name of the Helm release it installed:

signadot logs --sandbox hotrod-schema-change --resource locationdb --step provision -c main | grep "Helm release"

Installing Helm release: signadot-locationdb-<sandbox-id>

The database host and root password follow from that name, the same way provision.sh derives them. With signadot local connect still running:

export RELEASE=signadot-locationdb-<sandbox-id>
export DB_PASSWORD=$(kubectl -n signadot get secret $RELEASE-mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 -d)
mysql -h $RELEASE-mariadb.signadot.svc -u root -p"$DB_PASSWORD" location -e "DESCRIBE locations;"

+-------------+---------------------+------+-----+---------+----------------+
| Field       | Type                | Null | Key | Default | Extra          |
+-------------+---------------------+------+-----+---------+----------------+
| id          | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
| name        | varchar(255)        | NO   | UNI | NULL    |                |
| coordinates | varchar(255)        | NO   |     | NULL    |                |
| description | varchar(255)        | YES  |     | NULL    |                |
+-------------+---------------------+------+-----+---------+----------------+

The description column exists in the sandbox's database and accepts NULL, as the new tableSchema defines it. The shared database is untouched.

Clean up

Delete the sandbox. Its delete workflow runs deprovision.sh, which uninstalls the MariaDB release and its data with it:

signadot sandbox delete hotrod-schema-change
signadot local disconnect

The plugin stays registered for the next schema change. To remove it, delete every sandbox that uses it first, so each can deprovision its database, then run from plugins/mariadb:

signadot resourceplugin delete -f plugin.yaml
kubectl -n signadot delete -f k8s/mariadb-init.yaml

Automate schema testing for every pull request

Steps 3 and 4 are what CI runs for a pull request that changes a schema. Build and push the image for the commit, create the sandbox with that image (give each pull request its own sandbox name), run your API or end-to-end tests with the sandbox's routing key in the baggage header, and delete the sandbox when the run ends. Every pull request is tested against a fresh database with its own schema, and parallel pull requests never share one. Set up PR sandboxes and the GitHub Actions guide show the CI side.

Frequently asked questions

How do you test database schema changes without breaking a shared database?

Give the change its own database. A sandbox forks only the service whose schema changes, and a resource plugin creates a temporary database for that sandbox and passes its connection details to the fork as environment variables. Requests carrying the sandbox's routing key reach the fork and its new schema, while all other traffic keeps using the shared database unchanged.

How do you automate database schema testing for pull requests?

In CI, build the service image from the pull request, create a sandbox that runs it and requests a database from a resource plugin, then run your tests with the sandbox's routing key in the baggage header. Each pull request gets a fresh database with its own schema, and deleting the sandbox at the end of the run deletes the database.

What is a Signadot resource plugin?

A resource plugin defines a create workflow and a delete workflow that Signadot runs when a sandbox that requests the resource is created or deleted. Each step runs a script in a pod in your cluster, takes inputs from the sandbox spec, and writes outputs to files that the sandbox can map to environment variables of its forked workloads. The MariaDB plugin installs a MariaDB Helm release per sandbox and uninstalls it when the sandbox is deleted.

Does the schema change affect the shared database or other developers?

No. Only the forked service connects to the temporary database, and only requests that carry the sandbox's routing key reach the fork. The baseline service keeps reading and writing the shared database with the old schema, so other developers, sandboxes and tests are unaffected.

What data does the temporary database contain?

It starts empty: the plugin creates an empty database with the name given in the sandbox spec. In this tutorial, the HotROD location service creates the locations table and inserts its seed rows the first time a query finds no table, so the fork sees the new schema with seed data. For production-like data, seed it in the plugin's provision script, or use a branching database such as Neon, whose branches start with the parent's schema and data.

Can I use the same approach with PostgreSQL or a managed database?

Yes. A resource plugin can provision anything a script can create, from a Helm release to a cloud database or a schema in an existing server. The Signadot plugins repository includes plugins for PostgreSQL with Vault, Dolt, Terraform and Amazon SQS, and the Neon and Xata tutorials use the same mechanism to give each sandbox a copy-on-write database branch.

See also

• Using sandboxes with databases: shared data, a temporary database, or a branch per sandbox, and when to use each
• Resource plugins: how plugins work, with the spec and debugging reference
• Sandbox resources: requesting a resource in a sandbox spec and passing its outputs to forks
• Neon database branching: a copy-on-write Postgres branch per sandbox, with the parent's data
• signadot local connect: reach cluster services from your machine
• Developer access to sandboxes: the Chrome extension, routing headers and preview URLs
• Set up PR sandboxes: create a sandbox for every pull request