Run Automated Tests on PRs
Overview
As development teams iterate on microservices independently, it is important to ensure that relationships between services are healthy with each change being made. Running Integration & E2E tests early helps in ensuring that any critical flows are tested early in the development lifecycle.
In the absence of such testing, there are often issues that are discovered late in the development lifecycle causing long feedback loops and wasted development time in finding and resolving issues that could be caught much earlier.
High-Level Steps
1. Create Pull Request Sandboxes
Follow the steps outlined in the Pull Request Sandboxes guide to create a sandbox for each pull request. In the rest of the guide, we will set up automated tests to run on top of these Pull Request sandboxes.
2. Create a Job Runner Group
A Job Runner Group creates pods to run your tests. You can use your own docker images and pod templates. These pods will be created in the cluster and are pre-warmed to speed up the execution of jobs.
name: my-jrg
spec:
cluster: "my-cluster"
labels:
env: "test-env"
namespace: my-tests-ns # namespace where executors will live
image: runner/image:latest # runner imae
jobTimeout: 30m # jobs can run for a maximum to 30m
scaling:
manual:
desiredPods: 2 # creates 2 pods, can run 2 jobs in parallel
This above specification defines a Job Runner Group named my-jrg
which uses a
custom runner image. You can create my-jrg
directly in the UI, or using the
CLI. An example Job Runner Group specification for running cypress tests is
shown
here.
3. Create a Job Template to Run Tests
Next, set up a job template that takes a script to run the tests. Here’s an example configuration:
spec:
namePrefix: my-job
runnerGroup: my-jrg # which runner group to run within
script: |
#!/bin/bash
set -e
# Clone the git repo
echo "Clone repo"
git clone --single-branch -b "@{branch}" \
https://github.com/org/repo.git
# Run the test
cd tests/
./run-tests.sh
routingContext:
sandbox: "@{sandbox}" # controls the injection of routing env vars into the above script
uploadArtifact: # uploading artifacts
- path: path/output.mp4
There are other fields as well in the Job Runner Group that allow you to specify a custom Pod Template to attach secrets / credentials to the runner pods so that they are available to the jobs for say - cloning Git repositories. You can learn more about all the different options available in the Job Runner Group reference.
Note how this has two template variables, the branch
and the sandbox
. You
can now supply these parameters at run-time as shown:
signadot job submit -f ./job-template.yaml --set branch=main --set sandbox=my-sandbox --attach
This job template can be used with the signadot job submit
command and each
submission will have a distinct name that is derived based on the namePrefix
field. An example Job template for running cypress tests is shown
here.
3. Using the Routing Key in Tests
Within your test, ensure you use the routing key to target the Sandbox environment. There are the following environment variables available within the script of the Job specification:
SIGNADOT_ROUTING_KEY
SIGNADOT_SANDBOX_NAME
You can use the above parameters and pass them into your test, so that the tests can inject the appropriate routing key into requests they are making so that it can target the sandbox correctly.
4. Integrating with CI Pipeline
Finally, integrate the jobs with your CI pipeline. You'll typically run jobs
after creating the sandbox from your specific CI/CD pipeline using the
signadot
CLI. For detailed instructions, refer to the guide on integrating
with CI/CD.
Best Practices
Storing Specifications
Creating a Job Runner Group is typically a one-time activity, so, it could be specified GitOps style in a central repository, or created from the UI and managed entirely within Signadot.
The Job Template is typically stored in Git. Some of the common locations where job specifications are typically stored include:
- Source control / git within a monorepo (example)
- Source control / git in a central infrastructure repository (typically in a multi-repo setup)
In source control, they may take the following structure to store and maintain multiple different job specs as needed:
- signadot/job-integration-routeservice-mocha.yaml
- signadot/job-e2e-routeservice-cypress.yaml
...
Choosing the Image for a Job Runner Group
When choosing the Docker image for a Job Runner Group, you have a lot of flexibility depending on your specific needs and the nature of the jobs you are running.
Generic Upstream Docker Image: You can start with a general upstream Docker image like
ubuntu
and choose to install dependencies in the job'sscript
and run tests on it. This approach helps reuse the same runner group across many types of jobs without needing to create or maintain any new docker images.Custom Docker Images: As you optimize your tests, you may choose to create custom Docker images. In these images, you can bake in all the testing dependencies. This can significantly speed up the tests as the dependencies do not need to be installed every time a job is run.
We recommend starting with the general image. As you understand your testing needs better, you can create custom Docker images to optimize tests and make them run faster.
Conclusion
This guide provides a step-by-step process for setting up Automated tests for sandboxes, making it easy for users to implement this in your development workflow.