AI-Driven Contract Testing: Signadot SmartTest vs. Pact

Introduction

In today's environment, rapid changes to an API can disrupt an entire microservices architecture. This hands-on tutorial demonstrates how to use the HotROD demo application to compare two powerful contract testing approaches.

The tutorial involves deploying the HotROD demo application to a Kubernetes cluster and using it as a sandbox to implement both Signadot Smart Test and Pact. Instead of merely discussing contract testing in theory, this guide walks through writing tests, introducing breaking changes in isolated environments, and observing how each tool identifies those issues.

A unique aspect of this tutorial is the use of the same application and identical API changes across both testing frameworks. Readers will see how Signadot leverages AI to automatically detect contract violations, in contrast to Pact’s more traditional method of explicit contract definition. The content is structured to be accessible for beginners while remaining valuable to experienced developers. All necessary commands, configuration files, and scripts are included, allowing users to follow along at their own pace. Whether using the CLI locally or opting for the containerized approach with Signadot’s job runners, the setup is fully supported. By the end of the tutorial, readers will have hands-on experience with both tools and a clear understanding of their practical differences. Signadot's AI-powered analysis may significantly reduce maintenance overhead, while Pact offers the clarity of explicitly defined contracts.

The objective is not to declare a winner, but to provide practical insights that help in selecting the right approach based on specific requirements. Now, it’s time to launch the terminal and get started—future API changes will deploy much more smoothly as a result.

Getting Started: Implementing Contract Testing with Pact and Signadot

Let’s get hands-on and dive deeper into the real-world practical implementation of both Pact and Signadot. This section will clarify how each tool contributes to effective contract testing.

First things first: before implementing Pact or Signadot, it is essential to deploy the API service required to run contract tests.

Deploy the hotrod demo application  

HotROD is a microservice-based application designed for testing distributed systems. It can be deployed to a cluster using the following commands:

This command deploys HotROD with the DevMesh overlay in the hotrod namespace.

Create an Isolated Sandbox Environment to simulate Contract changes in the upcoming release

Signadot Sandboxes allow services to be forked for isolated testing without impacting the baseline environment.

Define the Sandbox

This sandbox version intentionally breaks the API contract to simulate how contract violations are reported by each tool. Specifically, HotROD’s LocationService modifies the field named Name to locName.

Create a file named location-breaking-sbx.yaml to fork the location-service deployment:

Apply the Sandbox

Note: If the Signadot CLI is not already installed on your computer, please refer to the following link and complete the installation.

https://www.signadot.com/docs/getting-started/installation/signadot-cli

After running this command, the output can be viewed in the Signadot web GUI under the Sandboxes section.

Now comes the part many have been curious about—the proof of implementation. Let's explore how Signadot Smart Test transforms contract testing in real-world scenarios.

The first step is to write a Smart Test file. All Smart Tests are expected to be placed in files with the .star suffix.

Create a file named location-contract.star to define the smart contract.

When using capture=True as an argument, Signadot will use AI to detect differences in the contract from upcoming API releases.

Now it's time to run the Signadot Smart Test and observe the differences between Pact and Signadot Smart Tests.

The results generated by the Signadot Smart Test can be viewed against the baseline environment.

Now, run the test against the sandbox environment.

The results generated by the Signadot Smart Test can be viewed against the sandbox environment.

As the results show, the API service in the sandbox has introduced several differences to the contract.

The two images below illustrate the differences identified for the upcoming version of the API, as displayed in the Signadot Web GUI.

The first image highlights the impact of the contract changes. The Smart Tests classify change relevance as High, Medium, or Low. This relevance is determined by the AI system based on how “unexpected” a change appears.

For example:

• If a field that was always present is suddenly missing in the pull request version of the API, it is assigned high relevance.
• If a field contains a new value, it is also flagged as high relevance.
• On the other hand, fields like timestamps—whose values naturally vary—are flagged as low relevance when differences are detected.

The image below highlights the key areas that have been changed in the upcoming version.

‍

Next Step: Configuring Pact Contract Testing

Pact can be used independently from the CLI or as a Kubernetes job. However, for demonstration purposes, a Signadot job runner group will be created to run Pact contract tests.

The first step is to write some boilerplate code snippets for the Pact configuration using JavaScript.

Create a file named pact.consumer.js to generate the consumer expectation Pact file.

Next, create a package.json file to install the required dependencies.

Note: If you are running Pact from the CLI, follow the steps below to run the test. Otherwise, you can skip these steps and proceed.

To execute the following commands, ensure that Node.js version 20 or higher is installed on your local machine.

This will generate the Pact file based on the definitions in the pact.consumer.js file.

Note: If you are running Pact in a containerized environment such as Docker, Kubernetes, or using Signadot job runners, follow the steps below to create the container. Otherwise, you may skip this section and proceed.

Create a Dockerfile to build pact.consumer.js as a container, which can be executed on any platform, including Kubernetes, Docker, or within a Signadot job runner group.

To run the following commands, ensure that Docker is installed on your local machine.

This will create a Docker image in your local Docker image repository.

Note: If you are running Pact using a Signadot Job Runner, follow this step. Otherwise, you can skip it and move forward.

Create a jobrunner.yaml file to define the Job Runner group in Signadot.

Note: If you are running these tests from the Job Runner group, you must push your locally built Docker image to a remote Docker repository such as Docker Hub. Follow this Docker documentation to push your local image to the remote repository.

To apply the Job Runner configuration:

Next, the producer verification will be run against the Pact file generated by the Pact consumer.

To perform producer verification, create a pact.provider.js script with dynamic parameters.

Now, the provider test will be run using the Signadot job template. If a different approach—such as Docker—is being used, this file should be injected into the container using a Docker volume mount. For guidance on mounting Docker volumes, refer to the official documentation: https://docs.docker.com/engine/storage/bind-mounts/.

Create a Parameterized Job Template

This job template is designed to dynamically route HTTP request traffic to a specified target destination—either the baseline or sandbox environment—for contract validation.

Create a job-template.yaml file to run the Pact provider job.

Now comes the handy part—viewing the output by running the command below:

InjectRoutingKey:

• disabled: Requests are routed to the baseline, as no routing key is injected.
• auto: Requests are routed to the sandbox with the routing key automatically injected by a proxy.

Note: trafficManager.injectRoutingKey does not work with HTTPS traffic. If the workload is running under HTTPS, refer to the following guide: https://www.signadot.com/docs/tutorials/testing/e2e-with-cypress#injecting-routing-context. It demonstrates how to inject routing keys into HTTP requests.

Great! The expected output was successfully obtained by executing the provider test against the baseline environment.

Similarly, the test failed when executed against the sandbox environment. The image below shows the reason for the failure.

As shown, the contract was broken by the API due to a change in the response body.

Comparing Pact and Smart Tests

Signadot Smart Test significantly reduced the need for manual intervention. By automatically generating and updating contracts through AI analysis of existing tests, it eliminated hours of maintenance work. The platform intelligently adapted to changing environments without requiring constant human oversight, allowing developers to focus on building features rather than maintaining contract testing infrastructure.

Pact initially appeared straightforward to configure and run, but the reality proved more complex. As services scaled, Pact demanded increasingly tedious manual effort—updating contracts with each API change, managing contract versions across environments, and maintaining test fixtures for every consumer-provider relationship.

Smart Tests vs. Pact: A Comparison

Organizational Impact

Pact typically requires significant organizational commitment, with teams often experiencing coordination bottlenecks as services scale. API producers and consumers must synchronize their development cycles, and version management becomes increasingly complex in larger environments.

Smart Tests offer a more pragmatic approach, delivering the core value of contract testing—preventing breaking changes—without the heavy organizational overhead. The AI-powered system continuously enhances its detection capabilities, becoming more effective over time while maintaining a low-effort implementation.

For teams seeking practical contract testing without extensive process changes, Smart Tests provide an accessible alternative that emphasizes results over formalities.

Conclusion

• Key Differences Between Pact and Signadot Smart Test
• Signadot Smart Test automates nearly all contract maintenance through AI-powered diffs and sandbox environments, reducing maintenance time from hours to just minutes. It provides a low-effort, adaptive solution that evolves with changing APIs and environments.
• In contrast, Pact offers fine-grained, explicit control over contract definitions but requires significant manual upkeep. As services and API changes increase, so does the complexity of maintaining Pact tests—making it more resource-intensive at scale.