Contract Testing Simplified with AI
Catch API contract breaks before they reach production
Simple Developer Workflow
Powered by AI
Frequently Asked Questions
Traditional contract testing tools like Pact require explicitly defining and maintaining contracts. SmartTests eliminates this by having you write simple Starlark tests that exercise your API endpoints. The system runs these on both baseline and sandbox environments, using AI to analyze differences between responses and automatically detect breaking changes without explicit assertions. This approach tests real behavior rather than maintaining abstract contracts, reducing maintenance overhead and false positives.
The AI system performs two main functions: First, it analyzes API responses from both versions to identify structural, data type, and content differences. The models determine which differences actually impact compatibility rather than flagging every change.
Second, it reduces test flakiness by learning from historical test runs, distinguishing between expected variations (timestamps, random IDs) and genuine breaking changes. The system adapts to your specific APIs over time, improving its accuracy with continued use.
SmartTests detects:
- Schema changes: Removed fields, changed data types, altered object structures
- Behavioral changes: Different response codes, new error conditions, altered validation rules
Because SmartTests observes runtime behavior rather than just validating static schemas, it can identify subtle issues that traditional approaches miss.
Integration requires two components:
1) The Signadot CLI integrated into your pipeline to create sandboxes for each PR
2) Smart Test triggers configured to automatically run tests against those sandboxes
Most teams have their first SmartTests running in under an hour:
1) Install the Signadot operator on your Kubernetes cluster (10-15 minutes)
2) Write your first test in Starlark to call your API endpoints (5-10 minutes)
3) Configure triggers to run the test on your sandboxes (5 minutes)
4) Integrate sandbox creation with your CI/CD pipeline (15-30 minutes)
Since SmartTests runs against your existing staging/pre-production environment, you avoid the typically time-consuming process of setting up separate test environments or managing test data.
SmartTests minimizes maintenance when APIs evolve:
- When you make compatible API changes (adding fields, extending functionality in backward-compatible ways), SmartTests automatically recognizes these as non-breaking and doesn't require test updates.
- For intentional breaking changes, you typically only need to update the test code that calls the changed endpoints - not assertions or expectations, since those are handled by the AI comparison system.
