# Signadot > Signadot is a verification platform for cloud native applications that scales to large enterprises. Verifying a code change takes a realistic environment and automated tests, and Signadot provides both. Its unique approach multiplexes many ephemeral virtual environments within one physical environment. They are light and fast, so hundreds run in parallel for large teams and fleets of coding agents. Signadot also runs your existing tests in these environments, and lets agents write and run new ones. - **How it runs:** Signadot installs as a Kubernetes operator in your environment and connects to a SaaS control plane that sees only resource metadata. The Signadot CLI and API manage everything else. - **An ephemeral environment for every code change:** changes from coding agents and developers working locally, and from pull requests, each get an ephemeral virtual environment, one of many multiplexed within a single physical environment such as staging. - **Only what changed is duplicated:** instead of copying the whole stack, each environment duplicates only the components that changed and shares the rest, so the change can be tested end to end, across services. - **You choose what is duplicated:** usually only the changed services, plus a database or queue when a change needs its own, up to the whole stack. - **Data and messages:** data isolation is built in, and so is isolation for messages that flow through a shared message broker or queue (Kafka, RabbitMQ, SQS, Pub/Sub, NATS), and for Temporal workflows. - **Resource plugins:** create and delete any resource with an environment's lifecycle, including resources outside Kubernetes such as managed databases and cloud services. - **Service mesh:** requests reach the right environment through the mesh you already run (Istio natively, the Gateway API, or a DevMesh sidecar with Linkerd or no mesh), with the routing key in OpenTelemetry baggage. - **Scale:** these virtual environments are light and fast, so hundreds run in parallel for large teams and fleets of coding agents. - **Automated tests in the same environments:** Signadot runs your existing tests in them and lets coding agents write and run new ones. Coding agents connect through an MCP server and agent skills. Common use cases: - **[Preview environments for every pull request](https://www.signadot.com/solutions/preview-environments/):** Each pull request to a microservices app on Kubernetes gets its own ephemeral environment and preview URL, duplicating only the changed services on the cluster you already run. - **[Environments where coding agents test their own changes](https://www.signadot.com/solutions/agentic-development/):** Claude Code, Cursor, Codex and background agents running in cloud sandboxes each get an ephemeral environment with real services, through the Signadot MCP server, agent skills or CLI. - **[Verifying AI-generated code before merge](https://www.signadot.com/validate-ai-generated-code-kubernetes/):** Agents and CI run a change against real services, databases and message queues and fix what fails, before anyone reviews the pull request. - **[Integration and end-to-end tests against real dependencies](https://www.signadot.com/solutions/microservices-testing/):** Your own integration, end-to-end and performance tests run in each environment, instead of against mocks or in a queue for shared staging. - **[Local development on Kubernetes](https://www.signadot.com/solutions/local-development/):** Run the service you are changing on your laptop, or let an agent run it, connected to real dependencies in a shared cluster. - **[Scaling past one shared staging environment](https://www.signadot.com/solutions/ephemeral-environments/):** Many lightweight environments run on one baseline environment, such as staging, so teams stop queuing for it without copying the whole stack for every pull request. This file is the full-content markdown export of key pages on www.signadot.com. The link index version is at https://www.signadot.com/llms.txt. Documentation pages are served at https://www.signadot.com/docs/. A separate llms.txt covering the full technical documentation is available at https://www.signadot.com/docs/llms.txt # Core Product Pages ## Homepage Source: https://www.signadot.com/ Summary: Signadot, the verification platform for cloud native applications. Each change from a coding agent, developer or pull request gets an ephemeral environment with real services, and the tests that run in it (Jobs and Plans), so agents and CI can verify AI-generated code before merge. Cloud native verification platform ### AI writes code. We make sure it works. Catch what AI-generated code breaks before merge. Each change gets an ephemeral environment without duplicating the stack, so agents and CI test hundreds of changes in parallel. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - No credit card required - Playground cluster available *Image: Architecture diagram: a browser request flows through an API gateway to cart, user, notification, auth, and config services backed by three databases. A sandboxed copy of user-svc for pull request 42 is spliced into the request path, and an Agent Skills panel shows end-to-end tests, API diff, and contract checks passing.* Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - What makes Signadot different #### Realistic environments, as many as your team and agents need Every change needs a realistic environment to be tested in, and coding agents make many changes at once. Signadot environments are realistic and light enough for each change to get its own. The usual way ##### A full copy of the stack per environment Most ephemeral environment tools copy the entire stack for each environment they create. Full copies are slow to start and costly to run, so teams can afford only a few, while coding agents need one for every change. A full stack per environment team-a full copy pr-42 full copy agent-7 full copy Signadot ##### Light enough for every change A Signadot ephemeral environment starts in seconds and costs a fraction of a full copy, yet behaves like an entire environment, with real services, data and queues. Hundreds share one cluster. [See how it works](#how-it-works) Ephemeral environments pr-41 checkout pr-42 payments agent-7 orders + db share the rest One shared environment, such as staging frontend checkout payments orders db queue ##### Your tests run in the same environments Jobs ###### Run your test suites Run the Playwright, Cypress or custom suites you already have inside your cluster, in parallel, each against its own ephemeral environment. [Learn more about jobs](https://www.signadot.com/product/jobs/) Plans ###### Checks coding agents run Compose actions into reusable plans that coding agents run to check their changes in an ephemeral environment. [Learn more about plans](https://www.signadot.com/product/plans/) ##### Results customers report 10x more integration capacity Miro replaced about 15 fixed test environments with 3 shared ones, and its teams and agents test in parallel. [Read the Miro story](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) 2.1x deploy frequency Bitso gives 250+ engineers and their coding agents an environment for every pull request, across 200+ microservices. [Read the Bitso story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) 10 min test cycles, down from 60 Earnest runs its end-to-end suites in parallel, so developers find issues before their code reaches staging. [Read the Earnest story](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) How it works #### Ephemeral environments for the whole cloud-native system An ephemeral environment copies only what changed and wires it into an environment you already run, such as staging. Signadot does the wiring for requests, messages and resources. ##### Routing requests Each ephemeral environment has a routing key. A request that carries it reaches the environment's copy of a service at every hop, and the shared version of everything else. Signadot sets up the routes in your service mesh, such as Istio, or with its own sidecar. request routing key: pr-42 routed by key checkout pr-42 copy checkout baseline next hop payments shared ##### Routing messages and tasks Messages, workflow tasks and jobs carry the routing key too. Consumers and workers in an ephemeral environment take only the work tagged for it, and the baseline takes the rest, so every environment shares the same brokers and queues. One shared broker or task queue msg msg · pr-42 task task · pr-42 each takes its own pr-42 worker pr-42 only baseline the rest ##### Resource plugins for the rest Some components need a copy of their own, such as a database for a schema change, or live outside Kubernetes, such as a managed queue. Your platform team describes how to create them once, as a resource plugin. Any environment can then have its own, created when it starts and removed when it ends. pr-42 starts resource plugin Inside or outside Kubernetes database queue bucket pr-42 ends removed nothing left running ##### Across your stack - Databases Share the database with each environment's data kept apart, or give an environment a temporary database or a branch from Neon or Xata. [Data isolation](https://www.signadot.com/docs/guides/set-up-data-isolation) - Message queues One shared broker for every environment, with guides for Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub. [Message queue isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - Workflows and batch jobs The routing key travels with every workflow, task and job, down to the steps they start, so the right workers pick them up. [Temporal workers](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - Cloud resources Managed databases, queues, buckets and other cloud services outside Kubernetes, created with Terraform, a cloud CLI or a script. [Resource plugins](https://www.signadot.com/docs/concepts/resources) Where it fits #### Fast feedback from local dev to pull request merge The same ephemeral environments serve both loops of development: developers and coding agents testing as they write code, and CI testing every pull request before merge. Inner loop Developers and coding agents test as they write code. 1 2 3 In seconds on every change 1. 1 Write code A developer or a coding agent makes a change, on a laptop or in a cloud sandbox. 2. 2 Run it in an ephemeral environment Through the CLI or the MCP server, the changed code joins an ephemeral environment in your cluster, ready in seconds. 3. 3 Test and iterate The agent runs end-to-end and integration tests against real services, reads the failures, fixes them and runs them again. Outer loop CI tests every pull request before merge. 4 5 6 Before merge on every PR 1. 4 An environment for every pull request CI creates an ephemeral environment for each pull request, the same kind the inner loop uses. 2. 5 Run your test suites End-to-end, regression and performance suites run against it, from CI or in the cluster. 3. 6 Merge with confidence The change has been tested in both loops before it merges. One Signadot Platform Promote forward · Iterate back Who it's for #### Built for how your team actually works Different roles, same platform. Signadot fits into your existing workflows. Engineering Leaders ##### For Engineering Leaders You're being asked to adopt AI coding tools and increase developer productivity without increasing headcount or cloud spend. Signadot is the infrastructure layer that makes that possible. Scale testing to match the pace of code generation, without scaling costs. [Learn more about Signadot for engineering leaders](https://www.signadot.com/schedule-a-call/) *Image: Brex logo* $2M saved annually > Signadot for the preview environments use case fit what we were trying to do better than anything else. It was a more mature solution than the other stuff that we were looking at. And the return on the investment was obvious... just in infrastructure costs, it saves us about $2 million annually. *Image: Phil Burrows, Head of Platform Engineering at Brex* Phil BurrowsHead of Platform Engineering Platform Engineers ##### For Platform Engineers You need infrastructure that fits your existing stack, not another abstraction to manage. Signadot installs as an operator in your Kubernetes cluster and works with your service mesh (Istio, Linkerd), CI/CD pipelines and OpenTelemetry. Give your developers and their agents self-service environments without filing tickets. [See our docs](https://www.signadot.com/docs/overview) *Image: Laurel logo* 82% lower change failure rate > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer Developers ##### For Developers Get instant feedback on every change, against real services. Your coding agents can spin up ephemeral environments, run tests, and iterate autonomously to give you back the time you'd spend babysitting their output. [See our docs](https://www.signadot.com/docs/overview) *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer Brex, SoFi, Wealthsimple, Bitso, and many more #### Start verifying changes in minutes Create your first ephemeral environment on our playground cluster. No credit card, and no Kubernetes of your own required. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The platform #### Everything your team needs to ship with confidence A closer look at the tools behind both loops: local development against real services, an ephemeral environment for every pull request, test suites in your cluster and plans that coding agents run. Together they let your team [test AI-generated code during development and before merge](https://www.signadot.com/validate-ai-generated-code-kubernetes/), so you ship as fast as code is written. ##### Local development with real services Developers and coding agents use the CLI or the MCP server to connect a local workstation to your Kubernetes cluster over a secure, two-way tunnel. Test local changes end to end from web, mobile or API frontends, with no mocks. [Learn more about local testing](https://www.signadot.com/docs/guides/use-cases/set-up-local-testing) Your machine Agent cart.go bi-directional tunnel Kubernetes cluster frontend orders users db ##### Ephemeral environments for every pull request Your CI creates an ephemeral environment for every pull request, hundreds at a time. Preview the change in a browser or on a phone, and run functional and non-functional tests against it before merge. [Learn more about pull request environments](https://www.signadot.com/docs/guides/set-up-pr-sandboxes) signadot bot 2 minutes ago Sandbox pr-276-frontend created Routing key nm2cqkbfxqzml * * * Tests Summary - checkout e2e - login e2e - orders api ##### Run E2E and integration tests pre-merge Run your Playwright, Cypress or other suites as Jobs, inside your Kubernetes cluster, against any ephemeral environment. End-to-end and integration problems show up before merge, not after. [Explore test scenarios](https://www.signadot.com/docs/guides/use-cases/run-automated-tests) signadot job submit \--set sandbox=pr-42 --attach dispatch Your cluster Job runner group · pre-warmed pods playwright running k6 running warm pod ready routing key: pr-42 checkout pr-42 fork cart, auth, pay shared services results e2e passed exit code 0 report · video · logs back in CI ##### Signadot Plans Platform teams build secure custom actions inside your Kubernetes cluster. Developers compose them into plans that coding agents run to check their changes. Every run is deterministic and uses no tokens, so plans stay fast and cheap at scale. [Learn more about plan-based validation](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Action catalog · governed by your platform team request-http playwright k6 compose plan: checkout-flow · tag v3 1 · request-http POST /cart 2 · playwright checkout flow 3 · check exit code 0 selection hint cart or checkout code changes signadot-validate picks it Runs against sandbox pr-42 routing key on every step Plan passed report · trace · video Voice of our customers #### Customer Success Stories - [*Image: Brex customer story video* *Image: Brex logo* Brex saved $2M annually using Signadot](https://vimeo.com/1081641515/808295f00f) #### Frequently Asked Questions How is Signadot different from other solutions? Most ephemeral environment tools copy the whole stack for every change. Signadot multiplexes many ephemeral virtual environments within one physical environment, such as staging, and each one duplicates only the components a change touched. That keeps them light and fast to create, so hundreds run in parallel for developers and coding agents. What capabilities does Signadot offer? **Local development.** Run the service you are changing on your laptop, or let a coding agent do it, inside an ephemeral environment with real dependencies. **Preview environments.** An ephemeral environment for every pull request, including web and mobile frontends. **Your own automated tests.** Run Playwright, Cypress, k6 or any other suite in your cluster against each environment. **Smart Tests.** API tests that compare a changed service with the baseline and flag breaking changes. **Coding agents.** An MCP server and agent skills let agents create environments and test their own changes. How does Signadot enable agentic development? Each coding agent gets its own ephemeral environment with real services and data, whether it runs on a developer's laptop or as a background agent in the cloud. The agent runs its code against the real system, reads the errors, fixes them and runs its tests again before anyone reviews the change, without touching anyone else's work. What coding agents does Signadot integrate with? Any coding agent that supports the Model Context Protocol (MCP) or [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills), including Claude Code and Cursor. Agents create and manage ephemeral environments and test their changes from inside their own workflow. Background agents running in cloud sandboxes get their own ephemeral environment too. What is the Signadot MCP server? The [Signadot MCP server](https://www.signadot.com/docs/integrations/mcp) connects coding agents to your Kubernetes clusters through the Model Context Protocol. It ships with the Signadot CLI and gives agents tools to create and manage ephemeral environments and read cluster context, so they can test their changes against real services from a prompt. What is a Signadot ephemeral environment? An ephemeral environment, called a Sandbox in the Signadot CLI and API, holds only the components a code change touches and shares everything else in your existing environment, such as staging. Whoever uses it sees what looks like a full copy of staging, without interfering with anyone else's work. Where do ephemeral environments run? Inside the Kubernetes clusters you already run. Signadot installs as an operator, so workloads, data and tests stay in your infrastructure. The Signadot control plane manages the environments but sees only resource metadata by default, never your code or application data. How much effort is it to get started? Most teams create their first ephemeral environment in about 15 minutes. You install the Signadot operator in your Kubernetes cluster and use the CLI to create an environment and run your tests, or start on our playground cluster with no cluster of your own. How does Signadot integrate with CI/CD systems? Through the Signadot CLI, as a step in any pipeline. The pipeline creates an ephemeral environment for each pull request, runs your tests against it, and deletes it when the pull request closes. The docs have [guides](https://www.signadot.com/docs/guides/integrate-ci) for GitHub Actions, GitLab CI, Jenkins and Bitbucket Pipelines. Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Does Signadot support running automated tests? Yes. Signadot runs your existing test suites, such as Playwright, Cypress, Selenium, Postman, k6 or your own, in your cluster against each ephemeral environment, on pre-warmed runners, with the results in one place. Coding agents can also write new tests and run them in their environment, which gives them immediate feedback on their changes. What are Signadot Smart Tests? [Smart Tests](https://www.signadot.com/product/smart-tests/) are API tests you write once in Starlark. Signadot sends the same requests to the changed service and to the baseline, and a model flags the differences that matter, such as a removed field or a new error, without an assertion for every field. They cover REST APIs with JSON and complement contract tests rather than replacing them. Is Signadot secure? Yes. Signadot runs in your infrastructure, and its control plane receives only resource metadata by default, never your code or application data. It supports SSO and role-based access control, and Signadot is SOC 2 Type II. See the [security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations). What types of applications work best with Signadot? Cloud native applications built from microservices on Kubernetes, especially with 10 or more services, where testing a change against real dependencies gets hard. It also suits teams adopting coding agents, where 50 or more agents and developers need environments at once. Resource plugins bring in what runs outside Kubernetes, such as managed databases and cloud services. How does pricing work? Signadot has a free Starter plan, and paid plans that scale with usage as your team and coding agents ship more changes. See the [pricing page](https://www.signadot.com/pricing/), or [talk to us](https://www.signadot.com/schedule-a-call/) for a custom quote. How does Signadot compare to testing tools like Playwright, Cypress or Selenium? Signadot works with them rather than replacing them. Those tools run your tests. Signadot gives each change a realistic ephemeral environment for them to run in, and runs them in parallel across many environments. #### Your team is only as fast as your verification loop Coding agents write code fast. Signadot gives every change an environment and tests, so you know it works before it ships. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Latest on Scaling Agentic Development - [ *Image: Code Generation Isn't the Bottleneck Anymore: What Platform Leaders Told Us Over Dinner* ### Code Generation Isn't the Bottleneck Anymore: What Platform Leaders Told Us Over Dinner Platform, security and AI leaders agreed on one thing at our recent executive dinner: agents made code cheap, and everything after the code is now the constraint. ](https://www.signadot.com/blog/exec-dinner-agentic-coding-throughput/) - [ *Image: Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk* ### Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk Drift, late staging failures and agent pull request volume are moving cloud-native delivery's bottleneck to verification. Key themes from DZone's Cloud-Native Foundations Trend Talk. ](https://www.signadot.com/blog/dzone-trend-talk-cloud-native-foundations/) - [ *Image: Independent Deployability Has Two Halves. Most Teams Built One.* ### Independent Deployability Has Two Halves. Most Teams Built One. Teams with mature deploy tooling still ship microservices in batches, because nothing lets them trust one change on its own. Coding agents are widening that gap. ](https://www.signadot.com/blog/independent-deployability-validation-gap/) --- ## Platform Source: https://www.signadot.com/product/platform/ Summary: How every change gets an entire ephemeral environment without copying the stack. A Sandbox duplicates only the workloads that changed, in the cluster or on a laptop, plus any resource that needs its own copy, and Signadot wires them into the rest of the environment: request routing through Istio (sidecar or ambient mode), the Gateway API or the DevMesh sidecar, with the routing key in OpenTelemetry headers; message routing on a shared broker (Kafka, RabbitMQ, NATS, SQS and SNS, Pub/Sub, Temporal task queues); databases that are shared, temporary or branched with Neon or Xata; and resource plugins for anything else, inside or outside Kubernetes. RouteGroups combine the Sandboxes of several pull requests into one environment per feature. Sandbox versions of services that run outside Kubernetes, such as on ECS, Lambda, Google Cloud Run or VMs, are coming soon. Platform ### Ephemeral environments without copying your stack Signadot duplicates only the parts of an environment that a change touches or that need their own copy, and wires them into everything else. Every developer, coding agent and CI run gets what feels like an entire environment of its own, and hundreds share the cluster you already operate. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Runs in your Kubernetes cluster - Works with OpenTelemetry, Istio and Gateway API - Playground cluster available Ephemeral environments Developer checkout running on a laptop Coding agent payments + db branch its own copy of the data Pull request 482 orders-worker its own messages from shared Kafka wired into the rest Your environment, such as staging frontend checkout payments orders db kafka Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - The model #### Duplicate only what a change needs Copying the whole stack for every pull request, developer or coding agent is slow and expensive, and it limits how many environments a team can run at once. A Signadot ephemeral environment duplicates only the components that changed, or that need a copy of their own, such as a database for a schema change. Signadot wires them into the rest of the environment, so requests, messages and data reach them on the right paths. To the developer, agent or CI job using it, it behaves like an entire environment of its own. [Sandboxes docs](https://www.signadot.com/docs/concepts/sandbox) [An ephemeral environment for every change](https://www.signadot.com/solutions/ephemeral-environments/) pr-42 sees an entire environment frontend shared checkout the change duplicated payments shared orders shared db shared kafka shared The Sandbox #### One model for local development, agents and CI A Sandbox describes an ephemeral environment. It lists the workloads to duplicate, which run in the cluster or on a developer's machine, the resources to create, its labels and a time to live. Signadot gives each Sandbox a routing key and preview URLs. Because the model is the same everywhere, one kind of Sandbox serves a developer's inner loop, a coding agent working through the CLI or the MCP server, and a CI job for every pull request. Routing, data isolation and resource plugins are set up once, and every Sandbox uses them. [Sandbox spec](https://www.signadot.com/docs/reference/sandboxes/spec) [Local workloads in a Sandbox](https://www.signadot.com/docs/reference/sandboxes/local) Who creates it Developer local connect Coding agent MCP or CLI CI job sandbox apply the same model Sandbox workloads · resources · routing key · TTL How it works #### Wired in at every layer The hard part is making a partial copy behave like an entire environment. Signadot does it for every way your system talks, from synchronous calls between services to asynchronous messages and workflows, data and cloud resources. - [ ### Synchronous services Requests carry a routing key to the duplicated services, through your service mesh. ](#requests) - [ ### Asynchronous services Messages and workflows reach the right consumers on one shared broker. ](#messages) - [ ### Databases Share the database with test data kept apart, or give an environment a temporary database or a branch. ](#data) - [ ### Cloud resources Resource plugins create anything else an environment needs, inside or outside Kubernetes. ](#resources) Synchronous services #### Requests reach the duplicated services Every Sandbox has a routing key. A request that carries it reaches the Sandbox's version of each service it duplicated, and the shared version of everything else, at every hop. Services forward the key with OpenTelemetry, and Signadot sets up the routes in Istio, in Gateway API or in its own DevMesh sidecar. A browser link, webhook or curl call can carry the key as a `?sd-routing-key=` query parameter instead of a header. [Request routing docs](https://www.signadot.com/docs/guides/request-routing)[Context propagation](https://www.signadot.com/docs/guides/set-up-context-propagation) With the routing key request key a1b2c3 checkout duplicated payments shared Without it request no key checkout shared payments shared Asynchronous services #### Messages reach the right consumers On a queue or event stream, the routing key travels as a message header. A consumer in a Sandbox asks Signadot's Routes API which keys are its own and processes only those messages, and shared consumers skip them, so every environment uses the same broker and topics. The same pattern keeps Temporal workflows apart on a shared task queue. When a consumer needs a topic or queue of its own instead, a resource plugin creates one for the Sandbox. There are guides for Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub. [Message queue docs](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) [Temporal workers tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) One shared topic msg msg · key a1b2c3 msg msg · key a1b2c3 each takes its own orders-worker pr-42: its key only orders-worker shared: the rest Databases #### Each environment gets the data it needs Most ephemeral environments share the existing database and keep their test data apart. A schema change or a seeded test can get a temporary database or schema from a resource plugin, or a branch of the database through the Neon and Xata integrations. Whatever a Sandbox creates is removed when it ends. [Data isolation docs](https://www.signadot.com/docs/guides/set-up-data-isolation) [Neon integration](https://www.signadot.com/docs/integrations/databases/neon) Shared Use the existing database and keep test data apart by a key such as an organization or user ID. Temporary A resource plugin creates a database or schema for the Sandbox, seeds it, and removes it at the end. Branch Each Sandbox gets a branch that starts with the parent's schema and data, then changes on its own, with Neon or Xata. Resource plugins #### Extend a Sandbox to any resource A resource plugin lets a Sandbox create and delete anything its environment needs, in the cluster or outside it, such as a database, a queue, a storage bucket, a cloud service or a virtual machine. Your platform team writes the create and delete steps once, with Helm, Terraform, a cloud CLI or a script. Signadot runs the create steps before a Sandbox's workloads start and the delete steps when it ends, and passes outputs such as a host or credentials to the workloads as environment variables. Plugins are versioned, so each Sandbox keeps the revision it was created with. [Resources docs](https://www.signadot.com/docs/concepts/resources)[Resource plugins](https://www.signadot.com/docs/reference/resource-plugins/overview) [Example plugins](https://www.signadot.com/docs/reference/resource-plugins/examples) sandbox.yaml ``` spec: resources: - name: customerdb plugin: mariadb params: dbname: customer # the duplicated service reads its own DB env: - name: MYSQL_HOST valueFrom: resource: name: customerdb outputKey: provision.host ``` Coming soon #### Routing beyond Kubernetes Signadot runs in Kubernetes, but its routing model is not tied to it. Soon a Sandbox will also include versions of services that run outside the cluster, on platforms such as AWS ECS and Lambda, Google Cloud Run or virtual machines. Those versions will join an ephemeral environment the same way a Kubernetes workload does, so a change that spans the cluster and the rest of your cloud is tested in one environment. Request or message routing key a1b2c3 to the Sandbox's version Wherever the version runs Kubernetes today ECS soon Lambda soon Cloud Run soon VM soon RouteGroups #### Combine pull requests into one environment per feature A feature often spans several services, and each change arrives in its own pull request with its own Sandbox. A RouteGroup combines those Sandboxes into one environment, with one routing key and its own preview URLs, so the whole feature can be tested before anything merges. It selects Sandboxes by label, so a new pull request for the feature joins as soon as its Sandbox is created. With many features in flight, each gets its own RouteGroup, and they are tested side by side on the same cluster. [RouteGroups docs](https://www.signadot.com/docs/reference/route-groups/usage) feature: checkout-v2 frontend PR 311 api PR 312 one routing key one preview URL feature: refunds payments PR 318 worker PR 320 one routing key one preview URL *Image: Brex logo* Developer CSAT up 28 points > Our Signadot based tooling had a CSAT that was 28 points higher than our older Platform Engineering tech. JS John SalemSenior Software Engineer #### One environment model, from a laptop to merge. Brex moved developer testing for hundreds of engineers onto Signadot instead of maintaining a fleet of full preview environments. [Start for free](https://www.signadot.com/signup/) [Read the Brex story](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) Open standards #### Works through the standards you already run Signadot doesn't put a proprietary layer between your services. It carries context in OpenTelemetry headers and routes through your own service mesh and Gateway API resources. - ### OpenTelemetry The routing key rides in the W3C baggage and tracestate headers, which OpenTelemetry instrumentation forwards on every call. No tracing backend is needed. - ### Istio Signadot adds routes to your VirtualServices and uses Istio's own proxies, in sidecar or ambient mode, with a naming contract that lets Argo CD and Flux accept its changes. - ### Gateway API Signadot derives HTTPRoute and GRPCRoute resources from the routes you already have, with Istio or Linkerd as the implementation. - ### Linkerd, or no mesh The DevMesh sidecar routes HTTP and gRPC requests alongside Linkerd, or on its own in a cluster without a mesh. [Request routing docs](https://www.signadot.com/docs/guides/request-routing) [All integrations](https://www.signadot.com/integrations/) Architecture & security #### Your code and data stay in your cluster Only the control plane runs in Signadot's SaaS. The Operator, your workloads and your data stay in your own Kubernetes cluster. The Operator connects out over a secure tunnel, duplicates workloads, programs routes and runs resource plugins, so nothing sensitive leaves your environment. Developers, coding agents and CI drive it through the CLI, the API or the MCP server. Signadot control plane SaaS: dashboard, API and MCP server secure tunnel Your Kubernetes cluster Signadot Operator Sandbox controller RouteGroup controller Route server DevMesh sidecars Resource plugin runners Your services service-a service-b service-b copy in a Sandbox Your data never leaves the cluster [Architecture docs](https://www.signadot.com/docs/concepts/architecture) [Install the Operator](https://www.signadot.com/docs/installation/signadot-operator) The rest of the platform #### Tests run in the same environments Ephemeral environments are the foundation of Signadot. Jobs run your existing test suites inside them, Plans let coding agents write and run new tests, and Smart Tests run managed API tests on every pull request and flag regressions. [Jobs](https://www.signadot.com/product/jobs/)[Plans](https://www.signadot.com/product/plans/) [Smart Tests](https://www.signadot.com/product/smart-tests/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster #### Frequently asked questions How does Signadot create an ephemeral environment without copying the stack? Each ephemeral environment is a Sandbox that duplicates only the workloads that changed, plus any resource that needs its own copy, such as a database. Signadot wires those copies into the existing environment with a routing key. Requests and messages that carry the key reach the Sandbox's versions, and everything else uses the shared ones, so hundreds of Sandboxes can share one cluster. How do ephemeral environments handle databases and message queues? Message queues stay shared. The routing key travels as a message header, and consumers in a Sandbox process only their own messages, with guides for Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub. For databases, a Sandbox can share the existing database with its data kept apart, get a temporary database from a resource plugin, or use a branch through the Neon or Xata integration. Does Signadot work with Istio, ambient mode and Gateway API? Yes. With Istio, Signadot adds routes to your VirtualServices and uses Istio's own proxies, including in ambient mode, without a sidecar of its own. It also derives Gateway API HTTPRoute and GRPCRoute resources, and its DevMesh sidecar handles clusters that run Linkerd or no mesh at all. Do developers, coding agents and CI use the same environments? They use the same model. A developer runs a service on a laptop as a local workload in a Sandbox, a coding agent creates Sandboxes through the CLI or the MCP server, and CI creates one for each pull request. Each Sandbox has its own routing key, so all of them work side by side on one cluster without interfering. What is a RouteGroup? A RouteGroup combines several Sandboxes, usually the pull requests that make up one feature, into one environment with a single routing key and its own preview URLs. It selects Sandboxes by label, so new pull requests join automatically, and many features can be tested in parallel. Can Signadot manage resources outside Kubernetes? Yes. Signadot runs in your Kubernetes cluster, and resource plugins create and delete resources inside or outside it, such as managed databases, queues, storage buckets and virtual machines, with Terraform, Helm, a cloud CLI or a script. Sandbox versions of services that run outside Kubernetes, such as on AWS Lambda, Google Cloud Run or virtual machines, are coming soon. #### Spin up your first ephemeral environment in minutes. Give developers, coding agents and CI an entire environment for every change, on the cluster you already run, without copying the stack. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Smart Tests Source: https://www.signadot.com/product/smart-tests/ Summary: AI-powered regression detection. Write API calls once in Starlark, and Signadot runs them against both the PR version and the stable version of a service in live sandboxes, using AI to flag real behavioral regressions instead of noise. Smart Tests ### Find every regression in your pull requests Smart Tests validates every PR in a live Kubernetes sandbox. Write the API calls once and AI compares the PR version against the stable version of your service, flagging real regressions instead of noise. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - No assertions to write or maintain - Relevance-scored findings posted to the PR smart-test.star API calls, no assertions runs twice checkout PR version checkout stable version AI diff Risk score High Medium Low Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - How it works #### Deep validation on every code change A SmartTest is a short script of API calls. Signadot runs it against both versions of your service, captures the behavior, and lets AI decide which differences actually matter. PR #142 opens a sandbox Your cluster Test runner runs twice checkout PR sandbox checkout stable version AI diff Findings signal, not noise 1 ##### Write API calls in Starlark Define the endpoints to exercise in a simple Python-like script. Capture is built in, so there are no assertions or expected values to encode. orders.star ``` resp = http.get( "/api/orders", capture=True) print(resp.status) ``` 2 ##### Trigger on every sandbox Triggers match tests to services, so the moment a PR sandbox is created or updated the right tests run automatically. Git-based tests run from CI too. sandbox created automatic test run the matching smart tests 3 ##### AI compares both versions Captured request and response pairs from the PR and stable versions are diffed by models trained on your traffic patterns, scoring each difference by relevance. Field PR version Stable version id 4471 4471 total missing 19.99 status "paid" "paid" currency "usd" "usd" 4 ##### Findings land in the PR High, medium, and low relevance findings post to the pull request as a risk score, so reviewers see behavioral impact next to the code diff. signadot posted a risk score on PR #142 1 High 0 Medium 5 Low #### Catch what assertion-based tests miss Traditional tests only check what someone thought to assert. Smart Tests observes the full runtime behavior of your service and flags any change that matters. checkout.star ``` resp = http.get( "/api/orders", capture=True) ``` ##### Calls, not assertions You only describe which endpoints to hit. The comparison against the stable version is the assertion, so compatible API changes never break a test. Raw differences timestamps · IDs · fields AI model 1 regression High ##### Signal, not flakiness Models learn from historical runs which variations are expected, like timestamps and random IDs, and which are genuine regressions worth a human look. Your cluster Test runner orders payments postgres ##### Real services, real data paths Tests execute on managed runner pods in your cluster against sandboxes that share real dependencies, so results reflect production behavior, not mocks. *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer #### Behavioral validation on every PR, without the test debt. Bitso gives 250+ engineers an isolated environment per PR on Signadot, the same environments Smart Tests validate against on every change. [Start for free](https://www.signadot.com/signup/) [Read the Bitso story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) The output #### A risk score you can read at a glance Every execution produces a structured result, not a wall of raw diffs. Findings are scored by relevance and broken down by where they appear in the traffic. ##### Relevance, not raw diffs Each difference between the two versions is scored by how unexpected it is. - High flags contract-impacting changes like removed fields, type changes, and new error codes. - Medium marks unusual differences worth a look before merging. - Low covers expected variation like timestamps and trace IDs. Surfaced, never blocking. ##### Request and response, separated - REQ Request findings cover the traffic your service sends downstream as it handles the call. - RES Response findings cover what consumers of your API actually receive, which is where breaking changes live. [Follow the Smart Tests quickstart in the docs](https://www.signadot.com/docs/tutorials/quickstart/smart-tests) get-checkout · execution Succeeded 1 High 0 Medium 5 Low Finding Stable PR Request Traceparent Low "00-4c3a…" "00-e6dd…" Request X-Request-Id Low "5a68…" "5849…" Response order.total High 19.99 removed Response Date Low "03:09:18" "03:09:52" checkout-flow 1/1 checks passed #### Frequently asked questions Is this a contract testing tool like Pact? No. Smart Tests is not a direct alternative to consumer-driven contract testing tools like Pact. Smart Tests is a provider-focused validation tool that gives you a "risk score" for your PR by analyzing its runtime behavior, rather than coordinating contracts between consumer/provider teams. How does the AI system in Smart Tests work? The AI system performs two main functions: First, it analyzes API responses from both versions to identify structural, data type, and content differences. The models determine which differences actually impact compatibility rather than flagging every change. Second, it reduces test flakiness by learning from historical test runs, distinguishing between expected variations (timestamps, random IDs) and genuine breaking changes. The system adapts to your specific APIs over time, improving its accuracy with continued use. What kinds of contract breakages can Smart Tests detect? Smart Tests detects: \- Schema changes: Removed fields, changed data types, altered object structures \- Behavioral changes: Different response codes, new error conditions, altered validation rules ‍ Because Smart Tests observes runtime behavior rather than just validating static schemas, it can identify subtle issues that traditional approaches miss. How can Smart Tests be integrated with my CI/CD pipeline? Integration requires two components: 1) The Signadot CLI integrated into your pipeline to create sandboxes for each PR 2) Smart Test triggers configured to automatically run tests against those sandboxes How long does it take to set up Smart Tests? Most teams have their first Smart Tests running in under an hour: 1) Install the Signadot operator on your Kubernetes cluster (10-15 minutes) 2) Write your first test in Starlark to call your API endpoints (5-10 minutes) 3) Configure triggers to run the test on your sandboxes (5 minutes) 4) Integrate sandbox creation with your CI/CD pipeline (15-30 minutes) Since Smart Tests runs against your existing staging/pre-production environment, you avoid the typically time-consuming process of setting up separate test environments or managing test data. What is the maintenance overhead of Smart Tests when APIs change? Smart Tests minimizes maintenance when APIs evolve: \- When you make compatible API changes (adding fields, extending functionality in backward-compatible ways), Smart Tests automatically recognizes these as non-breaking and doesn't require test updates. \- For intentional breaking changes, you typically only need to update the test code that calls the changed endpoints - not assertions or expectations, since those are handled by the AI comparison system. #### Give every PR a risk score before review Most teams have their first Smart Tests running in under an hour, on the cluster and staging environment they already operate. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Jobs Source: https://www.signadot.com/product/jobs/ Summary: Run existing Playwright, Cypress, k6, or custom test suites on pre-warmed runner pods inside your cluster, automatically routed to the sandbox under test. Logs stream to CI and reports, screenshots, and videos upload as artifacts. Jobs ### Run your test suites where your services run Jobs execute your existing Playwright, Cypress, k6, or custom suites on pre-warmed runner pods inside your cluster, automatically routed to the sandbox under test. Logs stream back to CI and artifacts upload on completion. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Bring any test framework or container image - Test execution stays inside your cluster playwright e2e login, cart and checkout specs submit Your cluster Runner pod pre-warmed rk: pr-142 checkout sandbox fork Shared services auth cart pay report Passed before merge 3 of 3 Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - How it works #### From CI trigger to test report in one submission A Job is a single unit of test execution. You define the runner environment once, then every submission runs your suite against the right sandbox with no environment setup in the pipeline. PR #142 opens a sandbox triggers Your cluster Runner pod checkout.spec.ts rk: pr-142 checkout sandbox fork Shared services auth cart pay checks e2e passed back on the PR 1 ##### Define a runner group Point it at your test image, resource limits, and secrets. The Operator keeps warm pods ready in your cluster so there is no cold start per run. runnergroup: e2e-suite image e2e · cpu 2 · secret ci keeps warm Warm pods pod ready pod ready pod ready pod ready 2 ##### Submit from CI or the CLI One command submits a script or command with a routing context binding it to the sandbox or route group under test. CI · checkout ``` $ signadot job submit ``` dispatches Runner pod already warm 3 ##### Traffic routes itself The routing key is injected into the HTTP traffic your suite emits. Requests hit the forked services under test and shared services for everything else. rk: pr-142 on every request the suite sends checkout sandbox fork auth, cart, pay shared services 4 ##### Results land in the PR Logs stream back to the CI job in real time. Reports, screenshots, and videos upload as artifacts you can pull from the dashboard or API. e2e / playwright passed artifacts report.html video.mp4 Execution infrastructure #### Bring the frameworks you already use Jobs do not replace the tests your teams trust. A runner group points at your own container image with your service accounts and resource limits, and the Operator keeps pre-warmed pods ready so dispatch is fast. - Any suite that runs in a container, from Playwright to fully custom harnesses - Your image, your service accounts, your resource limits - Pre-warmed runner pods, no cold start per run *Image: Playwright**Image: Cypress**Image: Grafana k6**Image: Selenium* *Image: Earnest logo* Test cycles cut from 60 to 10 minutes > Signadot is a key part of our solution for the most critical parts of our testing pyramid. For integration and end-to-end testing we need to have a sandbox up and running because they need a live service to talk to. EE Early EhlingerDev Experience Lead #### The hardest layers of the testing pyramid, covered pre-merge. Earnest runs its integration and end-to-end suites in parallel against live sandboxes on every change, catching problems before they reach staging. [Start for free](https://www.signadot.com/signup/) [Read the Earnest story](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) Routing #### Auto-routed to the sandbox under test A Job declares a routing context that binds it to a sandbox or route group. Signadot injects the matching routing key into the HTTP traffic the suite emits, so requests reach the forked services under test and shared services everywhere else. - Off-the-shelf suites work with no code changes - Execution waits for sandbox readiness, so results are coherent - The same suite validates any number of sandboxes in parallel Your suite unchanged rk: pr-142 checkout sandbox fork Shared services everything else Results #### Results where your team already looks Attach from the pipeline and output streams back in real time with a real exit code, so a failing run fails the check. Everything the suite produced is collected automatically when the run completes. - Streamed logs and exit codes in the CI job - Reports, screenshots, and videos upload as artifacts - Run history and downloads in the dashboard and API [Read the Jobs reference in the docs](https://www.signadot.com/docs/reference/jobs) signadot/hotrod pr-220 into master All checks have passed 7 successful checks - Go / build (pull\_request) 1m 32s - Go / sandbox-router (pull\_request) 48s - Go / sandbox-location (pull\_request) Skipped - Go / sandbox-driver (pull\_request) Skipped - Sandbox: pr-220-route Routing key: gbscspdnzvcc0 12s #### Jobs FAQ What test frameworks can Jobs run? Any framework that runs in a container. Teams run Playwright, Cypress, k6, pytest, and fully custom suites by pointing the runner group at their own image. A Job executes a script or command inside that image, so nothing about your existing tests needs to change. Where do Jobs execute? Inside your Kubernetes cluster, on pre-warmed runner pods managed by the Signadot Operator. The Signadot control plane orchestrates scheduling and stores results, while the test traffic itself stays between the runner pod and the services on your cluster. How does a Job target a specific sandbox? Each Job declares a routing context that binds it to a sandbox or route group. Signadot injects the matching routing key into the HTTP traffic the Job emits, so requests flow through the sandboxed versions of changed services and reach shared services for everything else. Off-the-shelf suites work without code changes. How do results get back to my CI pipeline? Submit a Job from CI with the Signadot CLI and attach to it. Stdout and stderr stream back to the CI log in real time, and any declared artifacts such as HTML reports, screenshots, and videos are uploaded when the Job completes and can be downloaded from the dashboard or API. #### Point your existing suites at every change Run the tests you already have against isolated sandboxes on the cluster you already operate, with results back in the PR in minutes. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Plans Source: https://www.signadot.com/product/plans/ Summary: Reusable validation workflows authored from natural language and run against live sandboxes. Platform teams govern an action catalog, and developers and coding agents run tagged plans by name until every check passes before the PR opens. Plans ### Reusable agent-native validation workflows A Plan is a reusable validation workflow authored from natural language and run against a live Sandbox. Developers and coding agents invoke it by name and get an end-to-end verdict on their change in seconds. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Authored from a prompt, versioned in git - Runs against real services in seconds Prompt Validate the ride request flow signadot-plan plan: ride-request-flow · v3 request-http playwright k6 runs against Your cluster location-svc sandbox fork rides · geo · postgres shared Passed in 14s every step green Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - How it works #### From prompt to reusable validation in four steps Plans separate what to validate from how to validate it. The platform team governs the building blocks, and every developer and agent composes them into fast, repeatable checks. Create once Validate the ride request flow a prompt signadot-plan author from prompt composes Action catalog request-http · playwright · k6 Plan library ride-request v1 · hint: ride flow changes Run on every change diff: location-svc a code change signadot-validate picks and runs plans ride-request Sandbox run location-svc fork · rides · geo All plans pass fail: fix and re-run 1 ##### Govern the Action catalog Actions like request-http, playwright, and k6 are defined in markdown with typed inputs and outputs, versioned in git, and curated by your platform team. request-http.md an action governed Inputs url string method string Outputs status number 2 ##### Author from natural language A developer or agent describes what should be validated. The signadot-plan skill compiles the prompt into a plan composed of catalog actions. validate ride request a prompt signadot-plan plan: ride-request request-http playwright k6 3 ##### Tag it into the library Plans are immutable, tagged, and carry a selection hint that tells agents when to use them. The library grows into shared coverage of your user flows. ride-request hint: ride flow changes v1 tag Library checkout-flow ride-request search-api 4 ##### Run on every change The signadot-validate skill picks the right plan for a diff and runs it in a Sandbox against real services. The agent fixes what broke and re-runs until the change passes. diff: location-svc signadot-validate ride-request v3 · running in a sandbox All plans pass 14s *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer #### Validation that keeps up with agent-speed development. Bitso scaled branch-based development for 250+ engineers and 200+ microservices on Signadot, with coding agents validating changes the same way. [Start for free](https://www.signadot.com/signup/) [Read the Bitso story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Agent skills #### Two skills close the loop for coding agents Agents interact with Plans through skills, discovering capabilities the moment they need them. One skill writes the validation, the other runs it on every change and keeps iterating until the change actually works. Skill ##### signadot-plan Authors the validation once A developer or agent describes what should be validated in plain language. The skill reads the action catalog, drafts the plan, runs it against your live services to confirm it works, then tags it with a selection hint and commits it alongside the code. Skill ##### signadot-validate Runs and iterates on every change The agent does not just get feedback it can act on. It gets feedback and iterates. The skill reads the diff, picks the matching plan, spins up a Sandbox with the change wired in, runs the plan, fixes what broke, and re-runs until the change is validated working against the real system. claude - ~/location-service (sandbox) \> signadot-plan create a plan that validates the ride request flow end to end reading the action catalog… drafting steps: request-http · playwright · k6 ✓ created plan "ride-request-flow" selection hint: runs when ride request or location code changes tagged v1 · committed to .signadot/plans/ \> signadot-validate change detected in location-service matched plan "ride-request-flow" by selection hint creating sandbox with location-service fork… running plan against the sandbox… ✗ step playwright failed: LocationName missing in response fixed 4 call sites · re-running… ✓ all steps passed in 14s ✓ change validated against real services [Install the agent skills from the docs](https://www.signadot.com/docs/integrations/coding-agents/agent-skills) [See how coding agents close the loop](https://www.signadot.com/solutions/agentic-development/) #### Governed by platform teams, used by everyone Plans give agents and developers an expressive validation vocabulary without giving up control over what runs inside the cluster. request-http · playwright · k6 approved run-anything denied ##### Platform-governed catalog The action catalog defines what validation is allowed and what it can touch. Agents compose plans from a vocabulary you control, not arbitrary scripts. 14s a plan run validate iterate fix and re-run ##### Fast enough for the inner loop Plans run in seconds against a live Sandbox, so agents validate every iteration as they work instead of waiting for a pipeline at the end. ride-request selectionHint v1 v2 v3 ##### Versioned and rediscoverable Tagged plans with selection hints are rediscovered and rerun across sessions and teams, so validation is written once instead of rebuilt every time. #### Plans FAQ What is a Signadot Plan? A Plan is a small, reusable validation workflow. It is authored from a natural language prompt, compiled into a sequence of typed action invocations, and stored as a versioned artifact that developers and coding agents invoke by name. Running a plan validates a change end to end against live services in a Sandbox. What are Actions? Actions are the typed, deterministic building blocks Plans are composed from. The catalog includes actions like request-http for API checks, playwright for browser flows, and k6 for load scenarios. Each action is defined in markdown with declared inputs and outputs, checked into a git repo your platform team controls, and synced into the action registry. How do coding agents use Plans? Through two agent skills. The signadot-plan skill authors a new plan from a prompt, tags it, and gives it a selection hint. The signadot-validate skill reads a code diff, picks the matching plan by its selection hint, runs it in a Sandbox against real services, and iterates on failures until the change passes. Both work with the Signadot MCP server or fall back to the CLI. Where do Plans execute? Plan steps run in isolated, rootless execution environments on Plan Runner Group pods inside your Kubernetes cluster. The control plane compiles and schedules the plan, resolves parameters and secrets, and collects outputs and artifacts, while the validation traffic itself stays on your cluster. #### Validate against your real system before the PR opens Ship working, agent-generated code at scale with lightning-fast, deterministic validation that runs before the PR opens. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Pricing Source: https://www.signadot.com/pricing/ Summary: A free Starter plan; a usage-based Business plan from $250 a month, metered on Sandboxes (ephemeral environments) created, test invocations and concurrent devboxes, with a 30 day free trial; and Enterprise plans with custom pricing. Business is also self-serve on AWS Marketplace, and Enterprise comes as an AWS private offer. ### Signadot Pricing Ephemeral environments plus the Jobs, Smart Tests and Plans that run tests in them, priced to scale as your team and your coding agents ship more changes. One platform, bundled pricing that grows with you. #### How pricing works Sandboxes are billed per creation. Updates to existing Sandboxes don't count toward billing. A Test Invocation is one Job you submit or one Smart Test run on a Sandbox. Each counts once, however many tests it runs, whether end-to-end, integration or API tests. We track concurrent devboxes. A devbox is either a local workstation or a Cloud Development Environment (CDE) that integrates with Sandboxes. Sandboxes created per month 100 The number of new sandboxes your team creates in a typical month. The Business plan includes 100 sandboxes. Extra sandboxes are billed in blocks of 50. Concurrent devboxes 15 The maximum number of devboxes running at the same time. The Business plan includes 15 concurrent devboxes. Extra capacity is billed in blocks of 5. Test Invocations per month 200 The number of Jobs and Smart Test runs you start in a month. The Business plan includes 200 Test Invocations. Extra Test Invocations are billed in blocks of 100. ##### Starter Free Ideal for individual developers. * * * 50 sandboxes/month 100 Test Invocations/month 10 Concurrent devboxes 1 Kubernetes cluster 1 Resource Plugin Local Development and Debugging AI Powered Contract Testing Pull Request Previews Run E2E Tests pre-merge Community Slack Support [Get started](https://www.signadot.com/signup/) ##### Business 30 day free trial $250 /month 10% discount for yearly subscription Perfect for small development teams * * * 100 sandboxes/month 200 Test Invocations/month 15 Concurrent devboxes Everything in free plus 3 Kubernetes clusters 5 Resource plugins SSO Add On (+$250/month) Community Slack and Email Support Pricing is a la carte, so you can buy more sandboxes, concurrent devboxes and Test Invocations. Excluding the SSO add-on, the maximum monthly spend allowed under this tier is $2,050 per month. [Get started](https://www.signadot.com/signup/) ##### Enterprise Custom For teams that need support as they grow and want help scaling their testing workflow * * * Custom sandboxes, Test Invocations & concurrent devboxes Unlimited Clusters and Plugins Multi-cluster RouteGroups SSO Integration & SOC2 Report Premium Support (Dedicated Slack) Custom Contracts and Multi-year Discounts Influence on Product Roadmap Dedicated white-glove Pilot [Contact sales](https://www.signadot.com/schedule-a-call/) AWS Marketplace ##### Buy Signadot through AWS Marketplace Signadot is listed on AWS Marketplace, so your subscription bills through your AWS account alongside the rest of your AWS spend. That spend can count toward an existing AWS commitment, such as an Enterprise Discount Program agreement. [View the AWS listing](https://aws.amazon.com/marketplace/pp/prodview-d4g65y3cpqvvw) [Ask for a private offer](https://www.signadot.com/schedule-a-call/) Business is self-serve. Subscribe on the listing yourself, with no sales call: the same base plan and add-ons as above, on a 1-month or 12-month contract (12 months saves 10%). Enterprise comes as a private offer. We put your custom pricing and terms into an AWS private offer. You accept it in your AWS account, and it bills through AWS like any other Marketplace purchase. Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering #### Savings calculators Developer bandwidth savings Infrastructure cost savings Quality savings #### How teams use Signadot AI coding agents and developers use Signadot to validate code changes rapidly in production-like environments. ##### Ad-hoc Testing for Agents & Developers Coding agents and developers spin up lightweight sandboxes to test changes instantly, with no shared staging environment needed. Instant sandbox creation from IDE or CI Works with Cursor, VS Code, and AI coding agents Test against real dependencies in your Kubernetes cluster [Learn more about ad-hoc testing](https://www.signadot.com/docs/guides/developer-environments) ##### Automated Testing Pre-merge Run API tests, end-to-end tests, and integration tests in isolated sandboxes on every pull request. Debug issues by inspecting API traffic and cluster data. API, integration, and end-to-end tests in sandboxes Inspect API traffic and cluster data for debugging AI-powered regression detection pre-merge [Explore testing features](https://www.signadot.com/product/smart-tests/) #### Frequently asked questions Where is Signadot hosted? The Signadot operator runs securely in your Kubernetes cluster. The Signadot Dashboard and API are hosted in our cloud which runs in AWS. What is the Pricing model? Our pricing is usage-based: you pay for Sandboxes created, Test Invocations and concurrent devboxes. Each Job you submit and each Smart Test run counts as one Test Invocation, however many individual tests it runs. You can track your usage in real time in the dashboard. What are Test Invocations? A Test Invocation is one run of tests through Signadot. A Job runs your own suite, such as Playwright, Cypress, Postman or k6, in your cluster against a Sandbox. A Smart Test run compares a changed service with the baseline. Each counts once, however many tests it contains. Tests your CI runs on its own against a Sandbox are not Test Invocations. How do I estimate the number of Sandboxes and Test Invocations? Start with your monthly pull request volume. You choose which pull requests get a Sandbox, and not every one needs it. Test Invocations follow how often you run Jobs or Smart Tests: once per pull request comes to about one per Sandbox, and on every commit to 3 to 5 per Sandbox, since pull requests usually have several commits. Concurrent devboxes are usually 70 to 100% of the developers who use local connect. How do I pay? For the Business plan you can pay by credit card or ACH, or subscribe through [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-d4g65y3cpqvvw) and pay on your AWS bill, where it can count toward an existing AWS commitment. For the Enterprise plan, contact us for flexible payment options, including an AWS Marketplace private offer. What is a Sandbox? A Sandbox is one ephemeral environment, the unit Signadot counts for billing. It duplicates only what a change needs, usually the changed microservices, and wires them into your existing environment with a routing key, so that change's requests and messages reach its versions and everything else stays shared. Do you offer a free trial? Yes! We offer a 30 day free trial of the Business plan. The Starter plan is free forever. What security standards does Signadot follow? We follow the best practices around privacy and security. Signadot is SOC 2 Type 2 certified. All data at rest is encrypted and TLS used for network communication. What does Professional Services cover? We offer Professional Services to help with various tasks like Installation & Configuration, CI integration, OpenTelemetry instrumentation and much more. Email us at [info@signadot.com](mailto:info@signadot.com) to learn more. #### Your team is only as fast as your validation loop AI coding agents generate code fast. Signadot makes sure every change actually works before it ships. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Solutions ## Ephemeral Environments Source: https://www.signadot.com/solutions/ephemeral-environments/ Summary: Lightweight per-change environments that virtualize your Kubernetes cluster. Real services and dependencies, ready in seconds and gone when the change ships. Ephemeral Environments ### An ephemeral environment for every change. Give every developer and every coding agent a lightweight per-change environment that virtualizes your Kubernetes cluster. Real services, real dependencies, ready in seconds and gone when the change ships. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Runs on the cluster you already operate - Plugs into any CI and any coding agent Your change payments \- order.ProductId + order.ProductCode spins up Ephemeral environment payments your change live Shared stable dependencies auth orders db queue validation Change validated then the environment tears down Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - The Signadot approach #### A full-stack environment per change, without the full-stack price. Signadot virtualizes your stack. It deploys only the services a change touches and routes everything else to the shared stable dependencies already running. Each change gets a real, full-stack environment that stays lightweight because nothing else is copied. Cloning the cluster per change is faithful but slow and costly. A shared staging environment is cheap but turns into a queue. Signadot gives every change its own environment while thousands share one staging cluster, with no contention. The four ways to build ephemeral environments on Kubernetes, and what each one costs, are compared in the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/). [Read the architecture in the docs](https://www.signadot.com/docs/concepts/sandbox) The traditional options Clone the cluster per change $$$ Faithful, but slow to spin up and far too expensive to give everyone one. One shared staging environment queue Cheap, but every change competes for the same environment and waits its turn. Signadot replaces both A new approach Signadot scalable A virtualized full-stack environment per change. Lightweight, isolated, and ready in seconds, with no queue. How it works #### Fork what changed, share everything else. A Signadot Sandbox is an ephemeral environment. It doesn't rebuild your stack. You fork only the few services your change touches. Everything else runs on your existing integration/staging cluster. Isolation fits the dependency: requests route to the forked service, while a stateful one like a database can get its own ephemeral copy. Your change runs against a real, full-stack system. Shared dependencies request service-1 and service-2 service-3, db Tagged request request routing key tagged service-1b fork service-3, db shared 1. 1 ### Deploy only the delta A change forks the one or two services it modifies. There is nothing to clone and no full stack to rebuild, so the environment is ready in seconds. 2. 2 ### Route the rest to shared deps Every request the fork does not own falls through to the shared stable dependencies already running, including services, databases, queues, caches, and third-party APIs. 3. 3 ### Virtualize the full stack The fork plus the shared dependencies present as one complete, production-like stack. The change is exercised end to end, not against mocks or a drifting copy. 4. 4 ### Scale to thousands Because each environment reuses the same shared dependencies, thousands run side by side on the cluster you already operate. Each tears down when its change ships. #### Thousands of full-stack environments, virtualized onto your cluster. Signadot virtualizes ephemeral environments onto the Kubernetes cluster you already run. Each one duplicates only the services that changed and routes everything else to shared stable dependencies. Thousands spin up in parallel, in seconds. Ephemeral environments, in parallel env-7f3a env-b21c env-9e04 env-4d8b env-c6f1 \+ thousands more all share Shared stable dependencies on your cluster Services Message queues Databases Caches Thousands in parallel Per-PR, per-task, and per-developer environments run side by side, with no queue, no scheduling, and no contention. Seconds to spin up Only the changed services deploy. Everything else routes to shared stable dependencies, so there is nothing to clone and no Docker builds to wait on. Full-stack fidelity Services, message queues, databases, caches, and third-party APIs. Every layer a change touches, exactly like production. *Image: Brex logo* $2M saved annually > Signadot for the preview environments use case fit what we were trying to do better than anything else. It was a more mature solution than the other stuff that we were looking at. And the return on the investment was obvious... just in infrastructure costs, it saves us about $2 million annually. *Image: Phil Burrows, Head of Platform Engineering at Brex* Phil BurrowsHead of Platform Engineering #### Every environment your team needs, on the cluster you already run. Replace cluster clones and staging queues with ephemeral environments that share your existing stack. Most teams have their first one running in an afternoon. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### One primitive, every workflow. The same ephemeral environment serves the whole team: a developer's inner loop, a coding agent's validation run, and the final check before merge. Coding agents #### Give agents a real cluster to validate against. Coding agents need real services to test against. Each agent task gets its own ephemeral environment, isolated from every other task and created through the Signadot MCP server in natural language. The agent runs end-to-end, fixes what breaks, and opens a PR you can trust. - One environment per task, isolated and reaped fast - Native MCP server for agent-driven provisioning - Same governance as developer environments: RBAC, audit, quotas [Coding agent environments](https://www.signadot.com/solutions/agentic-development/) [Coding agent docs](https://www.signadot.com/docs/integrations/coding-agents) agent env · task-7 TTL: 1h agent env · task-8 TTL: 30m agent env · task-9 TTL: 2h Local development #### Code locally, run against the real stack. Run the service you are changing on your laptop and route it into an ephemeral environment on the cluster. Your local code talks to real dependencies, with hot reload and no image builds, and without running the full stack on your machine. - No full-stack on your laptop, just the service you own - Live traffic and API overrides for fast debugging - Real dependencies, not mocks or a drifting local copy [Local development](https://www.signadot.com/solutions/local-development/) [Local development docs](https://www.signadot.com/docs/guides/local-development) your service on your laptop, hot reload $ signadot local connect Your cluster: shared stable deps auth orders db queue Pre-merge PR validation #### Catch integration bugs before merge. When a PR opens, your CI provisions an ephemeral environment for the change. Reviewers open a live URL, end-to-end tests run against real services, and the environment tears down automatically when the PR merges or closes. The integration break shows up on the PR, not three steps downstream in staging. - A preview URL per PR, posted as a comment - E2E against real dependencies, not mocked stubs - Auto-teardown on merge or close, with TTL backstop [Preview environments](https://www.signadot.com/solutions/preview-environments/) [CI/CD integration docs](https://www.signadot.com/docs/guides/integrate-ci) PR opens, CI provisions an environment $ signadot sandbox apply Preview URL posted to the PR pr-423.preview · reviewers click E2E runs against real dependencies 42 passed · 0 failed PR merges, environment torn down infra reclaimed automatically #### Plug into the CI you already run. Pre-built integrations for every major CI and GitOps tool. Add a few lines to your pipeline and your team has per-PR ephemeral environments by the end of the day. Routing works through the built-in devmesh or your existing service mesh. - *Image: GitHub Actions* ### GitHub Actions Drop in the Signadot Action. Per-PR environments, preview URLs as PR comments, auto-teardown on merge. - *Image: GitLab CI* ### GitLab CI Pipeline templates for per-MR environments that tear down on merge or close. - ### ArgoCD Manage ephemeral environments as Signadot Sandboxes alongside your GitOps workflow. - *Image: Jenkins* ### Jenkins Shared library plus CLI calls. Existing pipelines pick up environment provisioning in a few lines. - *Image: Bitbucket Pipelines* ### Bitbucket Pipelines Pipe definitions for per-PR environments. Works with self-hosted and cloud runners. - ### CircleCI Orbs and CLI integration. The same per-PR lifecycle, wired into your existing workflows. [Browse all CI and GitOps integrations](https://www.signadot.com/integrations/) #### Ephemeral Environments FAQ What is an ephemeral environment? An ephemeral environment is a short-lived, production-like environment that exists for the lifetime of a single change. Signadot does not copy the stack to make one. It duplicates only what the change needs, usually the services it touched, and wires those copies into the shared environment, so the change gets what behaves like a full stack of its own. It spins up in seconds and tears down when the change ships. Are Signadot Sandboxes the same as ephemeral environments? Yes. A Signadot Sandbox is an ephemeral environment. It is the core primitive the platform is built on, used for local development, pull request validation, integration testing, and coding agent runs. How is this different from cloning a cluster per change? Cloning the whole cluster for every change is slow and expensive, and it caps out long before a team can give every change its own environment. Signadot takes a different path. Each environment duplicates only what its change needs, usually the changed services, and wires them into the shared stable dependencies, so thousands run in parallel on the cluster you already operate. How do ephemeral environments work with coding agents? Each agent task gets its own ephemeral environment, isolated from every other task, created through the Signadot MCP server in natural language. The agent runs end-to-end against real dependencies, fixes what breaks, and opens a PR that is already validated. Per-task environments reap quickly when the run finishes. How do they fit into CI and GitOps? When a PR opens, your CI calls Signadot to create the environment, posts a preview URL, and tears it down on merge or close. Pre-built integrations exist for GitHub Actions, GitLab CI, Jenkins, Bitbucket Pipelines, and CircleCI. With ArgoCD, you can manage environments as Signadot Sandboxes alongside your GitOps workflow. How does request routing work? Requests for a change carry its routing key in the OpenTelemetry baggage header your services already pass along, and reach that change's versions. Everything else goes to the shared stable dependencies. On Istio, Signadot uses the mesh's own routing. With Linkerd or no mesh, the lightweight DevMesh sidecar routes requests. Messages carry the same key, so asynchronous work stays in the right environment. How long do ephemeral environments live? As long as the change they represent. A PR environment lives while the PR is open. An agent task environment lives for the run. Platform teams can set a global TTL so anything left behind is reaped automatically. #### Lightweight ephemeral environments built for agentic scale. Give every developer and every coding agent a full-stack environment for every change and every PR, scalable to thousands running in parallel on the cluster you already run. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Preview Environments Source: https://www.signadot.com/solutions/preview-environments/ Summary: Kubernetes preview environments for every pull request at enterprise scale. Each PR environment duplicates only what the change needs on the cluster you already run, with real databases, message queues and services behind it, and spins up in seconds. PR Environments ### Kubernetes preview environments for every pull request. Give every pull request its own PR environment on the Kubernetes cluster you already run, with your real databases, message queues and services behind it. Signadot duplicates only what each change needs, so thousands of pull request environments run side by side for your developers and coding agents. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Real data and dependencies in every PR environment - Thousands of previews on one cluster, each in seconds Pull requests PR #142 checkout PR #143 payments PR #144 search fork only what changed Your cluster checkout · fork pr-142.preview payments · fork pr-143.preview search · fork pr-144.preview auth · orders · db · queue · cache shared real dependencies Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Previewing a change shouldn't mean cloning your whole stack. On Kubernetes, the two usual options both break down: a full environment per PR is too slow and expensive, and one shared staging environment turns into a queue. Coding agents only widen the gap, pushing more PRs than either model can serve. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares every approach. Every PR clones Every service cost climbs per PR $$$ ##### Duplicating doesn't scale Duplicating every service for every PR is slow to provision and expensive to run. Cost climbs with each service and each open PR, so teams ration who gets an environment. PR #142 PR #143 wait in line Staging one bad deploy breaks it blocked ##### Shared staging is a bottleneck When everyone shares one environment, changes queue for their turn, and one bad deploy breaks it for the whole team. Reviewers wait, and integration bugs hide behind each other. Coding agents open PRs fast PR PR PR PR ##### Coding agents multiply PRs Agents open changes faster than people ever did. Heavyweight environments can't keep pace, so agent-written code merges with little validation and bugs surface downstream. The Signadot approach #### Fork what changed, share everything else. A Signadot preview duplicates only what a change needs, usually just the services it touches, and routes requests and messages for everything else to the shared, real dependencies around them. Each PR runs against a complete, production-like system, without copying your stack. Shared cluster request checkout and service-2 service-3, db Tagged PR request PR #142 routing key tagged checkout PR fork service-3, db shared 1. 1 ### Deploy only the delta The preview forks the one or two services the PR modifies. There is nothing to clone and no full stack to rebuild, so it is ready in seconds. 2. 2 ### Route by request header A routing key travels in the request header across every service call. Keyed requests reach the change; everything else uses the shared cluster. 3. 3 ### Test against real dependencies The fork plus the shared services present as one complete, production-like stack. For stateful needs, a Resource Plugin can give a PR its own isolated database. 4. 4 ### Scale to thousands Because every preview reuses the same shared dependencies, thousands run side by side on the cluster you already operate, each torn down when its PR closes. Real data and dependencies #### Every PR environment runs against your real stack. A preview is only as useful as what sits behind it. Signadot PR environments use the databases, queues and workflow engines already running on your cluster, and isolate only what a change writes or consumes. - ### Databases Share the real database with isolated writes, or give a PR its own database or a Neon branch through a resource plugin. [Read the docs](https://www.signadot.com/docs/guides/set-up-data-isolation) - ### Message queues Kafka, RabbitMQ, SQS, Pub/Sub and NATS work through message routing, so only the PR's consumers process its messages. [Read the docs](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - ### Temporal workflows Stable and PR workers share one task queue, and exactly one worker version processes each task. [Read the docs](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - ### Cloud resources Resource plugins create managed databases, queues and buckets for a PR with Terraform, Helm or a script, and delete them with it. [Read the docs](https://www.signadot.com/docs/reference/resource-plugins/overview) End-to-end previews #### Preview from your real web and mobile apps. A preview is a real, routable environment, not just a backend URL. Point your web frontend or mobile app at it by carrying the preview's routing key, and walk the whole flow end to end against the change while every other dependency is served by the shared cluster. Clients Web app Mobile app API client routing key Preview runs your change Shared cluster everything else The preview workflow #### From pull request to preview URL, automatically. Every PR gets a preview without a ticket, a queue, or a human in the loop. Your CI applies a sandbox from a template in the repo, the Signadot GitHub App posts the preview URL as a comment, and the environment tears down the moment the PR closes. - No manual steps, the same flow for developers and agents - Preview URL on the PR, with the routing key to reach it - Auto-teardown on merge or close, with a TTL backstop [Set up PR previews in the docs](https://www.signadot.com/docs/guides/set-up-pr-sandboxes) A developer or agent opens a PR push to any service CI applies the sandbox template $ signadot sandbox apply Preview URL posted to the PR pr-142.preview · tests pass PR closes, preview torn down infra reclaimed automatically #### Thousands of previews. On the cluster you already run. Signadot virtualizes preview environments onto your existing Kubernetes cluster. Each one duplicates only what a PR needs, usually just the services it changed, and routes everything else to shared real dependencies. Previews spin up in parallel, in seconds, and scale with the PR volume your developers and coding agents create. Ephemeral environments, in parallel pr-7f3a pr-b21c pr-9e04 pr-4d8b pr-c6f1 \+ thousands more all share Shared stable dependencies on your cluster Services Message queues Databases Caches A preview for every PR Per-PR and per-task previews run side by side, with no queue, no scheduling, and no contention for a shared environment. Seconds to spin up Only what the change needs deploys. Everything else routes to shared real dependencies, so there is nothing to clone and no full stack to rebuild. Production-like fidelity Services, message queues, databases, and caches, every layer a change touches, exercised exactly as it runs in production. *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer #### A preview environment per PR, at a fraction of the cost. Bitso replaced its homegrown full-environment-per-PR system with Signadot, giving 250+ engineers and 200+ microservices a lighter, faster preview for every change. [Start for free](https://www.signadot.com/signup/) [Read the Bitso story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > Signadot for the preview environments use case fit what we were trying to do better than anything else. It was a more mature solution than the other stuff that we were looking at. And the return on the investment was obvious... just in infrastructure costs, it saves us about $2 million annually. *Image: Phil Burrows, Head of Platform Engineering at Brex* Phil BurrowsHead of Platform Engineering *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Laurel logo* 82% lower change failure rate > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer What you get #### Previews that fit how your team already ships. No hosted platform to migrate to and no SDK to adopt. Previews run on your cluster, with your tools, for both the people and the agents opening PRs. - ### Cross-PR previews with RouteGroups Compose several PRs into a single preview with one URL to test how changes across services behave together, before any of them merge. - ### Bring your own tests Point your existing Playwright, Cypress, or k6 suites at a preview with Signadot Jobs. No framework to migrate and no code changes required. - ### Real dependencies, not mocks Changed services run against the real services, databases, and queues on your cluster, so you validate against production reality instead of an approximation. - ### Runs on your cluster Previews use your existing Kubernetes cluster and ingress, and the hosted control plane receives only resource metadata, never your code or data. Reach a preview by preview URL, request header or the browser extension. - ### Built for coding agents too Each agent task or PR gets its own preview, created through the Signadot MCP server or CLI, so agents validate against real services and open PRs you can trust. - ### Automatic teardown A preview lives while its PR is open and reaps on merge or close, with a global TTL backstop so nothing is left running. Zero orphaned resources. #### Plug into the CI you already run. Guides for the major CI systems, and a CLI for the rest. Add a few lines to your pipeline and your team has a pull request environment for every PR. Routing works through the built-in DevMesh or your existing service mesh. - *Image: GitHub Actions* ### GitHub Actions A preview per PR from your workflow. The Signadot GitHub App posts it on the PR and deletes it when the PR closes. - *Image: GitLab CI* ### GitLab CI A preview per merge request from your pipeline, cleaned up by a time-to-live. - *Image: Jenkins* ### Jenkins CLI calls in your existing jobs create and delete a preview for each PR. - *Image: Bitbucket Pipelines* ### Bitbucket Pipelines A preview per pull request from your pipeline, cleaned up by a time-to-live. - ### Any CI Any pipeline that can run the Signadot CLI can create a preview with signadot sandbox apply. - ### Istio and Linkerd Routing works through the built-in DevMesh sidecars or the service mesh you already run. [Browse all CI and GitOps integrations](https://www.signadot.com/integrations/) #### Preview Environments FAQ What is a PR environment on Kubernetes? A PR environment, or pull request environment, is a preview environment created automatically for one pull request so the change can be reviewed and tested before it merges. With Signadot, each one is an ephemeral environment that duplicates only what the PR needs on the Kubernetes cluster you already run. People and coding agents preview the change through your web and mobile apps, and the same environment runs your automated tests. It spins up in seconds and is deleted when the PR closes. How does Signadot create a preview environment for each pull request? When a PR opens, your CI creates an ephemeral environment from a template stored in the repo. Signadot duplicates only the changed workload on your existing Kubernetes cluster and gives it a routing key. Requests and messages carrying that key reach the changed service, and everything else uses the shared environment with its real dependencies. Nothing else is copied, so the preview is ready almost immediately. How is this different from spinning up a full environment or namespace per PR? Cloning the whole stack into a namespace or cluster per PR is faithful but slow and expensive, and the cost grows with every service and every open PR. Signadot deploys only the diff and shares the rest of the cluster, so each preview stays lightweight, starts in seconds, and thousands run side by side on the cluster you already operate. How do preview environments fit into CI/CD and GitHub? Add a few lines to your pipeline to apply a sandbox when a PR opens. On GitHub, the Signadot GitHub App posts the preview details on the PR and deletes the environment when the PR merges or closes. Guides cover GitHub Actions, GitLab CI, Jenkins and Bitbucket Pipelines, any other CI can call the Signadot CLI, and a time-to-live on every sandbox makes sure nothing is left running. Can preview environments keep up with the PR volume coding agents create? Yes. Coding agents push far more PRs than a queue of full-stack environments can serve. Because each Signadot preview is just a diff on a shared cluster rather than a full copy, previews scale with PR volume without scaling infrastructure. Each agent task or PR gets its own preview, created through the Signadot MCP server or CLI, so developers and agents share the same fast, lightweight workflow. Do PR environments work with databases, message queues and Temporal? Yes. A PR environment shares your real databases by default, and a resource plugin can give a PR its own database or a branch of one, such as a Neon branch. Kafka, RabbitMQ, SQS, Pub/Sub and NATS work through message routing, so only the PR's consumers process its messages. Temporal workers share a task queue, and exactly one worker version processes each task. Resource plugins also create cloud resources a PR needs and delete them with it. Can I run my existing tests against a preview environment? Yes. There is no SDK to adopt and no framework to migrate to. Point your existing Playwright, Cypress, or k6 suites at a preview using Signadot Jobs, or open the preview URL directly. Tests run against the changed services and the real dependencies on your cluster, not mocks or stubs. #### A preview environment for every change, built for the scale of agentic development. Give every developer and every coding agent a lightweight preview for every PR, scalable to thousands running in parallel on the cluster you already run. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Local Development Source: https://www.signadot.com/solutions/local-development/ Summary: Connect local workstations, CDEs, or agent runtimes directly to your Kubernetes cluster via isolated sandboxes. Develop against real databases, message queues, and microservices instead of mocks. ### Local development for the agentic era Connect local machines or CDEs directly to your Kubernetes cluster. Empower developers and agents to write, run, and verify code against real cloud dependencies without leaving the IDE. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Your machine user-svc running locally, hot reload signadot local connect Your cluster api-gateway cart-svc auth-svc postgres real responses Verified in seconds E2E tests, API diff, contract Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Hot reloads for your backend Your agents can generate code in seconds. But validating it still takes 20 minutes in CI. Signadot Local gives developers and agents a fast, high-fidelity feedback loop against real infrastructure. Change code. See it live. Move on. [Read the docs](https://www.signadot.com/docs/tutorials/quickstart/local-development) Old workflow CI/CD pipeline building, 15 min elapsed 60% Signadot Local Hot reload Change validated in 5 seconds #### Standardize the inner loop Empower developers and agents with a uniform way to connect, route, and debug across local and cloud environments. 1 ##### Connect to your cluster Run signadot local connect to establish a secure tunnel between any workspace and the cluster. Access cluster services via their DNS names locally. Works from laptops, CDEs, and agent runtimes. Your workspace laptop, CDE or agent runtime signadot local connect Secure tunnel through Signadot Your cluster, by DNS name route.hotrod driver.hotrod redis.hotrod 2 ##### Define your sandbox Define a simple YAML config that maps traffic from a remote Kubernetes deployment port to your local service port. Agents can interact with our MCP server to manage these definitions dynamically. route-local-sandbox.yaml ``` name: "route-local-sandbox" spec: cluster: "demo" ttl: duration: 1d description: "route local sandbox" labels: feature: "featurex" local: - name: "local--featurex" from: kind: Deployment namespace: "hotrod-istio" name: "route" mappings: - port: "8083" toLocal: "localhost:8083" ``` 3 ##### Start your service locally Run your microservice locally using your preferred setup: Docker, IDE, or native execution. Signadot automatically routes cluster traffic to your local instance for you or your agent to debug. Cluster traffic requests for the sandbox routed to route Docker, IDE or native, localhost:8083 Breakpoint hit debug with a live request 4 ##### Hot reload & validate Change code locally and see it live in seconds. Give your coding agents autonomous validation loops against the sandbox to verify end-to-end flows. Find and fix integration issues in seconds, not 20-minute CI cycles. Save a change hot reload, no image build live in seconds pr-202-frontend sandbox, routing key vfOju8vw5Owpu agent validates End-to-end flow passes no 20-minute CI cycle #### Talk to an engineer Hop on a call with one of our engineers. No slides, no sales pitch. Just you, us, and a whiteboard session. [Book a technical consult](https://www.signadot.com/schedule-a-call/) #### Inspect live traffic. Override any API. Enhanced debugging with powerful observability and granular control over cluster traffic directly from your laptop or agent workspace. ##### Record & inspect HTTP/gRPC traffic Stop guessing based on stale API docs. Record and inspect live request/response payloads between any cluster service. Give your agents the context to understand complex interactions and debug integration issues against systems that nobody wrote documentation for. live traffic · request ``` { "user_id": "usr_7x2k9m", "cart_id": "cart_4j8s1p", "items": [ { "product_id": "prod_92kl3", "quantity": 2, "price": 49.99 } ], "total": 99.98, "payment_method": "credit_card" } ``` live traffic · response ``` { "transaction_id": "txn_5m2p8k", "status": "success", "amount_charged": 99.98, "timestamp": "2025-01-15T14:23:11Z" } ``` ##### Surgically override APIs Make your code resilient. How does it handle a 5-second latency spike or a 500 error from a dependency? Surgically override any API endpoint to return exactly what you want. Test edge cases, simulate failures, and validate resilience locally in seconds. terminal - zsh ``` $ signadot local override \ --sandbox test-payment \ --workload payment-service \ --workload-port 8081 \ --with localhost:8081 ✓ Override middleware created ✓ Routing payment-service traffic to localhost:8081 ✓ Fallback to cluster baseline enabled Override active. Press Ctrl+C to stop. ``` #### Turn agents into validated shippers Code generation is the easy part. The bottleneck is validation. Give your agents the infrastructure to verify their work end-to-end, so they stop being expensive autocomplete and start actually shipping. Coding agent closed loop Its own environment run, validate, debug ##### Close the loop Give your coding agents the ability to provision environments where they can run, validate, and debug their code end-to-end in a closed loop. One shared cluster agent-1 agent-2 agent-3 agent-4 ##### Safe concurrency Let hundreds of agents explore and iterate autonomously in parallel in safely isolated environments with zero staging-breaking contention. Instant feedback in your IDE 20-minute CI pipeline ##### Maintain flow state Iterate on AI-generated code as fast as you can prompt. Get instant feedback in your IDE instead of context-switching to wait for a 20-minute CI pipeline. #### The Signadot MCP Server Forget learning new YAML. Equip your agents with native tools to manage sandboxes, route traffic, and verify code using natural language. No kubectl. No custom scripts. No ticket to the platform team. The Signadot MCP (Model Context Protocol) Server plugs directly into AI coding tools like Cursor, Claude and Windsurf. [Read the docs](https://www.signadot.com/docs/tutorials/quickstart/local-development) Infrastructure as prompts Agents can provision high-fidelity environments, update image tags, and inject environment variables using natural language commands. Autonomous debugging Have agents inspect live traffic, override API responses, or reproduce production issues to debug autonomously. Advanced routing Enable your agents to create route groups to combine multiple sandboxes, enabling complex integration testing across services. #### Do it all in local Debug complex issues Reproduce staging or production issues on your local machine. No more "it works on my machine". Get full access to logs, debuggers, and breakpoints against live data to fix human or agent-generated code. Validate schema migrations safely Run schema changes locally while connecting to real downstream services to catch integration issues before deployment. Unlock true collaboration Frontend developers, other teams, or autonomous agents can validate against your local backend changes instantly. No deployments, no coordination, and no "is staging free?" messages. #### Standardize once. Rollout everywhere. Signadot Local works with your existing Kubernetes clusters and integrates seamlessly with your IDE, debugger, and local development tools for fast integration testing. [Read the docs](https://www.signadot.com/docs/tutorials/quickstart/local-development) Signadot Local works with Kubernetes clusters the ones you run IDE and debugger Local dev tools Comparing your options first? Read our complete guide to [local development on Kubernetes](https://www.signadot.com/guide-to-local-development-kubernetes/). #### Unlock developer velocity. Get started in 5 minutes. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Microservices Testing Solution Source: https://www.signadot.com/solutions/microservices-testing/ Summary: An ephemeral environment for every pull request, developer and coding agent on the Kubernetes cluster you already run, so your own integration, end-to-end, performance and chaos tests run against real services. Covers REST and gRPC calls, Kafka, SQS, RabbitMQ and Pub/Sub messages, Temporal workflows, databases and resource plugins. Microservices Testing ### Microservices testing with real dependencies on every change. Signadot gives every pull request, developer and coding agent an ephemeral environment on the Kubernetes cluster you already run. Your integration, end-to-end, performance and chaos tests run against real services, message queues, databases and workflows before code merges. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - No credit card required - Runs on your existing cluster Coding agents PR #423 PR #424 PR #425 an environment each Kubernetes cluster env · pr-423 integration pass env · pr-424 e2e pass env · pr-425 performance pass shared services real dependencies Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Why microservices testing breaks down at scale The failures that reach production sit between services, and every common workaround either hides them or queues behind them. The [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) covers every layer, from unit tests to production. unit.test 12 passed, mocked merge production real service broken 500 ##### Mocks drift from the services they copy A mock records what another team's API did on the day someone wrote it. The real service keeps changing, the mock does not, and the suite stays green until the two meet after merge. dev PR CI collide Staging queue: 12 waiting busy ##### One shared staging cluster turns into a queue When every branch deploys to the same environment, a red test could belong to anyone. Engineers wait for a turn, then spend it working out whose change broke the run. cluster · dev-1 cluster · dev-2 × every developer Cloud cost $$$$ per dev ##### Copying the whole stack costs too much A namespace or cluster per pull request removes the queue, then multiplies the bill by the number of open branches and takes as long to start as the whole system. agent agent agent untested PRs Review queue keeps growing +12 ##### Coding agents multiply the changes Agents open pull requests faster than people can review them. If none of those changes met a real service first, the reviewer ends up doing the integration testing by reading diffs. #### Agents can only validate what they can run. Unit tests, types and lint need nothing but the code. Integration, end-to-end, performance, chaos and security tests need live services to call. Signadot supplies the services, and your tests do the checking. Without real services 3 of 9 test types run agents fall back to local checks Static analysis · runs unit tests types lint Runtime · blocked, nothing to call integration e2e contract performance chaos security In an ephemeral environment 9 of 9 test types run real dependencies, every change Static analysis · runs unit tests types lint Runtime · runs against real services integration e2e contract performance chaos security #### How Signadot fits your microservices testing Signadot is the environment layer. It gives each change a place that behaves like production, and leaves the tests, the runners and the reporting you already have where they are. 1 ##### An ephemeral environment for every change Signadot duplicates only what a change touched or needs to isolate, such as a service, a database or a queue, and links it to everything else through request routing, message routing and configuration changes. The changed service can run in the cluster or on a laptop, so local development, pull requests and agents test against the same real dependencies. Change service + diff in the cluster or local routing and config Shared dependencies, current versions services queue database workflows 2 ##### Your tests, run from wherever suits them Point the suites you already have at the environment's URL or routing key. Run them from a laptop while the change is still in the editor, from a CI pipeline in GitHub Actions, GitLab or Jenkins, or as Signadot Jobs, which run a suite in pods inside the cluster next to the services it tests. Where tests run local CI Signadot Job URL or routing key Ephemeral environment Playwright · Cypress · k6 · Postman 3 ##### Coding agents test before they open a pull request Agents create and delete environments through the Signadot CLI, the MCP server or agent skills, run the same suites against real services inside their own loop, and fix what fails. Every environment carries a time-to-live, so abandoned runs clean themselves up. agent session ``` > test the refund change against real services → creating environment for orders@pr-425 ✓ environment ready, routing key attached ✓ integration suite 24/24 ⠋ e2e: refund journey, waiting on inventory event ``` #### Synchronous calls, async messages and durable workflows. Microservices talk over REST and gRPC, through brokers and through workflow engines. A change's environment has to carry each kind of flow, or the tests stop at the first queue. - ### REST and gRPC calls A routing key in the request headers sends calls to the changed service and lets every other call fall through to the shared version. Istio, the Gateway API or the DevMesh sidecar does the routing, and OpenTelemetry carries the key from hop to hop. [Context propagation](https://www.signadot.com/docs/guides/set-up-context-propagation) - ### Kafka, SQS, RabbitMQ, Pub/Sub Producers carry the same key in message headers. Each consumer version takes only its own messages, so a change can publish and consume on the shared broker without touching anyone else's events. [Message isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - ### Temporal workflows A changed worker polls the shared task queue next to the stable one and picks up only the workflows started with its key. The steps it runs call the real services around it. [Temporal tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - ### Databases and other resources Share the database with each test's rows kept apart, or give a schema change its own temporary database or Neon branch. Resource plugins create anything else a change needs, in the cluster or outside it. [Resource plugins](https://www.signadot.com/docs/reference/resource-plugins/overview) *Image: Earnest logo* Test cycles cut from 60 to 10 minutes > Signadot is a key part of our solution for the most critical parts of our testing pyramid. For integration and end-to-end testing we need to have a sandbox up and running because they need a live service to talk to. EE Early EhlingerDev Experience Lead #### Integration and end-to-end tests need live services. Earnest runs the critical layers of its testing pyramid against real services in an environment per change, and cut test cycles from 60 minutes to 10. [Start for free](https://www.signadot.com/signup/) [Read the Earnest story](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) Customer results #### Teams testing microservices on Signadot *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering *Image: Laurel logo* 82% lower change failure rate > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer #### Microservices testing, built for coding agents. Claude Code, Cursor, Codex and Copilot write microservices changes faster than anyone can review them. Signadot gives each agent a real environment on your own cluster, and Plans let agents write the testing workflows they run in it. agent task #7 iterate env · task-7 integration ok e2e ok performance rerun ##### A real environment inside the agent's loop Each agent task gets its own environment in seconds, runs the suites against real services, and reads the failures in the same session it wrote the code. agent writes the plan plan: refund-flow create environment run integration suite replay traffic ##### Plans: testing workflows agents write Agents author Plans in plain language from an action catalog your platform team governs, then run them by name through the MCP server or CLI until every check passes. [How Plans work](https://www.signadot.com/product/plans/). #423 · opened by agent feat: add idempotency to payment all checks passed integration 24/24 · e2e 18/18 · perf ok review ##### Pull requests that arrive tested The change a reviewer opens has already run against real services, with results on the pull request, so review time goes to judgment instead of triage. [See the docs](https://www.signadot.com/docs/integrations/mcp) #### Microservices Testing FAQ What should a microservices testing solution provide? Somewhere real for every change to run. Teams already own their test frameworks, so the gap is an environment per pull request, developer and agent task that holds the current versions of the other services, its own state where a change needs it, and nothing another team is testing at the same moment. Signadot provides that environment layer on the Kubernetes cluster you already operate. Is Signadot a test automation tool? No. Signadot gives each change an ephemeral environment, and your tests decide whether the change works. Playwright, Cypress, Postman, k6, pytest or any other suite runs against it from CI, from an orchestrator such as Testkube, or from a pod in the cluster. For teams with nowhere else to run a suite, Signadot Jobs can start one in the cluster. Which kinds of tests can run in a Signadot ephemeral environment? Any test that needs live services to call. Teams run integration and end-to-end suites on every pull request, performance checks that compare the changed service with the shared version, fault-injection tests for timeouts and retries, and authorization tests between services. Unit tests, linting and contract checks keep running in CI as they do today, since they need no environment. How are Kafka, SQS and RabbitMQ handled when several changes test at once? The broker stays shared. A producer in the environment stamps each message with the change's routing key, and each consumer version takes only the messages addressed to it, so one change never eats another change's events. Signadot documents this pattern for Kafka, RabbitMQ, NATS, SQS with SNS and Google Pub/Sub. A change that needs a topic or queue to itself can get one from a resource plugin. Can Signadot test changes to Temporal workflows? Yes. The changed worker joins the same Temporal task queue as the stable worker, and workflows started with the change's routing key go to the new code while all other work stays on the current workers. Its activities then call real services, databases and queues. The Temporal tutorial in the Signadot docs shows the setup for Python, TypeScript, Go and Java workers. How is this different from a namespace per pull request? Cloning a namespace for each branch duplicates all of the services, so the bill climbs with every open pull request, startup waits on the slowest service, and the copy falls behind the moment it exists. A Signadot environment deploys only the services a change touched, plus any database or queue it must own, and links them to everything else already running. Hundreds of them fit on one cluster. How do coding agents test microservices changes with Signadot? Claude Code, Cursor, Codex, Copilot and similar agents call the Signadot CLI or MCP server to spin up an environment scoped to the task, point your integration and end-to-end suites at the real services, read what failed and try again. The environment expires on a timer, and the pull request reaches a reviewer with test results already attached. What does it take to start? Install the Signadot Operator in a Kubernetes cluster that already runs your services, then create environments from the CLI, from CI or through the MCP server. To try it before touching your own cluster, use the free Playground cluster, with no credit card required. Your code and data stay in your cluster. Building a strategy first? Read the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/), from test types to synchronous, asynchronous and workflow flows. #### Give every microservices change a real place to run. One cluster, an ephemeral environment per pull request and agent task, and your own tests against real dependencies before review. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Kubernetes Test Environments Source: https://www.signadot.com/solutions/kubernetes-test-environments/ Summary: Give every developer, PR, and coding agent a real Kubernetes test environment on the cluster you already run, with no contention. Your existing tests run against it from CI or inside the cluster, with real databases, message queues and Temporal workflows behind it. Kubernetes Test Environments ### Kubernetes test environments for every change. Lightweight ephemeral environments on the pre-production cluster you already run. Every developer, every PR, and every coding agent gets a real Kubernetes test environment with no contention. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Runs on your existing cluster - Scales to hundreds of envs Kubernetes cluster env-1 pass env-2 run env-3 pass env-4 run env-5 pass env-6 run in parallel Tests running in parallel integration · end-to-end · regression Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Testing on Kubernetes requires a real runtime. The bugs that matter live between services. Catching them requires a realistic runtime with the real services, queues, and databases your app depends on. The [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/) covers every layer of tests and where each one runs. orders drift, races, retries payments fails ##### The bugs live between services In a microservices system, the failures that reach production tend to live at the seams between services: contract drift, timing races, partial outages, retry storms. Unit tests do not surface them and shared staging rarely reproduces them. See the [complete guide to staging environments](https://www.signadot.com/staging-environments/) for why. dev PR agent collide Staging queue: 12 waiting busy ##### Shared staging is a queue, not an environment When every developer, every PR, and every agent task lands in one shared cluster, tests fail because of collisions, not real bugs. Triaging environments eats the day. Test surface services databases queues external APIs ##### The test surface is wide Real microservices runtimes mix dozens of services with databases, queues, and external APIs. Covering that surface in tests means running against a realistic Kubernetes stack, not a mocked one. #### Full-cluster duplication vs Signadot ephemeral environments. The traditional path to a real Kubernetes test environment is duplicating the cluster per developer or per PR. Signadot reaches the same fidelity by duplicating only what changed and sharing the rest. Feature Full-cluster duplication Signadot ephemeral environments Full stack dependencies Isolated per developer, PR, or agent task Spins up in seconds Scales to thousands of environments Coding-agent-ready *Image: Brex logo* Previews in under 5 minutes, down from 30 to 60 > A typical deploy in our Signadot based tooling, end-to-end, is less than five minutes versus the 30-60 minute deploy times we would see with preview environments. Local sandboxes allow us to take that down to almost instantaneous testing. JS John SalemSenior Software Engineer #### Every change tested against real Kubernetes dependencies. Every developer and every coding agent gets a real Kubernetes test environment to run integration, end-to-end, and regression tests before code lands. [Start for free](https://www.signadot.com/signup/) [Read the Brex story](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) #### How Signadot solves Kubernetes testing Four product capabilities that turn the cluster you already run into a realistic Kubernetes test environment for every developer and every coding agent. 1 ##### Ephemeral environments Each change gets a lightweight ephemeral environment: only the services that changed are deployed, and everything else is served by the shared, stable versions in the cluster. The changed service can run in the cluster or on a developer's laptop or an agent's workspace, so the inner loop and CI test against the same real dependencies. Change service + diff in-cluster or local everything else Shared dependencies, real service service queue db 2 ##### Native service mesh routing Signadot routes each environment's requests through the service mesh your cluster already runs. Istio is supported natively, along with the Kubernetes Gateway API. A lightweight sidecar handles Linkerd or no mesh at all, so the same routing model works whether the service runs in the cluster or on a developer's laptop. request Mesh routing Istio Linkerd Gateway API HTTP headers Environment changed service Shared stable version 3 ##### Signadot Jobs Bring your own test framework. Each Job runs in pre-warmed pods in your cluster and exercises the APIs and frontends routed through an ephemeral environment. Playwright, Cypress, Postman, k6, or your own runner. Hundreds of pre-merge runs in parallel with no contention, with logs and artifacts kept for every run. Playwright · Cypress · Postman your framework Signadot Job runs in a pre-warmed pod hits the environment's routes Ephemeral environment 4 ##### Plans, Skills and the MCP server Plans chain test steps into reusable workflows: deploy an environment, run tests, capture and replay traffic, gate a merge. Coding agents drive them through the Signadot MCP server and agent skills such as /signadot-validate, so they can test their own changes against real services with more autonomy. agent session ``` > /signadot-validate test the new payment flow against staging → invoking Plan: payment-validation ✓ deploy environment with service +diff ✓ run integration + e2e suite ⠋ replay captured traffic against the environment ``` #### Test against real databases, queues and workflows. Mocks hide the failures integration tests exist to catch. In a Signadot environment, tests reach your real data stores, brokers and workflow engines, kept apart per change without copying them. - ### Databases Tests share the existing database with their data kept apart, or get a temporary database, schema or Neon branch when a change alters the schema. [Data isolation](https://www.signadot.com/docs/guides/set-up-data-isolation) - ### Message queues Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub stay shared. Messages carry the routing key and each consumer processes only its own. [Message queue isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - ### Batch jobs and workflows Changed and stable Temporal workers poll one task queue, and the routing key decides which worker runs each workflow. Batch jobs read their share of the data. [Temporal tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - ### Resource plugins Create and delete anything an environment needs, in the cluster or outside it, such as a managed database, a topic, a storage bucket or a virtual machine, with Helm, Terraform or a script. [Resource plugins](https://www.signadot.com/docs/reference/resource-plugins/overview) Coming soon: versions of services that run outside Kubernetes, on AWS ECS or Lambda, Google Cloud Run or virtual machines, join an environment like any Kubernetes workload. [How the platform works](https://www.signadot.com/product/platform/). #### Ship reliable code on Kubernetes at agent scale. Test every change against a real microservices runtime. Catch bugs early, ship faster, and unblock coding agents to test their own code in Kubernetes. - Agent scalethousands of envs in parallel Every developer and every coding agent gets their own ephemeral environment on the cluster you already run. No coordination, no waiting for a slot. - Secondsto test every change Spin-up is fast enough that agents iterate against real services inside their own loop. The test environment stops being the bottleneck. - Reliablecode shipped faster Every PR arrives at review having already passed integration, end-to-end, and regression tests against real Kubernetes dependencies. Reviewers spend time on judgment, not triage. [Start for free](https://www.signadot.com/signup/) [View pricing](https://www.signadot.com/pricing/) #### Kubernetes Test Environments FAQ What is a Kubernetes test environment? A Kubernetes test environment is a cluster, or an isolated part of one, where integration, end-to-end and regression tests run against the same kinds of services and dependencies as production. With Signadot, each developer, pull request and coding agent task runs in its own ephemeral environment on the pre-production cluster you already operate, holding only what the change touches and sharing everything else. How is this different from kind, minikube, or vCluster? kind and minikube start a lightweight cluster on one machine, which works for operators and single-service checks but cannot hold the dozens of services, queues and databases a real application calls. vCluster gives each environment its own control plane inside a host cluster and still copies the workloads under test. Signadot deploys only the services that changed into the cluster you already run, so each environment is a few extra pods rather than a clone. Can I run integration and end-to-end tests in Kubernetes without contention? Yes. Each pull request and coding agent task gets its own ephemeral environment on one shared cluster. Its requests and messages carry a routing key, so they reach the changed services while everything else is served by the shared, stable versions. Two changes to the same service can be tested at once, with no queue and no second cluster to maintain. How does Signadot handle databases, message queues and workflows in tests? Tests use the existing database with each run's records kept separate, or receive a short-lived database, schema or Neon branch from a resource plugin when a change alters the schema. On a shared broker, messages carry the routing key and each consumer handles only its own, a pattern documented for Google Pub/Sub, Kafka, NATS, RabbitMQ, and SQS with SNS. Temporal workers share one task queue the same way. Does this scale to hundreds of concurrent test environments? Yes. Because only the changed services are duplicated and everything else is shared, hundreds of environments fit in one pre-production cluster. Brex runs pre-merge environments for hundreds of engineers on Signadot and cut about $2 million a year from its infrastructure bill compared with duplicating environments. Can I use my existing tests? Yes. Signadot provides the environment, and the tests stay yours. Playwright, Cypress, Postman, k6 or any other suite runs against an ephemeral environment from CI, a test orchestrator such as Testkube, or a pod in the cluster, using the environment's URL or routing key. Signadot Jobs can also start a suite in pre-warmed pods inside the cluster if you have nowhere else to run it. How do coding agents use Kubernetes test environments? Claude Code, Cursor, Codex, Copilot and other coding agents call the Signadot CLI, MCP server or agent skills to create an ephemeral environment for their task, exercise the real services in your cluster with integration and end-to-end suites, and fix failures in their own loop. The pull request a reviewer opens has already run against real dependencies. Does Signadot work with my CI? Yes. Any CI that can run a shell works, including GitHub Actions, GitLab CI, Bitbucket Pipelines and Jenkins. A typical pipeline builds the changed services, creates an environment with the Signadot CLI, runs the tests against it and deletes it afterward. Does my code or data leave my cluster? No. The Signadot Operator runs in your cluster next to your workloads and data. The hosted control plane gets metadata about Signadot resources, such as workload and namespace names, image references and status, and not source code, the contents of images, or application data. Opt-in features, such as traffic capture for Smart Tests and Job logs, store more, and that data is encrypted at rest. Signadot holds a SOC 2 Type II report. New to the model? Read our complete guide to [ephemeral testing environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#ephemeral-testing-environments). #### Kubernetes test environments built for coding agents. Lightweight ephemeral environments mean every developer, every PR, and every coding agent gets a real Kubernetes test environment, sharing the cluster you already run. No queue. No contention. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Coding Agent Environments Source: https://www.signadot.com/solutions/agentic-development/ Summary: An ephemeral environment for every coding agent task on your Kubernetes cluster. Claude Code, Cursor, Codex or any MCP-compatible agent creates one through the Signadot MCP server or CLI, runs its change against real services, databases, message queues and Temporal workflows, debugs what fails and opens a tested pull request. Only what the task changed is duplicated, so thousands of agents test in parallel on one cluster. Coding Agent Environments ### Give every coding agent an environment to test its own code. Coding agents write code in minutes, but with nothing real to run it against, their bugs wait for staging. A coding agent environment is an ephemeral environment where the agent runs its change against the real services, databases and message queues it depends on, then fixes what breaks before it opens a pull request. Signadot creates one for every task on the Kubernetes cluster you already run. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) - Works with Claude Code, Cursor and Codex through MCP - Real databases, queues and workflows in every environment Coding agent Claude Code, Cursor, Codex closed loop Until the tests pass Write code the change Environment real services Test integration, e2e Debug logs, requests passes Tested PR ready for review Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Agents write the code. Staging finds the bugs, days later. Without an environment of its own, a coding agent can only test the code it generates against mocks. Integration bugs surface after the pull request reaches staging, and every fix means rework: back to the agent, back through review, back to staging. unit.test mocked integration.test no services e2e.test no services ##### Unit tests only On a laptop or in a sealed cloud sandbox, the agent has no real services to call. It checks functions in isolation and never sees how its change behaves across the system. agent dev queued Staging one cluster for everyone conflict ##### One staging for every agent A shared staging cluster can't host every developer and every agent at once. Agents wait in line, or test on top of each other's changes and break each other's results. code PR, review, merge bug found days later staging ##### Late bugs mean rework A bug found in staging sends the change back to the start, long after the agent moved on. Reviewers become the test layer, and the time saved writing code is spent fixing it. #### Ephemeral environments for coding agents The best ephemeral environments for coding agents give every task its own copy of what the agent changed, connect it to real versions of everything else, and let the agent create and delete it from its own tools in seconds. Signadot does that on the cluster you already run, so thousands of agents test in parallel without copying the stack. ##### Where a coding agent can test its change, compared by approach Approach Real services and data Agents in parallel Ready in Finds integration bugs before the PR Signadot ephemeral environment Yes: the shared services and data on your cluster, kept apart per task Thousands on one cluster Seconds Yes, in the agent's own task Full copy of the stack per task Yes, a copy that drifts from the original As many as the budget allows Minutes Yes, at the cost of a stack per agent Shared staging cluster Yes One change at a time After a deploy and a queue After the merge Code-execution sandbox (container or microVM) Only what the agent installs in it Many Seconds Rarely: no services to call Developer's laptop or devcontainer Mocks, or the few services that fit One per machine Seconds Rarely #### Your coding agent closes the loop. The agent writes, runs, tests and debugs its change against real dependencies until the tests pass, without waiting on a person. It works on its own for longer, and you get a pull request that has already been tested end to end. Create envWrite codeBuild & runRun testsDebug ##### 1\. Create an ephemeral environment The agent asks Signadot for an ephemeral environment through the MCP server or the CLI. In seconds, the service it is about to change is wired to the real services, databases and queues in your cluster. ##### 2\. Write the change The agent edits the service, the same way it would anywhere else. ##### 3\. Build and run The agent runs the service on its own machine. Requests that carry the environment's routing key reach that process, and its own calls go out to real services in the cluster. ##### 4\. Run integration and end-to-end tests Your tests run against the environment and hit real services, not mocks. The agent can also run a Signadot plan, a saved test workflow it picks to match the diff. ##### 5\. Debug and repeat When a test fails, the agent reads the logs, inspects the request that failed, fixes the code and runs the tests again. Once they pass, it opens a pull request with the results. ~/payments-service ``` $ signadot local connect # cluster service names now resolve on this machine $ signadot sandbox apply -f pr-431-payments.yaml \ --set cluster=staging # local: Deployment payments-svc → localhost:8080 ✓ sandbox pr-431-payments ready in 2.1s payments-svc your machine api-gateway baseline auth-svc baseline ``` handler.go ``` 14 func HandlePayment(w http.ResponseWriter, r *http.Request) { 15 key := r.Header.Get("Idempotency-Key") 16 + if cached, ok := store.Get(key); ok { 17 + json.NewEncoder(w).Encode(cached) 18 + return 19 + } 20 + if store.IsExpired(key) { 21 + http.Error(w, "key expired", 409) 22 + return 23 + } 24 result := processPayment(r) 25 store.Set(key, result, 24*time.Hour) 26 } ``` ~/payments-service ``` $ eval $(signadot sandbox get-env pr-431-payments) # payments-svc's cluster environment, exported locally $ go run ./cmd/payments [info] listening on :8080 [info] connected → postgres.db.svc:5432 [info] connected → redis.cache.svc:6379 [info] connected → kafka.bus.svc:9092 ● ready: requests with the sandbox's routing key reach this process ``` ~/payments-service ``` $ signadot job submit -f e2e-job.yaml \ --set sandbox=pr-431-payments --attach # playwright test, routed to the sandbox's payments-svc Running 3 tests using 3 workers ✓ checkout.spec.ts › completes checkout (2.1s) ✓ auth.spec.ts › signs in (1.4s) ✘ payment.spec.ts › rejects an expired key (0.7s) POST /payments (expired key) returned 500 Expected: 409 Conflict 1 failed, 2 passed (4.2s) ``` debug · pr-431-payments ``` ◆ Inspecting the failing request from payment.spec.ts... → tail -f logs/payments-svc | grep idem 10:42:01 store.Get(key) → nil, ttl=expired 10:42:01 processPayment() called, missing IsExpired() guard → signadot traffic record --sandbox pr-431-payments req POST /payments Idempotency-Key: k_842 (expired) res 500 internal server error ◆ Root cause: expiry check runs after processing. Reordering checks and looping back. ``` How it works #### Only what changed runs in each environment. Signadot doesn't copy the stack for each task. It deploys only the services the agent changed, or the components a task needs to itself, and wires them into the rest of your cluster: request routing for synchronous calls, message routing for queues and batch work, and configuration for services that should point at the new version. To the agent, it looks like an entire environment of its own. - One cluster, thousands of environments. Agents test in parallel on the cluster you already run, with no queue for staging. - Each agent sees only its own change. Two agents editing different services never pick up each other's work. - Ready in seconds. No rebuild of the whole stack and no new cluster for each task. - Gone when the task ends. Environments expire on a timer, so idle agents leave nothing running. agent A task-7 · edits payment agent B task-8 · edits search task traffic Your cluster payment +diff task-7 env search +diff task-8 env Shared services and data auth orders catalog notify Postgres Kafka *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer #### Bitso's coding agents test every change in its own environment. With pull requests per developer up 60% as coding agents took on more of the work, Bitso moved 250+ engineers and their agents to ephemeral environments on one cluster and cut its change failure rate by 83%. [Start for free](https://www.signadot.com/signup/) [Read the Bitso story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) #### Same prompt. Very different outcomes. Two agents get the same task. One checks its work with unit tests and a linter. The other tests against live services in an ephemeral environment and catches three broken consumers before anyone reviews the code. Without Signadot Unit tests and a linter claude - ~/location-service ❯ Make the location service's response return the id field as a JSON string called locID instead of a number, and rename name to id and ensure unit tests pass. ◆ I'll update the location service. → Editing services/location/service.go... → Renaming \`name\` → \`id\` → Encoding \`id\` as JSON string \`locID\` → Running unit tests... TESTS go test ./services/location ✓ TestGetLocation ✓ TestLocationJSON ✓ TestLocationRename ✓ TestLocationErrors ✓ 4 tests passed → Running linter... ✓ golangci-lint: no issues ◆ Done. Tests pass. Pushing the change. Ships broken consumers Unit tests pass, but three downstream services break on the schema change. The agent never sees it. Staging does, days later. With Signadot Tested against live services in an ephemeral environment claude - ~/location-service (sandbox) ❯ Make the location service's response return the id field as a JSON string called locID instead of a number, and rename name to id and ensure unit tests pass. Validate with signadot-validate on the cluster. ◆ I'll update the service and validate against live consumers. → Editing services/location/service.go... → Renaming \`name\` → \`id\`, encoding as \`locID\` string ✓ Unit tests passing (4/4) → Creating sandbox pr-289-location... ✓ Sandbox ready (2.1s) SKILL signadot-validate → reading diff: services/location → matched plan: location-flow via selectionHint PLAN location-flow ✓ GET /locations/1 → 200 ✗ frontend: renders "Location: Undefined" ✗ dispatch-svc: json parse error on \`id\` ✗ 2 consumer failures ◆ Frontend reads old \`name\`. Dispatch expects numeric \`id\`. Fixing consumer contracts... → Editing frontend/src/components/Location.tsx → Editing services/dispatch/handler.go → Re-running signadot-validate... PLAN location-flow ✓ GET /locations/1 → 200 ✓ frontend: renders "Location: LOC-42" ✓ dispatch-svc → 200 ✗ tracing-svc: missing legacy id field ✗ 1 consumer failure ◆ tracing-svc still reads numeric \`id\`. Adding backward-compat field alongside \`locID\`... → Editing services/location/service.go → Re-running signadot-validate... PLAN location-flow ✓ GET /locations/1 → 200 ✓ frontend rendering → OK ✓ dispatch-svc → 200 ✓ tracing-svc → 200 ✓ analytics-svc → 200 ✓ All 5 consumers passing ✓ PR #289 opened, validated against live services Ships tested code Caught three consumer breakages and kept fixing until every downstream service passed. The pull request is ready for review. #### An agent's tests are only as good as the dependencies behind them. A test that runs against mocks passes whatever the code does. Every Signadot environment reaches real databases, message queues, and batch and workflow systems, with each task's data and messages kept apart, so an agent's tests catch the failures that would otherwise surface in staging. - ### Databases Share the existing database with each task's rows kept apart, or give a schema change its own temporary database or Neon branch, created with the environment and deleted with it. [Data isolation](https://www.signadot.com/docs/guides/set-up-data-isolation) - ### Message queues Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub stay shared. Messages carry the task's routing key, so the agent's consumer reads its own messages and nobody else's. [Message queue isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - ### Workflows and batch jobs A changed Temporal worker polls the shared task queue next to the stable one, and the workflows the agent starts run on its new code. Async and batch work follows the same routing key. [Temporal tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - ### Resource plugins Anything else the task needs, created when the environment starts and deleted when it ends: a managed database, a topic, a storage bucket, even a virtual machine outside Kubernetes. Written with Helm, Terraform or a script. [Resource plugins](https://www.signadot.com/docs/reference/resource-plugins/overview) Native MCP server #### Connect your coding agent through MCP. The Signadot MCP server gives Claude Code, Cursor, Codex and any other MCP client the operations it needs to create and use ephemeral environments, in plain language. The same operations are in the Signadot CLI, and agent skills such as signadot-validate tell the agent when to use them: create an environment, run the right tests, read the failure, try again. Environments Create, update and delete ephemeral environments, add the services a task changes, map them to local processes and get preview URLs. Discovery List clusters, find workloads and service endpoints, and browse the resource plugins an environment can use. Routing Group environments into route groups, so a change that spans several services or tasks is tested together. [MCP server docs](https://www.signadot.com/docs/integrations/mcp)[Agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills) Coding agent Claude Code, Cursor, Codex MCP Signadot MCP server or the CLI, with agent skills Tools Environments create · update · delete · preview URLs Discovery clusters · workloads · endpoints Routing route groups across environments Your Kubernetes cluster #### Whichever agent you pick, Signadot is the environment. Every major coding agent reaches the same environments through MCP or the CLI. No per-agent integration and no lock-in. - *Image: Claude Code logo* - *Image: Cursor logo* - *Image: Codex logo* - *Image: Windsurf logo* - *Image: Zed logo* Bring your own agent In-house agents connect through the same MCP tools or CLI, and background agents running in cloud sandboxes connect with an API key, as the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial) shows. #### Coding Agent Environments FAQ What is a coding agent environment? It's where an AI coding agent runs and tests the code it writes: the service it changed, the real services and data stores that service calls, and the routing that keeps the agent's test traffic apart from everyone else's. With Signadot, each agent task gets one as an ephemeral environment on your Kubernetes cluster, ready in seconds and deleted when the task is done. What are the best ephemeral environments for coding agents? Look for four things. The agent creates and deletes environments from its own tools (an MCP server, a CLI or skills), each environment reaches real services and data instead of mocks, many run in parallel on shared infrastructure, and each one starts in seconds. Signadot does all four by duplicating only what a task changed on the cluster you already run. Tools that copy the whole stack per environment also give real services, but every agent then costs a full stack and minutes of startup. How is a coding agent environment different from a code-execution sandbox? A code-execution sandbox is a container or microVM where the agent itself runs, isolated from your systems for safety. A coding agent environment is what the agent tests against: your real services, databases and queues, with its change wired in. The two work together. An agent can run in a cloud sandbox and connect to a Signadot environment to test, as the Modal example on GitHub shows. Signadot's product calls its environments Sandboxes, but they solve the second problem, not the first. How does a coding agent test against real databases and message queues? The environment reaches the databases and brokers your services already use, with each task's data kept apart. Messages on Kafka, RabbitMQ, NATS, SQS and Google Pub/Sub carry the task's routing key, so the agent's consumer reads only its own. A schema change can get its own temporary database or Neon branch, and resource plugins create anything else a task needs, inside or outside Kubernetes. How do I connect Claude Code, Cursor or Codex? Install the Signadot CLI, which includes the MCP server, and add the server to your agent's MCP settings, or let the agent run the CLI directly. Then add the Signadot agent skills, which tell the agent when to create an environment, which tests to run and how to read a failure. Any agent that speaks MCP or can run a shell command works, in-house agents included. Is this an alternative to staging for AI-generated code? It replaces the queue for staging, not the cluster. Ephemeral environments run on the staging or pre-production cluster you already have and share its services and data. Each agent tests its AI-generated code in an environment of its own before the pull request, so a bug is caught in the agent's task instead of after the merge, and staging stays stable for everyone else. How many coding agents can test at the same time? Thousands, on one cluster. Each environment deploys only the services its task changed, plus any database or queue the task needs to itself, and shares everything else. An environment costs a few pods, not a copy of the stack, and it expires on a timer so idle agents leave nothing behind. Can a platform team control what agents do? Yes. Agents act through service accounts with admin, member or viewer roles, and every change lands in the audit log. Signadot runs in your cluster, and by default its control plane receives only resource metadata such as workload names and status, never your source code or application data. New to the model? Read [what coding agents change about ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#what-ai-coding-agents-change-about-ephemeral-environments), our guide to [validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/), and, for running agents across the whole delivery pipeline, the complete guide to [software factories](https://www.signadot.com/software-factory/). #### Give every agent task an environment of its own. Your coding agents create ephemeral environments on the cluster you already run, test against real services and data, and open pull requests that are ready for review. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Coding Agent Integration ## Watch Claude Code Validate a Microservices Change in a Closed Loop Source: https://www.signadot.com/blog/claude-code-signadot-validate-demo/ Summary: Video demo of two Claude Code sessions given the same HotROD change (rename the location service's response fields). The session without a verification loop stops when its build and unit tests pass; the session with signadot-validate runs the change against the cluster, finds the broken front end with Playwright, fixes it and validates again. Links the harness and prompts to run the comparison yourself. ### Watch Claude Code Validate a Microservices Change in a Closed Loop Two Claude Code sessions, the same microservices task. One ships a change that looks correct. The other uses signadot-validate to catch a cross-service break before calling the work done. Author Anirudh Ramanathan Published June 4, 2026 Updated October 15, 2026 Reading time 9 min Topics [Agents](https://www.signadot.com/blog/category/agents/) Coding agents are good at writing code. They are not good at knowing whether that code is actually safe to ship. In a cloud-native system, those are two very different things. The gap shows up the moment a change touches more than one service. An agent can make a change to a microservice, build it, and pass its unit tests, then declare the work done. But in a distributed system, correctness lives in the contracts between services, not inside any single one. A field rename that compiles cleanly and passes local tests can still break the service downstream that depends on the old shape. To verify a change in a microservices application, agents need a realistic environment with the changed service running against its real dependencies. Spinning up the full stack for every agent iteration is slow and expensive, and mocks or unit tests can’t catch cross-service contract breaks by design. The agent ships what looks correct, and the break surfaces later in review, in CI, or in production. Across many agents at once, that gap is what the [complete guide to software factories](https://www.signadot.com/software-factory/) calls running open loop. That open loop is the bottleneck for agentic development in cloud-native teams. A closed loop, also called a verification loop, means the agent runs its change against the real system, reads what happened, and fixes what broke before it reports the work done. Boris Cherny, who built Claude Code, [calls it the most important tip](https://x.com/bcherny/status/2044847849662505288) for working with Claude: give it a way to verify its work, which he says gets two to three times more out of it. Agents generate change faster than ever, but every unverified change becomes more work for the reviewers who have to catch what the agent didn’t. Closing the loop is what makes a coding agent trustworthy enough to hand real work to. #### The demo: same task, two agents To show the difference, we gave two Claude Code sessions the exact same task in our demo app, [HotROD](https://github.com/signadot/hotrod): change the location service’s response so the `id` field becomes a JSON string called `locID`, and rename `name` to `id`. HotROD is a ride-share application made up of four microservices plus persistence, which we extended with Kafka and MySQL. Both sessions ran Opus 4.7 at medium effort. The harness and prompts are [on GitHub](https://github.com/signadot/examples/tree/main/closed-loop-comparison), so you can [run the same comparison yourself](#run-the-comparison-yourself). The session on the left is standard Claude Code with no feedback loop. It can’t verify its change against a running environment. The session on the right uses Signadot with the signadot-validate skill, which provisions a lightweight ephemeral environment and guides the agent in validating its code against the actual cluster before calling the work done. Standard Claude Code No feedback loop - Edit the location service - Build and unit tests pass - Declares the work done Front-end contract broken Caught later in review, CI, or prod Claude Code + signadot-validate Closed loop, validates against the cluster - Edit the location service - Build and unit tests pass - Ephemeral environment, real dependencies - Playwright e2e finds the break - Fix the front end, revalidate Verified end-to-end Contracts intact before the change ships Same model. Same prompt. Two outcomes. ##### Left: fast, and wrong The left session made the change, built the Go code, ran the unit tests, and declared success. Build passes, tests pass, done. There was no feedback loop beyond that one service. The change renamed two fields of the location service’s `Location` struct, the unit tests for that single microservice still passed, and the agent had no way to know it had just broken the contract the front end depends on. It completed a change to one service without accounting for the fact that the service is part of a larger stack. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Right: closing the loop The right session made the same change and ran the same build, then paused to set up validation. With Signadot, only the service that changed runs locally in the session. The Signadot operator routes traffic to the rest of the services in the cluster, creating a realistic virtualized environment around the single changed service. All the other dependencies, including Kafka and the database, are the shared stable versions running in the cluster. We asked the agent to use Playwright to test the change end-to-end, driving a real browser through the UI to confirm the user flow still works. It spun up the location microservice locally, confirmed with curl that the new response shape had taken effect, and drove the front end through the ephemeral environment with its routing key. The validation surfaced a real downstream break. The front end’s Go code shares the location service’s `Location` struct, still in its old shape, so it could no longer decode the new response: `GET /splash` returned a 500 with `json: cannot unmarshal string into Go struct field Location.id of type int64`, and the page never got past Loading. The agent also found that the React UI matches locations by their old numeric id, so rebuilding the Go client alone would not fix it. The rename was not safe to propagate and not safe to ship. The agent gave us a clear choice: propagate the rename through to the React front end, or revert. The same change the left session had already declared correct. *Image: Two Claude Code sessions side by side. The left session has finished and gone quiet after its build and tests passed. The right session reports a validation summary that surfaces the downstream break, with the root cause: the cluster frontend shares the location struct through the Go client, so the renamed fields no longer match.* We told it to make the corresponding front-end change and validate again. The agent mutated the environment to add the front end locally alongside the location microservice, with everything else still served from the cluster, and ran Playwright again. This time, the golden path passed end-to-end. The right session made more changes than the left. It also shipped a change that actually works across the whole application, with contracts intact and end-to-end flows verified, all with minimal intervention. The left session was faster, but only because it stopped before finding out it was wrong. #### Why this matters The difference between the two sessions is not model quality. Both ran the same model and prompt at the same effort. The difference is that one agent could see the consequences of its change, and the other could not. This is what Signadot gives coding agents in a cloud-native system: the ability to iterate against a production-like environment, catch cross-service breaks where they work, and deliver code that is correct across the full stack rather than correct in isolation. That’s the layer most agentic development stacks are still missing, and it’s what turns agent speed into output you can actually merge. Because each environment duplicates only the services a change touches, many agents can run this loop at once on one cluster. [Coding agent environments](https://www.signadot.com/solutions/agentic-development/) covers how that works for a whole team. #### Run the comparison yourself The harness behind this comparison is the [closed-loop-comparison](https://github.com/signadot/examples/tree/main/closed-loop-comparison) folder of the signadot/examples repository. It starts Claude Code in a [Docker Sandbox](https://docs.docker.com/ai/sandboxes/get-started/), once on its own and once with Signadot, each with a fresh clone of HotROD, so both sessions begin from the same code. You need: - The Docker Sandboxes CLI, `sbx`, installed and logged in, and `git` - For the Signadot session: a Signadot account and API key, and a Kubernetes cluster connected to your Signadot org with HotROD deployed (the [quickstart](https://www.signadot.com/docs/tutorials/quickstart/first-sandbox) installs it) Start each session in its own terminal to watch them side by side: ``` git clone https://github.com/signadot/examples.git cd examples/closed-loop-comparison # Claude Code on its own ./sbx-run.sh --without-skill # Claude Code with Signadot export SIGNADOT_API_KEY=... export SIGNADOT_ORG=my-org export SIGNADOT_CLUSTER=my-cluster ./sbx-run.sh --with-skill ``` The Signadot session gets everything the closed loop needs, inside the sandbox: - The [Signadot agent skills](https://github.com/signadot/agent-skills), signadot-validate among them - The Signadot CLI, logged in with your API key, and the Signadot MCP server registered with Claude Code - Playwright with Chromium and its MCP server, to drive the front end, and grpcurl, to call gRPC services - A session start hook that runs `signadot local connect`, so the session reaches the services in the cluster Then give both sessions the same task. The prompts are in [prompts/HOTROD.md](https://github.com/signadot/examples/blob/main/closed-loop-comparison/prompts/HOTROD.md), and the third is the rename from the video. The baseline session gets its first version, ending in “validate your work”. The Signadot session gets the second: ``` Make the location service's response return the id field as a JSON string called locID instead of a number, and rename name to id, and validate your work with /signadot-validate ``` The file has more tasks to compare: a fare estimate and ride history feature across three HotROD services, and three prompts for Google’s [microservices-demo](https://github.com/GoogleCloudPlatform/microservices-demo), which you run by passing its URL to `--repo`. Where a prompt has only one version, the Signadot session’s copy ends with “verify with /signadot-validate and playwright on the demo cluster”. To try it on your own services, point `--repo` at your repository and use a cluster where they run. #### Try signadot-validate The signadot-validate skill follows the open Agent Skills format, so it works with Claude Code, Cursor, Codex and the other coding agents that read it. It needs the Signadot CLI, configured against your cluster. Install it with the skills CLI: ``` npx skills add signadot/agent-skills --skill signadot-validate ``` - [Agent skills docs](https://www.signadot.com/docs/integrations/coding-agents/agent-skills) - [Tutorial: closed-loop validation with Claude Code](https://www.signadot.com/docs/tutorials/closed-loop-microservices-validation-claude-code) - [Set up the Signadot MCP server in Claude Code](https://www.signadot.com/docs/integrations/mcp/claude-code) - [The comparison harness and prompts on GitHub](https://github.com/signadot/examples/tree/main/closed-loop-comparison) - [Validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [Get started with Signadot](https://www.signadot.com/signup/) #### Frequently asked questions What is a closed loop for a coding agent? A closed loop, also called a verification loop, is when a coding agent runs its change against the real system, reads the result and fixes what broke before it reports the work done. In an open loop the agent stops once its code builds and its unit tests pass, and anything it broke in another service surfaces later, in review, in CI or in production. Is Claude Code a closed loop on its own? Only for what it can run in its own terminal: it builds the code and runs the tests it finds. A change to one microservice can break another service the agent cannot run, so for cloud-native code the loop stays open. Signadot extends it to the whole application: through the signadot-validate skill, the Signadot CLI and the MCP server, Claude Code runs its change against the real services, queues and databases in a Kubernetes cluster. Why did the unit tests miss the break? They test the location service alone, and the location service still worked. The break was in the contract between services: the front end still decoded the old field names and types, so it failed on the new response. Only a run that sent requests through the front end to the changed service could show it. Does the agent need its own copy of the whole application? No. Only the changed service runs in the agent's session. Signadot routes requests that carry the environment's routing key to it, and every other service, Kafka and MySQL included, is the shared version already running in the cluster. When the fix touched the front end too, the agent added the front end to the same environment. How can I run this comparison myself? Clone the signadot/examples repository and run the sbx-run.sh script in its closed-loop-comparison folder twice, once without Signadot and once with it. Each run starts Claude Code in a Docker Sandbox with a fresh clone of HotROD, and the prompts are in prompts/HOTROD.md. The Signadot run needs an API key and a Kubernetes cluster with HotROD deployed. #### Related Posts - [ *Image: Catch Performance Regressions Before the Pull Request With k6 and Signadot Plans* Tutorials/GuidesAgents ### Catch Performance Regressions Before the Pull Request With k6 and Signadot Plans July 16, 2026 ](https://www.signadot.com/blog/catch-performance-regressions-with-k6-and-signadot-plans/) - [ *Image: Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk* AgentsEphemeral Environments ### Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk October 21, 2026 ](https://www.signadot.com/blog/dzone-trend-talk-cloud-native-foundations/) - [ *Image: Introducing the signadot-validate Skill: Close the Loop for Coding Agents Building Microservices* Product AnnouncementsAgents ### Introducing the signadot-validate Skill: Close the Loop for Coding Agents Building Microservices May 8, 2026 ](https://www.signadot.com/blog/introducing-signadot-validate-skill/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Guides ## Guides Source: https://www.signadot.com/guides/ Summary: Every Signadot guide in one listing, newest first: ephemeral, preview and staging environments, local development on Kubernetes, Kubernetes, microservices and contract testing, Kubernetes sandboxes, testing MCP servers, validating AI-generated code and software factories. ### Signadot Guides [ *Image: Preview Environments: The Complete Guide* Featured #### Preview Environments: The Complete Guide Arjun Iyer · July 31, 2025 What a preview environment is, how it relates to staging and ephemeral environments, the four ways to build one, how the platforms compare, and what coding agents change about them. Read more ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [ *Image: Software Factories: The Complete Guide for Cloud-Native Systems* ### Software Factories: The Complete Guide for Cloud-Native Systems What a software factory is, what a cloud-native one needs, and why most agentic factories close the loop too late: they scaled the coding agents, not the checks that tell them a change works. Arjun Iyer · October 15, 2026 ](https://www.signadot.com/software-factory/) - [ *Image: Kubernetes Testing: The Complete Guide* ### Kubernetes Testing: The Complete Guide How to use Kubernetes as your test platform: why it beats CI runners on cost, security and scale, the tools that run tests in a cluster, and how to give every change its own test environment. Arjun Iyer · October 11, 2026 ](https://www.signadot.com/kubernetes-testing/) - [ *Image: Contract Testing: The Complete Guide* ### Contract Testing: The Complete Guide What a contract test proves and misses, consumer-driven versus provider-driven contracts, the tools from Pact to SmartTests, and what changes when coding agents write both sides of an API. September 16, 2026 ](https://www.signadot.com/contract-testing/) - [ *Image: Staging Environments: The Complete Guide* ### Staging Environments: The Complete Guide What a staging environment is for, why one shared copy becomes a bottleneck at microservices scale and under coding agent volume, and the alternatives teams use instead, compared. September 3, 2026 ](https://www.signadot.com/staging-environments/) - [ *Image: How to Test MCP Servers: A Complete Guide* ### How to Test MCP Servers: A Complete Guide Four levels of MCP server testing, from unit tests and the MCP Inspector to integration tests against real dependencies and agent evals, with each change isolated on a shared cluster. August 31, 2026 ](https://www.signadot.com/testing-mcp-servers/) - [ *Image: Local Development on Kubernetes: The Complete Guide* ### Local Development on Kubernetes: The Complete Guide Four ways to develop locally when services run on Kubernetes, from local clusters and sync tools to a shared cluster and cloud development environments, weighed on speed, quality and cost. July 10, 2026 ](https://www.signadot.com/guide-to-local-development-kubernetes/) - [ *Image: What Is a Kubernetes Sandbox?* ### What Is a Kubernetes Sandbox? The two things a Kubernetes sandbox can mean, a secure box for untrusted or AI-generated code and an isolated environment for testing a change against real services, and when to use each. June 21, 2026 ](https://www.signadot.com/kubernetes-sandbox/) - [ *Image: Validating AI-Generated Code Against Real Kubernetes Dependencies* ### Validating AI-Generated Code Against Real Kubernetes Dependencies Coding agents have moved the bottleneck from writing code to verifying it. What a closed loop against real Kubernetes dependencies needs, from who writes the checks to scaling to agent volume. June 21, 2026 ](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [ *Image: Ephemeral Environments: The Complete Kubernetes Guide* ### Ephemeral Environments: The Complete Kubernetes Guide What ephemeral environments are, the four ways to build them on Kubernetes, the tools compared, and how they handle cost, databases, message queues and the volume coding agents bring. February 28, 2026 ](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [ *Image: Microservices Testing for FinTech* ### Microservices Testing for FinTech Why traditional staging fails FinTech testing, where compliance, customer trust and speed all matter, and how the sandbox model works in practice at Brex and Earnest. February 28, 2026 ](https://www.signadot.com/microservices-testing-for-fintech/) - [ *Image: The Complete Guide to Microservices Testing: From Unit Tests to Production* ### The Complete Guide to Microservices Testing: From Unit Tests to Production How to test services that call each other over the network and through queues and workflows, which tests each layer needs, and where to run them so a passing suite means something. Arjun Iyer · February 28, 2026 ](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [ *Image: The Future of API Validation: A Deep Dive into AI-Powered Contract Testing* ### The Future of API Validation: A Deep Dive into AI-Powered Contract Testing Why traditional contract testing buckles as services and teams grow, how AI-powered checks on every pull request take its place, and what DoorDash, Brex and Earnest gained. February 28, 2026 ](https://www.signadot.com/the-future-of-api-validation-a-deep-dive-into-ai-powered-contract-testing/) - [ *Image: Preview Environments: The Complete Guide* ### Preview Environments: The Complete Guide What a preview environment is, how it relates to staging and ephemeral environments, the four ways to build one, how the platforms compare, and what coding agents change about them. Arjun Iyer · July 31, 2025 ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## The Complete Guide to Microservices Testing Source: https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/ Summary: What microservices testing is, why distributed systems are hard to test, and how to test microservices at every layer: unit, contract, integration, end-to-end, performance, chaos and security tests. Covers synchronous REST and gRPC calls versus asynchronous flows through Kafka, SQS, RabbitMQ and Pub/Sub, how to test services that use a durable workflow engine, real databases and queues, the split between test automation tools and environment platforms, what coding agents change, and a best practices checklist. ### The Complete Guide to Microservices Testing: From Unit Tests to Production How to test services that call each other over the network and through queues and workflows, which tests each layer needs, and where to run them so a passing suite means something. Author Arjun Iyer Published February 28, 2026 Updated October 11, 2026 Reading time 46 min Microservices testing is the practice of checking that each service in a distributed system works on its own, and that the services keep working when they call each other or hand work off through queues and workflows. It exists because a microservices architecture moves correctness out of any single codebase and into the calls, messages and workflow steps between services that different teams build, deploy, and change on different schedules. The biggest risk is not a missing test. It is a suite that passes at every level while the thing it ran against no longer matches production: a mock written against last quarter’s API, a staging environment three deploys behind, a dependency that changed its response shape on Tuesday. Coding agents make this worse by volume. CircleCI’s 2026 report found [engineering throughput up 59 percent year over year while roughly 30 percent of merge attempts fail](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/), which means more changes reaching the seams between services and fewer of them verified there first. The [complete guide to software factories](https://www.signadot.com/software-factory/) looks at the same gap from the delivery side, as agents scaling output faster than verification. This guide covers what microservices testing is, why distributed cloud-native systems are hard to test, and the test types they need: unit, contract, integration, end-to-end, performance, chaos and security. It shows how to test microservices at each stage from a laptop to production, how to test synchronous REST and gRPC calls and asynchronous flows through Kafka, SQS, RabbitMQ and Pub/Sub, and how durable workflow engines change the job. It then covers real databases and queues, what coding agents change, whether you still need a staging environment, how to build a strategy, which tools belong to which layer, and a best practices checklist. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) covers the per-pull-request environment model and the platforms that offer it, and the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/) covers Kubernetes as the platform those tests run on. #### What is microservices testing, and what is it for? Microservices testing validates two things: the behavior of an individual service in isolation, and the behavior of the system when services interact. Unit and component tests cover the first. Contract, integration, and end-to-end tests cover the second, and performance, chaos and security tests check how the whole system holds up under load, under failure and under attack. People also call it testing microservices, microservice testing, testing in microservices, testing of microservices, or distributed systems testing, and the practice is the same under each name. It is for catching three classes of problem before users do: logic bugs inside a service, contract breaks between services, and behavior that only appears when real services run together under real conditions. It is not a substitute for observability in production, and it is not a way to prove a system correct. It narrows the gap between what a test checked and what will run. In a monolith, most of the second category does not exist. A function calling another function shares a process, a compiler, and a deploy, so a changed signature breaks the build. In a microservices architecture, the same call crosses a network to a service that may be on a different version, written in a different language, and owned by a team you have never met. Nothing breaks at build time. It breaks at runtime, in whichever environment first runs the two versions together. That shifts where the effort goes. Testing a monolith is mostly about coverage of logic. Testing microservices is mostly about coverage of interactions: which services call which, with what payloads, through which queues, and under what failure conditions. It is also about whether the version of each service you tested against is the version that will be running when your change ships. Kubernetes made this the default problem rather than a niche one. The CNCF’s 2025 annual survey, published in January 2026, found that [82 percent of container users now run Kubernetes in production](https://www.cncf.io/announcements/2026/01/20/kubernetes-established-as-the-de-facto-operating-system-for-ai-as-production-use-hits-82-in-2025-cncf-annual-cloud-native-survey/). Most of those systems are built from services that talk to each other, and every one of them needs an answer to the question this guide is about. #### Testing in microservices: why distributed systems are hard to test Testing in microservices is hard because correctness lives in the interactions between independently deployed services rather than inside any one of them. Even a small system makes the point. api gatewaycheckoutordersinventorypayments apiorders topicinventory db Six pieces, seven network calls. Each arrow is a seam that no single service's own tests can see across. api gatewaycheckoutordersinventorypayments apiorders topicinventory db Six pieces, seven network calls. Each arrow is a seam that no single service's own tests can see across. These are the microservices testing challenges teams meet first, and they appear long before the system reaches thirty services. - **Failures happen at the seams.** In a monolith, a bug is usually inside a function. In microservices, the bug is usually in the contract between two services: a renamed field, a changed default, a timeout that one side expects and the other does not honor. Neither service’s own tests can see it, because each service is correct by its own definition. - **Every service is on a different version.** Services deploy independently, so at any moment the system is running a mix of versions nobody tested together. A test that pins its dependencies to specific versions is testing a combination that may never exist in production. - **The network is part of the system.** Calls can be slow, can fail partway through, can be retried and arrive twice. Unit tests do not exercise any of this. Integration tests only exercise it if the dependency on the other end is real. - **Data is split.** Each service owns its data, so a workflow that spans services spans databases. Tests that need consistent data across services need either a shared fixture every service agrees on, or a way to give a test its own isolated data without copying every database. - **Async flows have no call stack.** When services communicate through [Kafka](https://kafka.apache.org/) or a queue, a producer’s test cannot see what a consumer did with the message. [Testing synchronous and asynchronous flows](#testing-synchronous-and-asynchronous-flows-in-microservices) covers how to follow a message through a broker. - **Workflows outlive deploys.** A durable workflow can start on one version of the code and finish on another, retrying steps along the way. [Testing durable workflows](#testing-durable-workflows) covers what that means for testing. - **Many teams share one environment.** A test that fails on an environment twenty other changes are using could have been broken by any of them, so every failure starts with an investigation of whose it is. The common thread is that correctness moved out of any one service and into the relationships between them. That is why the environment a test runs against, and how current its dependencies are, matters more for microservices than for anything that came before. #### The testing pyramid for microservices The microservices testing pyramid has five layers rather than the classic three: unit, contract, integration, end to end, and production validation. The two extra layers exist because two new failure modes appear between “the unit works” and “the whole system works”. A service’s API can drift from what its callers expect, and a release can behave differently under real traffic than under any test traffic. productionend-to-endintegrationcontractunitcontinuoustens of minutesminutessecondsseconds Run many cheap tests and few expensive ones. The width of each layer is how many you should have, the label on the right is what each run costs you in time. productioncontinuousend-to-endtens of minutesintegrationminutescontractsecondsunitseconds Run many cheap tests and few expensive ones. The width of each layer is how many you should have, the label beneath each name is what a run costs you in time. Layer What it checks What it runs against What it misses Typical tools Unit A single function or class in isolation, with no network calls Nothing external, only in-process fakes Whether two services still agree on how to talk to each other JUnit, pytest, Go testing, Jest Contract That a service's API or messages still match what its consumers expect: request shapes, response fields, status codes, event payloads A recorded specification of the other service, not the running service How the service behaves under real conditions, past the shape of its responses Pact, Specmatic Integration Real calls and messages between services at the versions deployed Live dependencies at their current versions, most reliably on a shared cluster with request routing Full user journeys spanning the interface and every downstream service at once Language test frameworks, Testkube to run them in the cluster End to end A complete user journey across every service it touches, treating the system as a black box A fully deployed system Little, but slowly and expensively, and a failure rarely says which service caused it Cypress, Playwright, Selenium Production validation That a release behaves correctly under real traffic Production, on a slice of real users Nothing, but it finds problems after users have already met them Canary deploys, feature flags, synthetic checks Two things follow from the table. First, the layers are not interchangeable. A thousand unit tests say nothing about whether the orders service still returns the field the checkout service reads, and a passing contract test says nothing about what happens when the real orders service is slow. Second, the integration layer carries more weight for microservices than the pyramid’s shape suggests. Interactions are where distributed systems fail, so the layer that tests interactions against real services is the one that catches the most expensive bugs. Keeping that layer current is the hard part, because an integration test is only as good as the dependencies it runs against. [Is the testing pyramid broken?](https://www.signadot.com/blog/is-the-testing-pyramid-broken/) argues the case that the classic shape undersells it. Three kinds of test cut across the layers instead of sitting on one. Performance, chaos and security tests each run at several of them, from a benchmark in a unit test to a load test in production, and [the section on them below](#performance-chaos-and-security-testing-for-microservices) covers where each one pays off. #### How to test microservices step by step You test microservices in four stages: unit and local integration checks while the code is in the editor, integration tests against live dependencies at the pull request, end-to-end and performance tests before release, and validation under real traffic in production. Each stage catches a different class of problem, and each one depends on the stage before it having done its job. ##### 1\. Local development: test the one service you changed against real neighbors Start with unit tests for the logic you touched. They run in seconds, need no infrastructure, and should be the first thing that fails when you break something. The harder question at this stage is what to do about dependencies. Running fifty services on a laptop stops working past a handful of services, and [Docker Compose](https://docs.docker.com/compose/) files that try to recreate the system fall out of date as fast as the system changes. The alternative is to run only the service you are changing locally and connect it to a shared Kubernetes cluster that already runs the current version of everything else. [Telepresence](https://www.telepresence.io/) and [mirrord](https://metalbear.com/mirrord/) do this by intercepting traffic between your local process and the cluster. A lightweight ephemeral environment on the cluster does it by tagging your test requests so the cluster sends them to your local service and everything else to the shared copies. The [local development on Kubernetes guide](https://www.signadot.com/guide-to-local-development-kubernetes/) compares the options in detail. Whatever the mechanism, the goal at this stage is the same: find integration problems while the code is still in your editor, when the fix costs minutes rather than a CI round trip. ##### 2\. Pull request and CI: run integration tests against current dependencies before merge Once a change is in a pull request, CI should run the unit suite and then the integration tests for the services the change touches. The second part is where most pipelines fall short. CI runners start in a clean environment with no dependencies. Teams either mock everything, which loses the point of an integration test, or defer integration testing to staging, which moves the feedback to the slowest and most contended point in the pipeline. The fix is to give each pull request an isolated place to run against live dependencies. A lightweight ephemeral environment deploys only the changed services alongside a shared cluster and routes test traffic through them, while every other call falls through to the shared versions. The [Kubernetes sandbox guide](https://www.signadot.com/kubernetes-sandbox/) explains the mechanism. Integration tests run against real services at their current versions, in parallel across every open pull request, without one full environment per pull request. This is also where contract tests belong if you use them. A contract test checks that your service’s API still matches what its consumers expect, and it can run in CI without any dependency being live. It catches a narrower class of bug than an integration test, but it catches it earlier and cheaper. ##### 3\. Before release: end-to-end and performance tests on the paths that matter Before release, run the thin end-to-end suite for the journeys that carry revenue or trust, and a performance check on the paths the change touched. Both need the whole system, not one service, so the environment they run in decides how much a pass is worth. [End-to-end testing microservices](#end-to-end-testing-microservices) and [performance testing](#performance-and-load-testing-microservices) below cover how to run each and where. ##### 4\. Production: validate under real traffic and limit the blast radius Some behaviors only appear under production traffic: real data shapes, real concurrency, real third-party latency. Production validation is the top of the pyramid because it is the only layer that sees them. Canary deployments route a small slice of traffic to the new version and compare error rates and latency against the current version before widening. Feature flags separate deploying code from enabling it, so a change can ship dark and be turned on for one percent of users. Synthetic checks run scripted journeys against production on a schedule so a regression surfaces before a user reports it. None of these replace the earlier layers. They are the safety net for what the earlier layers could not see, and a team that leans on them to catch integration bugs is paying for those bugs in production incidents. [Feature flags versus preview environments](https://www.signadot.com/blog/microservices-testing-feature-flags-vs-preview-environments/) goes deeper on where flags help and where they hide problems. #### Integration testing for microservices Integration testing checks that two or more real services behave correctly when they call each other. Microservices integration testing is the layer that catches the seam failures described above, and the layer where the choice of what to test against makes or breaks the result. ##### Mocks versus live dependencies A mock stands in for a dependency and returns canned responses. Mocks are fast, deterministic, and free of infrastructure, which is why [WireMock](https://wiremock.org/) and [MockServer](https://www.mock-server.com/) are on most teams’ shortlists. They are the right tool for unit and component tests, and for third-party APIs you cannot call from a test at all, which [mocking third-party APIs in an ephemeral environment](https://www.signadot.com/blog/mocking-and-testing-3rd-party-apis-with-sandboxes/) covers. They are the wrong tool for verifying integration between your own services, for a reason that has nothing to do with the mocking library. A mock encodes what one team believed the other team’s API did on the day the mock was written. The other team keeps shipping. Nothing in the pipeline updates the mock, and nothing fails when the mock and the real service diverge. The tests keep passing against a version of the dependency that no longer exists, and the first environment to run the real pair is the one that breaks. [Integration tests pass with mocks but staging still breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) walks through this failure in detail. mocked dependencycheckoutorderslive dependencycheckoutorders A mock encodes what one team believed the other's API did on the day it was written. Only the live dependency tells you what it does now. mocked dependencycheckoutorderslive dependencycheckoutorders A mock encodes what one team believed the other's API did on the day it was written. Only the live dependency tells you what it does now. ##### Contract tests narrow the gap without closing it Contract tests are the disciplined middle ground. The consumer publishes what it expects from the provider, the provider checks it can meet those expectations, and a broker tracks which versions are compatible. They catch breaking API changes before merge and run without a live dependency. What they do not catch is behavior. A provider can satisfy every contract and still return the wrong data, time out under load, or fail on the third retry. Contract tests check the shape of the conversation, and integration tests against the live service check the conversation itself, so most teams that get this right run both. The [complete guide to contract testing](https://www.signadot.com/contract-testing/) covers consumer-driven and provider-driven contracts and the tool landscape in full, and the [guide to AI-powered contract testing](https://www.signadot.com/the-future-of-api-validation-a-deep-dive-into-ai-powered-contract-testing/) covers how the contract layer is changing as agents write more of the code on both sides of an API. ##### Where the live dependencies come from If integration tests need live dependencies at current versions, something has to provide them. There are three models, and they differ in cost, in how far they drift, and in what they require of your services. Model How it supplies live dependencies Where it breaks Prerequisites Best fit Shared staging environment One deployed copy of every service that all teams test against Becomes a queue past a few teams, and drifts from production once keeping it in sync is nobody's job A CD pipeline that deploys to it Under ten services with one or two teams Full-stack ephemeral environment per change A fresh copy of every service for every pull request Each copy costs as much as staging, takes as long to build as the whole system, and starts drifting the moment it is created Namespace or cluster automation, seeded data per copy, budget that scales with pull requests Small systems, or changes that touch most services at once Lightweight ephemeral environment on a shared cluster Deploys only the changed services and routes everything else to the shared copies already running Needs a routing header propagated through every service, and stateful side effects isolated deliberately Header or trace-context propagation across services, a proxy or service mesh Ten or more services, multiple teams, or coding agents opening pull requests The third model is the one most teams past a certain size land on. It is the only one whose environment stays current on its own. There is one shared set of dependencies, the CD pipeline already keeps it current, and every change gets its own isolated slice of it. The prerequisite is real, though. If a service drops the routing header on the way through, requests behind it fall back to the shared copies silently, and the test passes against the wrong version without saying so. #### End-to-end testing microservices End-to-end testing microservices means driving one complete user journey through every service it touches and checking the outcome the user sees, with the system treated as a black box. It is the only test that proves the services work together as a product, and the most expensive one to run and to debug, so the suite should stay small and the environment under it should be trustworthy. ##### How to test microservices end to end 1. **Pick the journeys.** Choose the handful that carry revenue or trust, such as sign-up, checkout, a payment and a refund. Leave edge cases to integration tests, which fail closer to the cause. 2. **Drive them from the outside.** Use [Playwright](https://playwright.dev/) or [Cypress](https://www.cypress.io/) from the UI, or an API client against the public gateway when the journey has no UI. 3. **Own the test data.** Each run creates the users, carts and accounts it needs and tags them, so two runs never read each other’s records. 4. **Wait for outcomes, not for time.** When a step goes through a queue or a workflow, poll for the result with a deadline instead of sleeping a fixed number of seconds. Fixed sleeps are a common source of flaky end-to-end tests in event-driven systems, because the right wait changes with load. 5. **Carry a trace ID.** Start every journey with a trace context, so a failure points at the service and the span where it broke instead of at the whole system. 6. **Run against one change at a time.** A failure on a shared environment could belong to any change deployed there. A failure in an environment that holds one change belongs to that change. ##### Where to run end-to-end tests The traditional answer is a [staging environment](https://www.signadot.com/staging-environments/): one deployed copy of the whole system that every team’s changes pass through before production. Staging works until the number of teams sharing it turns it into a queue, and until the effort of keeping it in sync with production lets it drift. Once it drifts, a passing staging run stops meaning much. The alternative is an [ephemeral environment](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) per change, and there are two ways to build one. A full-stack ephemeral environment copies every service for every pull request, which removes the queue but multiplies the cost and gives each copy its own chance to drift. A lightweight ephemeral environment deploys only the changed services onto a shared cluster and routes everything else to the current copies already running there, so every change gets an isolated slice of one current environment. Whichever you choose, [environment parity](https://www.signadot.com/articles/environment-parity/) with production is the property to protect. #### Testing synchronous and asynchronous flows in microservices Synchronous and asynchronous flows fail in different ways, so they need different tests. A synchronous REST or gRPC call fails while the caller waits, and the error comes back up the chain. An asynchronous flow through Kafka, SQS, RabbitMQ or Pub/Sub fails after the producer has moved on, so the producer’s test passes and the failure shows up later, in another service. Most systems use both, often in the same user journey. The [guide to how microservices communicate](https://www.signadot.com/blog/how-microservices-communicate-with-each-other/) covers the protocols themselves. This section covers how to test each kind of flow. Synchronous · the caller waits checkout REST orders gRPC payments orders publishes and moves on Asynchronous · the producer moves on orders topicKafka consumed later inventory inventory db Test the left side by asserting on the response. Test the right side by asserting on what inventory did, because the publish succeeds either way. ##### Testing synchronous call chains: REST and gRPC A synchronous chain fails at the seam between two services: a renamed field, a changed status code, a timeout one side sets shorter than the other side’s retries. Four checks catch most of it. - **Schema compatibility before merge.** For gRPC, [`buf breaking`](https://buf.build/docs/breaking/) compares a Protobuf schema against a past version and fails the build on a breaking change. For REST, a consumer-driven contract or an OpenAPI diff does the same job. - **The real chain at the pull request.** Send requests through the changed service and the real services it calls, at their current versions. Timeouts, retries and wrong values in correctly shaped fields show up here and nowhere earlier. - **Failure paths, not only the happy path.** Make a dependency slow or return errors and check that the caller times out before its own caller does, retries with backoff, and never retries a call that is not idempotent. [Chaos testing](#chaos-testing-microservices) covers how to inject those faults without hurting anyone else. - **Header propagation.** Trace and routing headers have to survive every hop. A service that drops them breaks tracing in production and breaks request routing in a test environment. [grpcurl](https://github.com/fullstorydev/grpcurl) is the quickest way to call a gRPC service by hand with the headers set. ##### Testing asynchronous flows: Kafka, SQS, RabbitMQ and Pub/Sub Testing async microservices starts from one fact: an asynchronous flow has no call stack to follow. The producer’s test can only prove that it published. Whether the consumer parsed the message, handled a duplicate, or processed messages out of order shows up somewhere else, later. - **Assert on the effect, not the publish.** Publish a real message to a real broker and assert on what the consumer did: the row it wrote, the event it emitted, the call it made. Poll for it with a deadline. - **Test redelivery and ordering on purpose.** Deliver the same message twice and out of order. Brokers do both in production, and an idempotency bug only shows when they do. - **Check message contracts.** [Pact supports message pacts](https://docs.pact.io/getting_started/how_pact_works) for asynchronous messages, and an [AsyncAPI](https://www.asyncapi.com/docs/reference/specification/latest) document, now at version 3, describes channels and payloads the way OpenAPI describes endpoints. - **Keep each test’s messages apart.** On a shared broker, a consumer under test will happily process another team’s messages, and the stable consumer will process the test’s. Either give the test its own topic or queue, or tag each message with a routing key so only the consumer under test picks it up. The routing-key approach keeps one broker for everyone. Producers carry the key in message headers, and each consumer version processes only the messages meant for it. [Message isolation for Kafka, SQS and more](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) shows how that maps onto each broker. The [guide to testing Kafka-based microservices](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) compares cluster, topic and message-level isolation and walks through a working setup, and there are hands-on versions for [SQS](https://www.signadot.com/docs/tutorials/testing-sqs-microservices), [RabbitMQ](https://www.signadot.com/docs/tutorials/testing-rabbitmq-microservices) and [Google Pub/Sub](https://www.signadot.com/docs/tutorials/testing-google-pub-sub-flows). [Testing event-driven architectures](https://www.signadot.com/blog/testing-event-driven-architectures-with-signadot/) covers the same ground at the architecture level. Testing synchronous and asynchronous flows in microservices, October 2026. Synchronous: REST, gRPC Asynchronous: Kafka, SQS, RabbitMQ, Pub/Sub How it fails In the call, while the caller waits After the producer has moved on, in another service What the producer's own test proves The response it got back Only that it published Typical bugs Renamed fields, changed status codes, timeouts, retry storms Parse failures, duplicates, out-of-order processing, poison messages Contract check `buf breaking` for Protobuf, Pact or an OpenAPI diff for REST Pact message pacts, AsyncAPI documents What to assert on The response, its latency and its errors The consumer's effect, polled with a deadline Isolation per change Route requests by a header Route messages by a header, or a topic or queue per change #### Testing durable workflows Durable workflow engines make multi-step processes survive crashes and restarts by recording each step and replaying the record after a failure. That is why teams adopt them, and it is also what makes them harder to test than a plain service. A workflow outlives deploys, retries steps on its own, and depends on code that behaves the same way every time it replays. ##### Why durable workflows add testing complexity - **Replay has to be deterministic.** The engine rebuilds a workflow’s state by replaying its recorded history through the workflow code. A change that alters the order of steps can break every workflow already in flight, and no test of a single request sees it. - **Workflows outlive deploys.** A workflow started on Monday may still be running when Thursday’s version ships. The test that matters is old history against new code, plus new code against the services its steps call today. - **Retries are automatic.** The engine retries a failed step without asking, so a step with a side effect that is not idempotent charges twice, sends two emails or writes two rows. - **Time is part of the logic.** Timers, deadlines and sleeps of days have to be tested without waiting days. - **Steps call real services.** A workflow is a sequence of calls to other services, databases and queues, so its integration test has the same dependency problem as any other. 1 · Workflow logic test serverskips time replayerold histories, new code then 2 · Against real services workflow engineshared worker v2its routing key only worker v1everything else services, db, queueshared, current versions The engine stays shared. Work that carries the change's routing key goes to the changed worker, and everything else to the current one. ##### How to test services that use a durable workflow engine Test them in two steps, and run the first one everywhere. 1. **The workflow logic, with the engine’s own tools.** Workflow SDKs generally ship a test environment that runs workflows against a local test server and skips time, so a workflow that sleeps for a week finishes without waiting a week. Most also replay recorded histories against new code, so a change that would break workflows already in flight fails the build. Both run in CI with no cluster. 2. **The changed code against real services.** Signadot treats the workflow engine like any other shared dependency, the way it treats a database or a broker, so the engine stays shared. Work started by the change carries its routing key, the changed worker picks up only that work, and the current workers handle everything else. The steps the changed code runs call the shared services, routed by the same key, and a step that writes data can get its own database or schema for the change. Nothing another team is running is affected. The [Temporal workers tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) walks through the second step end to end, with workers in Python, TypeScript, Go and Java sharing one task queue. #### Testing microservices with real databases, queues and workflows Mocks miss the bugs that live in state and in asynchronous handoffs: a migration that breaks a service still on the old schema, a consumer that misreads a renamed field, a workflow step that runs twice on retry. Catching them takes the real database, broker and workflow engine in the test, with each change’s use of them kept apart from everyone else’s. ##### Why mocks miss async and stateful bugs A mock answers the question it was written for. These bugs come from questions nobody wrote down. - **Schema and data.** A migration adds a required column while another service, not yet redeployed, still inserts rows without it. An in-memory fake accepts the insert. The real database rejects it. - **Messages.** A producer renames a field in an event, and the consumer’s tests still pass against last quarter’s sample payload. Brokers redeliver and reorder, which a stubbed queue never does. The [section on asynchronous flows](#testing-asynchronous-flows-kafka-sqs-rabbitmq-and-pubsub) covers both. - **Durable workflows.** A durable workflow retries and replays steps on its own, so a step that is not idempotent fails on the second attempt that a mock never makes. - **Batch and scheduled jobs.** A nightly job processes whatever the online services wrote that day, so its test is only as good as the data and messages it runs against. Contract tests catch part of the message case when a team publishes schemas for its events. None of the four shows up in a unit test. ##### How to give each change real state without copying everything A full copy of every database, broker and workflow cluster per pull request is the slowest and most expensive part of a full-stack environment, and most changes do not need it. There are three ways to keep a test’s state apart, from cheapest to most isolated: 1. **Share the stable copy and partition it.** Tests write rows under their own tenant or user and clean up after themselves. This covers most changes that read data or add rows. 2. **Share the broker or task queue and route by key.** Producers and workflow starters attach the change’s routing key, and the consumer or worker under test handles only work carrying it while the stable versions handle the rest. No second Kafka or Temporal cluster is needed. 3. **Give the change its own copy.** A schema change or destructive test gets a temporary database, schema or branch, created when the environment starts and deleted when it ends. The same goes for a topic, queue or bucket the change needs to itself. The [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) covers each mechanism in detail. A Signadot ephemeral environment uses all three. It duplicates only what changed or needs isolating and links it to the rest: it routes requests by key, routes messages and Temporal tasks by the same key, and changes a forked service’s configuration to point it at its own database or queue. [Resource plugins](https://www.signadot.com/docs/reference/resource-plugins) create those per-change resources, in the cluster or outside it, such as managed databases, queues and buckets, and delete them with the environment. [How Bitso branches a Neon database for every environment](https://www.signadot.com/blog/how-bitso-is-scaling-branch-based-development-with-signadot-and-neon/) shows the database case in production use. #### Performance, chaos and security testing for microservices Performance, chaos and security tests check properties of the whole system rather than a feature: how fast it is under load, how it behaves when a dependency fails, and whether one service can reach data it should not. In microservices each of those properties lives in the calls between services, so each test needs real services to call. Performance, chaos and security testing for microservices, October 2026. Test What it finds in microservices Where it runs Tools Performance and load Latency that adds up along a call chain, a new loop that calls a downstream service once per item, connection pool limits, consumers falling behind a topic Per change against the changed service, and at full scale in an environment sized like production Grafana k6, Gatling, Apache JMeter, Locust Chaos Missing timeouts, retry storms, fallbacks that never fire, one slow dependency taking down its callers Per change for a single injected fault, and as scheduled experiments in pre-production and production Toxiproxy, Istio fault injection, Chaos Mesh, LitmusChaos, AWS Fault Injection Service, Gremlin Security Broken authorization between services, internal APIs open to any caller, vulnerable images, misconfigured clusters Static checks in the editor and CI, dynamic tests against running services Semgrep, Trivy, kube-bench, ZAP ##### Performance and load testing microservices Performance testing microservices has two jobs that need two different setups. The first is catching regressions: did this change make the checkout path slower? That test is small, runs on every pull request against the changed service and the real services behind it, and compares the result with the shared version of the same service measured at the same time. Compare relative numbers rather than fixed thresholds on a shared pre-production cluster, because the dependencies under both versions are shared and their load moves. The second job is capacity, which is what most writing on microservices performance testing means: how much traffic the system takes before it degrades. That test needs an environment sized like production with nothing else running on it, and it runs on a schedule or before a launch, not on every change. In microservices a regression rarely sits inside one service. Latency adds up along a call chain, a new loop calls a downstream service once per item, or a consumer falls behind its topic, so the test has to run through the real chain to see it. [Grafana k6](https://grafana.com/docs/k6/latest/) takes scripts in JavaScript or TypeScript and has a [gRPC client](https://grafana.com/docs/k6/latest/javascript-api/k6-net-grpc/) built in. [Gatling](https://github.com/gatling/gatling), [Apache JMeter](https://jmeter.apache.org/) and [Locust](https://locust.io/), which takes scenarios in Python, cover the same ground. [Running scalable k6 tests against a pull request’s version of a service](https://www.signadot.com/docs/tutorials/k6-load-testing) walks through the per-change setup. ##### Chaos testing microservices Chaos testing injects failures on purpose, such as latency, errors, dropped connections or a killed pod, to check that the system degrades the way its design says it should. For microservices the useful questions are concrete. Does the caller time out before its own caller does? Does it retry with backoff or all at once? Does the fallback ever fire? Start small and per change. A fault injected between the changed service and one dependency, with a TCP proxy such as [Toxiproxy](https://github.com/Shopify/toxiproxy) or the mesh’s own [fault injection in Istio](https://istio.io/latest/docs/tasks/traffic-management/fault-injection/), tests the change’s timeout and retry handling in minutes. Scope matters on a shared cluster. Inject the fault only on the path of the change under test, or every other team’s tests fail with it. Cluster-wide experiments, such as killing pods or cutting off a zone, belong in scheduled runs in pre-production and in production game days. [Chaos Mesh](https://www.cncf.io/projects/chaosmesh/) and [LitmusChaos](https://www.cncf.io/projects/litmus/), both CNCF incubating projects, run them in Kubernetes, and [AWS Fault Injection Service](https://docs.aws.amazon.com/fis/latest/userguide/what-is.html) and [Gremlin](https://www.gremlin.com/) offer managed versions. ##### Security testing microservices Microservices security testing adds a question a monolith never had to ask: which service may call which, with what data? The first risk on the [OWASP API Security Top 10](https://api-security.owasp.org/), last revised in 2023, is broken object level authorization, an API that returns any record it is asked for and trusts the caller to have checked. Inside a cluster, internal APIs are often called with no authorization at all, on the assumption that only trusted services can reach them. Most of the work is automated and runs early: static analysis such as [Semgrep](https://github.com/semgrep/semgrep) in the editor and CI, image and configuration scanning with [Trivy](https://trivy.dev/), and [kube-bench](https://github.com/aquasecurity/kube-bench) to check the cluster against the CIS Kubernetes Benchmark. Dynamic tests need a running system. [ZAP](https://www.zaproxy.org/), the open source scanner now backed by Checkmarx, probes a service’s API from the outside, and authorization tests call each service as the wrong user and as the wrong service and expect a refusal. Those tests mean most against real services, where the mesh’s mutual TLS and authorization policies apply the way they do in production. #### What AI coding agents change about testing microservices Coding agents change microservices testing in two ways: they multiply the number of changes that need verifying, and they need that verification to happen without a person in the loop. Testing AI-generated code in a microservices architecture is the same problem as testing any other change, run at a volume and cadence the old pipeline was not built for. The volume is measurable. GitHub’s own account of its August 2026 outage puts [merged pull requests at about 130 million a month](https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/), well over double the monthly rate it reported for 2025, and [Claude Code](https://code.claude.com/docs/en/overview), [Cursor](https://cursor.com/), Codex, and [GitHub Copilot](https://github.com/features/copilot) all open pull requests on their own now. None of that volume removes a bottleneck that already existed in microservices testing. It multiplies the changes arriving at each one, and every one of those pull requests touches services whose neighbors it cannot see. Agents are also consumers of the test environment, not only producers of changes. An agent works in a build-test-fix loop: write the change, run it against its dependencies, read the failure, fix, repeat. That loop cannot wait for a shared staging slot or a fifteen-minute environment build. An agent without a fast integration environment falls back to unit tests and mocks, which is exactly the layer that cannot see the seams. Verification has to happen before human review, not after. A reviewer’s time is the scarce resource once agents generate the code, so the integration and end-to-end tests that used to run after approval need to run before the pull request reaches a person. The pull request arrives with evidence that the change worked against real dependencies, or it does not arrive at all. Each agent run needs its own isolation. Two agents testing conflicting changes to the same service on one shared environment corrupt each other’s results, so every run needs an isolated slice of the dependency graph, created and destroyed on the agent’s timeline. That also means cost and lifetime controls: a time-to-live on every environment, and a cost model that scales with changed services rather than with the number of agents running. An agent reaches the environment through the same interfaces a developer uses. It can run the CLI, call the API, or use a Model Context Protocol (MCP) server that exposes environment creation and test execution as tools it invokes directly, and which of the three it uses depends on where the agent runs. [Validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) covers that path end to end, including how an agent reads a failed integration run and iterates on it. #### Do you need a staging environment to test microservices at all? You need what staging is for: a shared environment where a change meets the current versions of everything it depends on before users do. What stops working past a few teams is handing that environment to one change at a time. The “staging is dead” argument is right that one shared copy of the system turns into a queue, and wrong when it concludes that mocks, contracts, and canaries can replace the integration layer entirely. Staging as one shared copy is still the right answer for small systems. Under about ten services with one or two teams, conflicts are rare, drift is visible because everyone can see the whole system, and the cost of one shared copy is trivial. Replacing it with per-change environments at that size adds machinery without removing a problem. Past that size, keep the shared environment and change how changes use it. Every change still runs against live dependencies at current versions, but each one gets an isolated slice of the shared environment rather than a turn on it. [Kubernetes staging environments](https://www.signadot.com/blog/kubernetes-staging-environment/) covers how teams make that move without a flag day. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Building a microservices testing strategy A microservices testing strategy is the decision about how much to invest at each layer of the pyramid and where the integration layer gets its live dependencies. Most microservices testing strategies come down to four questions, and one principle organizes the answer. ##### How many services, and how many teams own them? Below about ten services with one or two teams, a shared staging environment and a thin end-to-end suite usually work. Conflicts are rare and drift is manageable because everyone can see the whole system. Past that, staging becomes a queue and the drift becomes invisible, because no one team owns the whole picture any more. This is the point where integration testing needs an isolated place to run per change, and where the choice between copying environments and sharing one becomes the central architecture decision in the strategy. ##### How do your services talk to each other? A system built mostly on synchronous calls puts its weight on integration tests of call chains and on contract checks for each API. A system built on Kafka, a queue or a workflow engine needs message and workflow tests at the pull request too, plus chaos tests for redelivery and retries, because that is where its failures hide. Map the flows first, then decide which [synchronous and asynchronous tests](#testing-synchronous-and-asynchronous-flows-in-microservices) each one needs. ##### Where do failures come from? Pull the last quarter’s production incidents and sort them: bugs inside a single service, failures at the seam between two services, and configuration or environment differences. Teams that do this usually find the second and third categories dominate while their test investment is still weighted toward the first. If that is your result, shift effort up the pyramid: fewer redundant unit tests and more integration tests against live dependencies. Add a hard rule that every pull request runs its integration tests before merge rather than after. ##### How fast is the change rate growing? A pipeline sized for today’s pull request volume will be undersized within a year for most teams, and within months for teams adopting coding agents. Design for the volume you expect, not the volume you have. The cost model decides whether that is affordable. A full-stack environment per change scales cost with the number of pull requests. A lightweight ephemeral environment on a shared cluster scales cost with the number of changed services per pull request, which is usually one or two regardless of how many pull requests are open. ##### Shift-left testing for microservices Shift-left testing means running each layer of the pyramid at the earliest stage that can support it: unit tests in the editor, integration tests at the pull request, end-to-end tests before release rather than after. For microservices the shift that pays most is moving integration tests from staging to the pull request, because that is where seam failures are cheapest to fix and where a queue of changes waiting on one environment forms first. [Shift-left testing for microservices](https://www.signadot.com/articles/shift-left-testing/) works through which layers move, what the pull request environment has to provide, and the two ways to build it. ##### How do you know the strategy is working? Measure it with the four [DORA metrics](https://dora.dev/guides/dora-metrics-four-keys/): lead time for changes, deployment frequency, change failure rate, and time to restore. A testing strategy that moves integration tests to the pull request should show up as shorter lead time and a lower change failure rate within a quarter, because the seam failures that used to surface in staging or production now surface before merge. If change failure rate does not move, the tests are still running against the wrong dependencies. If lead time does not move, the environment is still a queue. [How to do DORA metrics right](https://www.signadot.com/blog/how-to-do-dora-metrics-right/) covers how to instrument the four without gaming them. #### Test automation tools vs environment platforms Microservices testing needs two layers of tooling that are easy to confuse. Test automation tools write, run and orchestrate the tests, from unit frameworks and contract brokers to load generators and in-cluster orchestrators such as Testkube. Environment platforms give each change somewhere real to run those tests, against current dependencies and isolated from other changes, which is what Signadot, Release, Bunnyshell and Okteto do. Most teams need something from each layer, and two tools from the same layer still leave the other gap. Test automation tools vs environment platforms, October 2026. Layer The question it answers Examples What it leaves to the other layer Test automation and orchestration What do we test, and how do the tests run and report? JUnit or pytest, Pact and Specmatic, Playwright and Cypress, k6 and JMeter, Testkube to run them in the cluster The services under test, their dependencies, and keeping both current Environment platforms Where does this change run, against which versions of its dependencies, isolated from every other change? Signadot, Release, Bunnyshell, Okteto, Qovery, or an in-house namespace per pull request What to test, and whether the change works ##### Where the two layers overlap The one place the layers meet is running the tests. An orchestrator such as Testkube executes suites inside the cluster but expects the environment to exist already. Some environment platforms can start a suite too: Signadot can run one as a Job in pods next to an ephemeral environment, for teams that have nowhere else to run it. The [comparison of Testkube and Signadot Jobs](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/#kubernetes-test-orchestration-testkube-and-signadot-jobs) covers what each one does. For a testing strategy, the decision is where results land, not which runner is better. Keep one place that runs and reports tests: CI, the orchestrator you already have, or a pod in the cluster, pointed at each change’s environment. The environment platform provides the place to run, and the tests, written by people or by agents, decide whether the change works. ##### Match tools to each layer of the pyramid Most teams end up combining tools, one or two per layer of the pyramid. The [survey of microservices testing tools for Kubernetes](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) compares the field in depth, environment platforms included. In brief: - **Unit and component:** the language’s own test framework, with WireMock or MockServer standing in for HTTP dependencies. - **Contract:** [Pact](https://pact.io/), the open source framework, with SmartBear’s [PactFlow](https://pactflow.io/), now sold as Swagger Contract Testing, as the hosted broker, or [Specmatic](https://specmatic.io/) for teams that want to drive contracts from OpenAPI specs. [Signadot versus Pact](https://www.signadot.com/comparison/pact/) covers where contract tests stop and live integration tests start. - **Local development against a cluster:** [Telepresence](https://www.telepresence.io/), a CNCF Sandbox project created at Ambassador, which [Gravitee acquired in 2025](https://www.gravitee.io/blog/ambassador-labs-gravitee-what-our-acquisition-means-for-you). And mirrord from MetalBear, which began as a way to run a local process against cluster traffic and has grown cluster-side features since. Its Team tier adds queue splitting for Kafka and SQS and database branching, and its Enterprise tier adds preview environments and CI use. [Signadot versus mirrord](https://www.signadot.com/comparison/mirrord/) compares the two models. So does [Signadot versus Telepresence](https://www.signadot.com/comparison/telepresence/). - **Test orchestration in Kubernetes:** [Testkube](https://testkube.io/) runs existing Postman, k6, Playwright, Cypress, and pytest suites as workflows inside the cluster, against an environment you provision. - **End to end:** Playwright or Cypress from the UI. - **Performance, chaos and security:** k6, Gatling or JMeter for load, Chaos Mesh or LitmusChaos for fault injection, Trivy and ZAP for security, as covered in [the section above](#performance-chaos-and-security-testing-for-microservices). - **Where integration and end-to-end tests run:** an environment platform that gives each change live dependencies at current versions, such as the lightweight ephemeral environment per change covered in the closing section. #### Running the shared test environment like a product Whichever model supplies your live dependencies, the shared environment behind it needs an owner, a definition of current, and a way to tell whose change broke what. Teams that skip this end up with the drift they were trying to escape, one layer down. Ownership comes first. The shared set of dependencies is a platform team’s product, with a stated objective for how far behind the main branch it may fall and an alert when it does. Parity is a mechanism, not a wish: the same CD pipeline that deploys production deploys the shared environment, from the same images and configuration, so the two cannot diverge by hand. Data and async flows need tiers. Shared read-mostly reference data can live once. Anything a test writes, publishes or starts needs the isolation described in [testing with real databases, queues and workflows](#testing-microservices-with-real-databases-queues-and-workflows), or a test on the orders service ends up processing another team’s events and both results are noise. A migration under test must never reach the copy every other change depends on. Access decides whether the environment gets used. QA, product owners, and reviewers need a URL per change. Developers need to connect a local process into the shared environment from their editor. Agents need the same through a CLI or MCP, with a time-to-live on everything they create. When a test fails against forty real services, tracing has to attribute the failure to the one service that changed, or every failure becomes an investigation. #### Microservices testing best practices checklist Twelve yes-or-no questions. A team that can answer yes to all of them has closed the gap this guide is about. 1. Does every service have fast unit tests that run in the editor and in CI? 2. Do contract tests run before merge for every API and event another team consumes? 3. Does every pull request run integration tests against live dependencies before a human reviews it? 4. Are those dependencies at their current versions, kept in sync by the same pipeline that deploys production? 5. Can two changes to the same service be tested at the same time without interfering? 6. Does each change that writes data get its own isolated schema or database? 7. Do async flows through Kafka, a queue, or a durable workflow engine get the same isolation as synchronous calls? 8. Is your end-to-end suite thin enough that a failure is investigated the same day, and does it wait on outcomes rather than fixed sleeps? 9. Do latency-sensitive paths get a performance check on every change, and critical dependencies a fault-injection test for timeouts and retries? 10. Is authorization between services tested, rather than assumed because the caller is inside the cluster? 11. Can developers and coding agents create, test against, and destroy an environment on their own, with a time-to-live on each one? 12. When an integration test fails, does tracing point at the one service that changed? #### How lightweight ephemeral environments keep every test running against current dependencies A lightweight ephemeral environment keeps tests current by giving every change its own isolated copy of only the services it touched, running against the shared versions of everything else, so integration and end-to-end tests always meet current dependencies. Signadot is a mature implementation of this pattern, and the [microservices testing solution](https://www.signadot.com/solutions/microservices-testing/) page covers how teams adopt it. The mechanism, in plain words: duplicate only what changed or needs isolating, link it to the rest through request routing, message routing and configuration changes, and manage the lifecycle through a CLI, an API, or MCP. One Kubernetes cluster runs the current version of every service, kept current by the same CD pipeline that deploys production. When a change needs testing, an ephemeral environment deploys only the changed services alongside them and expires on a timer once the change merges or is abandoned. Test requests carry a routing key in a header. A service mesh or lightweight sidecar reads that key and sends the request to the forked service, while every other call falls through to the shared copy. The same key travels through message queues and Temporal task queues for async flows, and an environment that touches data can carry its own database or schema so its writes never reach the shared copies. Resource plugins create anything else a change needs to itself, inside the cluster or outside it. Versions of services that run outside Kubernetes, on ECS, Lambda or virtual machines, are coming soon. requestservice-2service-1service-1bFORKservice-3dbtagged requestshared dependencies The tagged request reaches the forked service and then calls the same shared dependencies as everything else. requestservice-1service-1bFORKdbtagged requestshared dependencies The tagged request reaches the forked service and then calls the same shared dependencies as everything else. An integration test for a two-line change therefore runs against every other service at its current version without deploying those other services. Spin-up takes seconds because only the changed service is deployed, and cost scales with changed services rather than total services. [Brex runs pre-merge environments this way for hundreds of engineers](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) and saves about $2 million a year in infrastructure compared with duplicating environments. Drift stops being a problem because there is one environment, and the CD pipeline already keeps it current. That is the property the rest of this guide has been circling. When every test runs against dependencies that are current, the distance between a passing test suite and a working production system gets small enough to trust. It stays that small as the number of services and the number of authors keep growing. #### Where to go next If you run fewer than ten services and staging still works, read the [staging environments guide](https://www.signadot.com/staging-environments/) and keep it healthy. If staging has become a queue, start with the [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) and the comparison of environment architectures above. If most of your flows run through Kafka or a workflow engine, start with [asynchronous flows](#testing-asynchronous-flows-kafka-sqs-rabbitmq-and-pubsub) and [durable workflows](#testing-durable-workflows) above. If agents are opening your pull requests, start with [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/), because the pipeline you build for them is the pipeline everyone else ends up using. #### Related reading - [Contract Testing: The Complete Guide](https://www.signadot.com/contract-testing/) - [Best Microservices Testing Tools for Kubernetes in 2026](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) - [Integration tests pass with mocks but staging still breaks: how to catch it earlier](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [How Microservices Communicate with Each Other](https://www.signadot.com/blog/how-microservices-communicate-with-each-other/) - [Testing Microservices: Message Isolation for Kafka, SQS, More](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) - [Tutorial: Testing Kafka-Based Microservices in Kubernetes](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) - [Tutorial: Test Temporal Workers on Kubernetes](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - [Tutorial: Load Testing Pull Requests with Grafana k6](https://www.signadot.com/docs/tutorials/k6-load-testing) - [Is The Testing Pyramid Broken?](https://www.signadot.com/blog/is-the-testing-pyramid-broken/) - [Shift-Left Testing for Microservices on Kubernetes](https://www.signadot.com/articles/shift-left-testing/) - [Local Development on Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-local-development-kubernetes/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [Microservices Testing for FinTech](https://www.signadot.com/microservices-testing-for-fintech/) - [Kubernetes Testing: The Complete Guide](https://www.signadot.com/kubernetes-testing/) #### Frequently asked questions What is microservices testing, and how is it different from testing a monolith? Microservices testing checks that each service works on its own and that services keep working when they call each other over the network or hand work off through queues and workflows. Testing a monolith is mostly about logic inside one codebase, where a changed signature breaks the build. Between services the same break happens at runtime, across versions owned by different teams, so more of the effort moves to integration. What is the difference between microservices testing and integration testing? Integration testing is one layer inside microservices testing, not a separate practice. Microservices testing covers unit, contract, integration, end-to-end, performance, chaos and security tests together. Integration testing checks specifically that two or more real services behave correctly when they call each other, at the versions actually deployed. Contract testing sits between the two and checks API and message shapes without running the other service. What types of tests does a microservices application need? Seven kinds, in different amounts. Unit tests cover logic inside a service. Contract tests check API and message shapes. Integration tests run real services together. End-to-end tests drive a whole user journey. Performance tests find slow call chains, chaos tests check behavior when a dependency fails, and security tests check authorization between services. In microservices testing, the last five only mean something against real dependencies. How do you test microservices end to end? Keep the suite thin and run it against real services. Pick the few user journeys that carry revenue or trust, drive each through the UI or the public API with Playwright, Cypress or an API client, and run it in an isolated environment for each change instead of a shared one. Poll for asynchronous outcomes with a deadline instead of sleeping. End-to-end testing microservices this way leaves one change to blame for a failure. How do you test synchronous and asynchronous flows between microservices? Test a synchronous REST or gRPC call by sending a request through the real chain of services and checking the response, the latency and the error handling. Test an asynchronous flow by publishing a real message to a real broker and asserting on what the consumer did. In microservices testing both need isolation per change: request routing for calls, and message routing or a separate topic or queue for Kafka, SQS, RabbitMQ and Pub/Sub. How do you test microservices that use a durable workflow engine? In two steps. First test the workflow logic with the engine's own tools: a test server that skips time, and a replay of recorded histories against new code to catch changes that would break workflows already in flight. Then run the changed workflow code against the real services its steps call. Microservices testing that stops at the first step misses the retry, replay and versioning bugs durable workflows are prone to. Why do microservices tests pass but production still breaks? Because the tests ran against dependencies that no longer match what runs in production. Mocks go stale as the services they imitate change, and a staging environment three deploys behind gives the same false confidence. The fix in microservices testing is to run integration tests against the current shared versions of dependencies rather than against static mocks or a copy of the system that has drifted. How many environments do you need to test microservices safely? Three layers, not a separate dev, QA, staging and pre-production environment apiece. You need a local or agent working copy of the one service being changed, one shared cluster holding the current versions of every other service, and a way to give each change its own isolated slice of that cluster. That is enough for microservices testing at any number of services, because one cluster serves every change at once. Can a coding agent test a microservices change against real dependencies before it opens a pull request? Yes, if it has somewhere to run the test. An agent works in a build-test-fix loop and needs an isolated environment with real dependencies it can create, test against, and tear down without waiting on a person. Given a CLI, an API or an MCP server that exposes those actions, agents like Claude Code and Cursor run microservices testing before a reviewer ever sees the change. How do AI coding agents change microservices testing requirements? They raise the volume of changes and remove the person from the loop. Every agent run needs its own isolated environment with live dependencies, a time-to-live so abandoned runs clean themselves up, and integration results attached to the pull request before human review. Microservices testing for agent-written code is the same set of layers as before, run per change at a cadence a shared staging environment cannot keep up with. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Kubernetes Testing: The Complete Guide Source: https://www.signadot.com/kubernetes-testing/ Summary: Kubernetes testing as using Kubernetes as your test platform. Why a cluster makes better test infrastructure than CI runners, a dated comparison of hosted CI runners, self-hosted runners on Kubernetes (Actions Runner Controller, Jenkins, GitLab, CircleCI) and tests as cluster workloads on cost, security and scale, the Kubernetes testing ecosystem (Tekton, Argo Workflows, Testkube, the k6 Operator, kind), which tests belong on a cluster, a dated comparison of Kubernetes test environments (local clusters, namespace or vCluster per PR, full copies, selective isolation), running integration and end-to-end tests on Kubernetes from CI, and what coding agents change. ### Kubernetes Testing: The Complete Guide How to use Kubernetes as your test platform: why it beats CI runners on cost, security and scale, the tools that run tests in a cluster, and how to give every change its own test environment. Author Arjun Iyer Published October 11, 2026 Reading time 24 min Kubernetes testing means using Kubernetes as the platform your tests run on: test runners as pods, test environments as workloads in a cluster, and the services under test running right next to them. It is how teams move integration, end-to-end and performance tests off CI runners and onto infrastructure shaped like production. (The same words are sometimes used for validating a cluster itself, with conformance suites and upgrade rehearsals. That is a platform team’s job and outside this guide.) The reason to do it is that CI runners were built to compile code and run unit tests, not to test distributed systems. A GitHub-hosted runner sits outside your network, holds one job’s worth of compute, and sees none of the services, databases and queues your change will meet in production. A Kubernetes cluster already runs all of those, schedules work elastically, and gives every pod the same DNS, network policies and service mesh as production. Coding agents make the gap expensive. In CircleCI’s 2026 data, [throughput rose 59 percent year over year while close to a third of merge attempts failed](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/), and a 50-engineer team that used to open 100 to 150 pull requests daily may see 1,000 when agents open them. Each of those changes needs real tests on real infrastructure, many times over, and runner minutes and concurrency caps add up fast. This guide covers why Kubernetes makes good test infrastructure, how it compares with running tests on GitHub Actions, CircleCI or Jenkins runners on cost, security and scale, the Kubernetes testing ecosystem, which tests belong on a cluster, why Kubernetes is the natural place to test microservices, how to run integration and end-to-end tests there from CI, and what coding agents change. What makes microservices hard to test in the first place, from synchronous and asynchronous flows to workflow engines, is the subject of the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). For a vendor-by-vendor list, see the [Kubernetes testing tools comparison](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/). #### What is Kubernetes testing? Kubernetes testing is the practice of running software tests on a Kubernetes cluster instead of, or alongside, CI runners and developer machines. Three things move onto the cluster: - **The test runners.** CI jobs run as pods, started for a job and removed after it, through tools such as Actions Runner Controller, the Jenkins Kubernetes plugin or the GitLab Kubernetes executor. - **The test environments.** The services a change is tested against run as workloads in the cluster, either copied for each change or shared with only the changed parts isolated. - **The tests themselves.** Suites run as Kubernetes jobs next to the services they exercise, started by a test orchestrator such as Testkube, a pipeline engine such as Tekton or Argo Workflows, or a plain Kubernetes Job. What it is for: - Reaching private services, databases and queues from tests without exposing them to the internet. - Running integration, end-to-end and load tests in the same runtime as production: same images, manifests, DNS, mesh and policies. - Scaling test capacity with demand instead of buying a bigger runner plan. - Giving every pull request and every coding agent its own test environment in parallel. What it is not for: unit tests and builds, which run fine and cheaply on any CI runner, and production monitoring, which no pre-production test replaces. People also ask how to test Kubernetes or search for k8s testing, testing in Kubernetes and Kubernetes automated testing. When they mean their applications, this is the practice they are after. #### Why Kubernetes makes good test infrastructure Kubernetes was built to run many short-lived, isolated workloads on shared machines, which is a description of a test fleet. Five properties do the work. **Elastic capacity.** The scheduler packs test pods onto nodes by their resource requests, and a cluster autoscaler or Karpenter adds nodes when the queue grows and removes them when it drains. A burst of a hundred pull requests becomes a burst of a hundred pods, not a hundred jobs waiting for a runner slot. **The same runtime as production.** A test pod runs the same container images, receives configuration through the same manifests, and resolves services through the same cluster DNS as production workloads. Failures that come from configuration, resource limits, sidecars or network policy show up in the test, not after the deploy. **Network locality.** Tests run inside your network, next to the services they call. A private database, an internal API or a message broker is reachable without a public endpoint, a VPN or a tunnel from a hosted runner, and secrets never leave your environment. **Isolation primitives.** Namespaces, RBAC, network policies, resource quotas and dedicated node pools let you fence test workloads off from each other and from anything they should not touch. A pod per job means nothing persists from one run to the next. **An ecosystem that speaks the same language.** Runners, pipelines, test orchestrators, load generators and environment platforms all install as Kubernetes resources and controllers. Test infrastructure is declared in Git and deployed like everything else, instead of living in a CI vendor’s settings page. None of this is free. A cluster has to be operated, test workloads compete with anything else on it unless they get their own nodes, and a misconfigured runner can expose more than a hosted one would. The comparison below is about where that trade pays off. #### Kubernetes vs CI runners: where should tests run? CI systems such as GitHub Actions, CircleCI and Jenkins decide when tests run. Where they run is a separate choice, and there are three common answers. The table compares them as of October 2026. Where tests run What the tests can reach Cost model Scale What you operate Best fit Hosted CI runners: GitHub-hosted runners, CircleCI cloud executors The public internet and whatever the job starts itself, usually mocks or Docker Compose Per runner minute, by machine size Capped by the plan's concurrent job limit Nothing Builds, unit tests, linting, contract tests Self-hosted runners on Kubernetes: [Actions Runner Controller](https://docs.github.com/en/actions/concepts/runners/actions-runner-controller), the [Jenkins Kubernetes plugin](https://plugins.jenkins.io/kubernetes/), the [GitLab Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes/), [CircleCI container runner](https://circleci.com/docs/guides/execution-runner/container-runner/) Your private network, as far as network policy allows Your nodes, no per-minute fee from GitHub As far as the cluster autoscales The cluster and the runner controller Any CI job that needs private access, more compute or lower cost at volume Tests as Kubernetes workloads next to the services: Testkube, Tekton, Argo Workflows, the k6 Operator, plain Kubernetes Jobs The services under test, through the same DNS, mesh and policies as production Your nodes As far as the cluster autoscales, many suites in parallel The cluster, plus the orchestrator Integration, end-to-end and load tests against real dependencies The second and third rows are often combined: CI still triggers the run and reports the result, a runner pod in the cluster starts it, and the suites run as jobs next to the services they test. Hosted CI runner Runner VMoutside your network Private servicesvia tunnel or not at all Runner pod in the cluster Runner podone per job, then removed Private servicesreachable, policy allowing Test job next to the services Test jobsame DNS, mesh, policies Services under testreal versions, in the cluster The closer a test runs to the services it exercises, the more of production's behavior it sees. ##### Cost Hosted runners bill by the minute. GitHub’s [published rates](https://docs.github.com/en/billing/reference/actions-runner-pricing) are $0.006 a minute for a 2-core Linux runner, $0.012 for 4 cores and $0.022 for 8 cores. A thousand 20-minute integration runs a day on 4-core runners is 20,000 minutes, about $240 a day or $7,200 a month, before retries. Self-hosted runners are [free to use with GitHub Actions](https://docs.github.com/en/actions/concepts/runners/self-hosted-runners), so the cost becomes the nodes underneath. A 32-vCPU node such as an m6a.8xlarge runs about $1.38 an hour on demand, per [AWS’s price list](https://aws.amazon.com/ec2/pricing/on-demand/), which works out to roughly a quarter of the hosted rate per 4-core minute if the node stays busy. Spot instances and scale-to-zero node pools push it lower. Idle nodes, oversized resource requests and the hours spent operating runners push it back up, so the saving is real at steady volume and thin for a team that runs a few hundred minutes a week. The larger cost is usually not the runner. A test that needs its own copy of every service to run against costs far more than the minutes it runs for, which is where the environment model further down matters. ##### Security A hosted runner has to reach your services somehow. Either they get a public endpoint for the test, or the job opens a tunnel into your network with credentials stored in the CI system. Running tests inside the cluster keeps traffic and secrets in your network. Self-hosted runners bring their own risk. GitHub’s hardening guidance says they [should almost never be used for public repositories](https://docs.github.com/en/actions/reference/security/secure-use), since any pull request can run code on them, and that they lack the clean-machine guarantee hosted runners have. On Kubernetes the mitigations are standard: ephemeral runners that take one job and are removed, a dedicated node pool for CI, network policies that allow test pods to reach only what they test, service accounts scoped to the namespaces a job needs, and no cluster-admin credentials in any runner. ##### Scale Hosted runners are capped by plan. GitHub [allows 20 concurrent jobs](https://docs.github.com/en/actions/reference/limits) on standard hosted runners on the Free plan, 40 on Pro, 60 on Team and 500 on Enterprise. A burst beyond that waits in a queue. On Kubernetes, Actions Runner Controller starts a runner pod for each queued job and the autoscaler adds nodes, so the limit is the cluster’s capacity and your budget. ##### How realistic the test is This is the difference that matters most and gets discussed least. A runner, hosted or self-hosted, only executes the test. What the test runs against decides whether a pass means anything. Most CI pipelines still start a few containers with Docker Compose or Testcontainers, or call mocks, because the runner has nowhere better to point. On Kubernetes the test can run against the real services, at their current versions, in the same environment they run in. The rest of this guide is mostly about making that practical for every change. #### The Kubernetes testing ecosystem The tools fall into six jobs. Most teams use one or two from each row they need. - **CI runners on Kubernetes.** [Actions Runner Controller](https://docs.github.com/en/actions/concepts/runners/actions-runner-controller) is GitHub’s operator for self-hosted runners: it watches for queued jobs, starts a runner pod per job and removes it after. The [Jenkins Kubernetes plugin](https://plugins.jenkins.io/kubernetes/) starts an agent pod for each build, the [GitLab Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes/) creates a pod for each CI job, and [CircleCI container runner](https://circleci.com/docs/guides/execution-runner/container-runner/) runs CircleCI jobs on your Kubernetes cluster. - **Pipelines and workflows defined in the cluster.** [Tekton](https://tekton.dev/) defines CI/CD pipelines as Kubernetes resources, and [Argo Workflows](https://argo-workflows.readthedocs.io/en/latest/) orchestrates parallel jobs on Kubernetes as a graph of steps, CI and test runs among them. - **Test orchestration.** Testkube runs existing Postman, k6, Playwright, pytest and other suites as workloads in the cluster, collects their results and reports on them. It runs against whatever environment you point it at and leaves creating that environment to another tool, which is where the next group comes in. - **Load generation.** The [k6 Operator](https://grafana.com/docs/k6/latest/testing-guides/running-distributed-tests/) spreads a k6 load test across many pods in the cluster from a single TestRun resource. - **Throwaway clusters.** [kind](https://kind.sigs.k8s.io/), [k3d](https://k3d.io/stable/) and [minikube](https://minikube.sigs.k8s.io/docs/) start a small cluster on a laptop or a CI runner, and the [Testcontainers K3s module](https://java.testcontainers.org/modules/k3s/) starts one from inside a test. They are the right tool for operators and controllers, and too small for a whole application. - **Environments per change.** Namespaces, virtual clusters with vCluster, full-copy platforms, and selective isolation with Signadot, compared in detail below. This is the layer that decides what a test actually runs against, and the one most pipelines are missing. The [Kubernetes testing tools comparison](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) covers the vendors in each group, with pricing and what each does and does not do. #### Which tests belong on Kubernetes Not every test should move. The rule of thumb: a test belongs on Kubernetes when it needs something only the cluster has, such as real services, private data stores, production configuration or more compute than a runner offers. Layer What it checks Where it should run Why Unit Logic inside one service CI runner, laptop, agent workspace Needs nothing but the code, and finishes in seconds Contract That a provider still honors what its consumers expect CI runner Needs the contracts, not the services Component and operator One service or controller against real Kubernetes APIs Throwaway cluster on a CI runner: kind, k3d Needs an API server, not the rest of the system Integration The changed service against current versions of what it calls Kubernetes, in an environment for the change Needs the real services, data stores and network path End-to-end A user journey across many services Kubernetes, in an environment for the change Needs every service in the path, running and current Performance and load Latency, throughput and limits under load Kubernetes, sized like production Needs production-shaped capacity and the k6 Operator or similar to generate load The top three rows are cheap and parallel by nature, and moving them off hosted runners saves money only at large volume. The bottom three are where Kubernetes testing earns its keep. How many tests to write at each layer is a strategy question, covered by the [testing pyramid for microservices](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/#the-testing-pyramid-for-microservices). CI runner unitno cluster contractno cluster Throwaway cluster componentkind, minikube, k3d operatorkind, minikube, k3d Kubernetes, real dependencies integrationenvironment per change end-to-endenvironment per change performancesized like production The further right a test runs, the more of the real system it needs, and the more it pays to run it on Kubernetes. #### Why Kubernetes is the natural place to test microservices A microservices application already runs on Kubernetes in production, so the cluster is the one place where a change can meet the current versions of everything it calls. The question is how to give every change that chance without the changes getting in each other’s way. ##### Why one shared test cluster breaks under parallel pull requests Most teams start with one shared pre-production cluster, usually called staging, where each change is deployed for its integration and end-to-end tests. It breaks in three ways once several teams, and now their agents, push changes at once. **Changes queue.** A shared cluster holds one version of each service, so two pull requests that change the checkout service cannot both be tested there at the same time. One waits, and the wait grows with every team. **Results stop meaning anything.** With several changes deployed together, a failing test could come from any of them. Engineers rerun until green and stop trusting red builds, which is how real failures reach production. **The environment drifts.** Half-finished deployments, manual fixes and leftover test data pile up until the shared cluster matches neither production nor anything else. PR #318checkout PR #319checkout agent taskpayments agent taskorders all deploy into Shared staging clusterone version of each service at a time 9 waiting One shared cluster holds one version of each service, so parallel changes either wait or overwrite each other. Deploy calendars, a lock in a chat channel, feature flags and a second or third staging cluster each help for a while, and none scales with the number of parallel changes. The [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers why those fixes fail. The answer is to keep one pre-production cluster and stop deploying every change into it directly. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Kubernetes test environments compared There are six common ways to give a change somewhere on Kubernetes to run its tests. They differ in what each test run gets, how fast it starts and what a hundred of them at once cost. The table compares them by approach, as of October 2026. Approach What each test run gets Dependencies Start time Cost with many parallel runs Best fit Local cluster: [kind](https://kind.sigs.k8s.io/), [minikube](https://minikube.sigs.k8s.io/docs/), [k3d](https://k3d.io/stable/) A small cluster on a laptop or CI runner Only what the test deploys, usually fakes for the rest Minutes, growing with each service deployed Free compute, but someone maintains every manifest and fake Operators, controllers, single-service component tests One shared staging cluster A turn on one environment Real and current, when nobody has broken them Already running, but changes queue Flat, paid for in waiting and reruns A few teams, release rehearsal Namespace per pull request A copy of every service in its own namespace Copies, seeded with test data As long as the slowest service takes to start Grows with services times open pull requests A handful of services, low change volume Virtual cluster per pull request: [vCluster](https://www.vcluster.com/docs/vcluster/) Its own control plane, with workloads on the host's nodes Copies, as with a namespace Namespace start plus a control plane Grows like a namespace copy Changes to CRDs, operators or cluster configuration Full copy per pull request: a cluster per change, or platforms such as [Release](https://www.signadot.com/comparison/release/) and Bunnyshell A complete, isolated copy of the application Copies, often from snapshots Longest of all Highest, and paid even for services nobody is testing Small applications that need total isolation Selective isolation on a shared cluster: [Signadot](https://www.signadot.com/solutions/kubernetes-test-environments/), and in narrower forms [Okteto Divert](https://www.okteto.com/docs/core/divert) and [mirrord preview environments](https://metalbear.com/mirrord/docs/use-cases/preview-environments) Only the services the change touches, plugged into the shared environment Real and current, shared, with a database or topic of its own when a change needs one Seconds Grows with the changed services only Many services, many parallel pull requests and agent runs Local cluster Service under testin kind or minikube Dependenciesfaked or left out Namespace or vCluster per PR Every servicecopied per PR Databases, queuescopied and seeded Selective isolation Changed servicedeployed per PR Everything elseshared and routed, not copied What one pull request costs under each approach. The more it copies, the longer it takes to start and the more a hundred of them cost. **Local clusters** are the right tool for Kubernetes-facing code. kind runs a cluster in Docker containers and its own docs say it was built mainly for testing Kubernetes itself, while minikube and k3d target local development. They stop working as a test environment once the application needs more services than a laptop or a runner can hold. The [guide to lightweight Kubernetes environments](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) compares them. **Namespaces and virtual clusters per pull request** give real isolation, and Kubernetes namespaces were designed for exactly this kind of partitioning. Either way, every environment holds a complete copy of the application, so cost and start time scale with the system’s size times the number of open changes. [Namespace vs cluster for test environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) goes through the tradeoffs. **Full copies** offer the simplest mental model and the highest bill. Brex found that [99.8 percent of what a duplicated environment cost was spent on services no one was testing](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/), and moving off duplicated environments saves it about $2 million a year. **Selective isolation** deploys only what a change touches and plugs it into the real, shared environment around it. Signadot is built on this model. Each change gets an ephemeral environment on the existing cluster: the changed workloads are deployed next to the stable versions, and a routing key carried in request headers, in OpenTelemetry `baggage`, decides which requests reach them. Requests with the change’s key go to its version of the service, and everything else goes to the shared one. Hundreds of these environments share one cluster without seeing each other’s work, and each starts in seconds, since the rest of the system is already running. Routing uses the service mesh already in the cluster, Istio natively, the Kubernetes Gateway API, or a lightweight sidecar with Linkerd or no mesh, and services need to pass the routing header along, something OpenTelemetry instrumentation handles out of the box for most languages. [What a Kubernetes sandbox is](https://www.signadot.com/kubernetes-sandbox/) covers how the routing works. ##### Data, queues and workflows on a shared cluster Sharing a cluster raises the question of shared state. On Kubernetes the answers are cluster-native. Tests share a database with each run’s data kept apart, or a resource plugin creates a temporary database, schema or branch for a change that alters the schema and deletes it when the environment ends. Because a plugin step is just a script run in a pod, it can also create things outside the cluster, such as a managed queue, a storage bucket or a VM. Messages through a shared Kafka, RabbitMQ, Pub/Sub, SQS or NATS broker carry the routing key so only the right consumers process them, and Temporal workers share a task queue the same way. Why synchronous calls, asynchronous messages and durable workflows each make microservices hard to test is covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/#testing-microservices-with-real-databases-queues-and-workflows), and the isolation patterns in the [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems). #### How to run integration and end-to-end tests on Kubernetes from CI Put together, the pattern most teams converge on keeps the CI system as the trigger and the scoreboard, makes Kubernetes the place tests run, and gives each change its own environment to run them against. The tests are your own, run by whatever already runs them: 1. A pull request opens. CI builds container images for whatever it touched, on hosted runners or runner pods. 2. CI creates an isolated environment for the change on the shared cluster, holding the new images and sharing everything else. 3. The integration and end-to-end suites start as jobs inside the cluster, aimed at that environment by its URL or routing key. 4. Results, logs and artifacts flow back to the pull request. 5. The environment is deleted on merge, on close, or after a time limit. Kubernetes integration testing done this way catches the failures mocks hide: a field another team renamed, a timeout that only appears behind a sidecar, a consumer that rejects a new message shape. Kubernetes end-to-end testing drives a whole user journey through the frontend or API gateway, and the same environment serves API testing, front end testing on Kubernetes and acceptance testing by people while the suite runs. Three details decide whether the results can be trusted: - **Run the suite inside the cluster.** A test pod resolves the same cluster DNS names and passes through the same network policies and mesh as the service, and does not depend on a tunnel from a runner staying up. - **Test against current versions.** A long-lived integration namespace drifts. The environment should point at whatever is currently deployed for everything except the change. - **Keep changes apart.** Two pull requests that deploy the same service into one environment overwrite each other, and both report on code that is not theirs. Shared state is also the usual root cause behind flaky suites, as [how to fix flaky tests](https://www.signadot.com/blog/how-to-fix-flaky-tests/) explains, and behind integration tests that [pass while staging breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/). Kubernetes performance testing follows the same pattern on a schedule rather than per change: the k6 Operator or a JMeter or Gatling job runs against a pre-production cluster sized like production, with a short latency check per pull request to catch the worst regressions early. Running the service under change on a laptop or in an agent’s workspace, connected to the same cluster, extends the pattern to before the push. [Shift-left testing in Kubernetes](https://www.signadot.com/blog/shift-left-testing-in-kubernetes/) covers that practice, and the [local development guide](https://www.signadot.com/guide-to-local-development-kubernetes/) weighs the tools. #### What coding agents change about Kubernetes testing Coding agents affect Kubernetes testing twice over: they raise the number of changes that need test infrastructure, and they use that infrastructure themselves. The volume is already measurable. CircleCI’s 2026 data puts the [year-over-year throughput gain at 59 percent](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/), and the gain is lopsided: the top 5 percent of teams came close to doubling throughput, and the bottom quarter gained nothing. About three in ten merge attempts fail, and the median time to recover has climbed 13 percent to 72 minutes. The teams that absorb a 10x surge in pull requests are the ones whose test infrastructure scales with it. Fifty engineers opening two or three pull requests a day each produce 100 to 150, and pipelines were sized for that. [With agents, the same team can see 1,000](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/). On hosted runners that means a much larger minutes bill and a queue behind the plan’s concurrency cap. On Kubernetes it means more pods. Agents also test differently. Claude Code, Cursor, Codex and Copilot work in a build, test, fix loop: write a change, run the tests, read the failure, try again, often ten times for one task. Unit tests run fine in the agent’s workspace. Integration and end-to-end suites, the ones that would catch a broken contract, need a cluster with the real services, on every iteration, in seconds, with no person in the loop to book a slot. That moves Kubernetes testing ahead of review. If the agent can run the integration and end-to-end suites against real dependencies before it opens a pull request, the reviewer reads code that already works against the real system. If it cannot, the reviewer becomes the test environment, and review queues grow with every agent added. Isolation matters more with agents than with people. Ten agents testing in one shared environment produce failures nobody can attribute, and batching their changes into one run means one bad change fails all ten. An environment per agent task keeps every failure attached to the change that caused it. Cost and cleanup need guardrails. Environments for agent tasks should start in seconds, carry a time limit, and delete themselves, so a stalled agent does not leave services running for a week. Deploying only the changed services is what keeps a thousand runs a day affordable. Agents reach these environments the same way developers do. Signadot’s CLI, its [MCP server](https://www.signadot.com/docs/integrations/mcp) and its [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills) let an agent create an ephemeral environment, run tests in it and read the results without a person in between. [Validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/) walks through the whole agent workflow. #### Kubernetes testing best practices checklist 1. Do unit tests, contract tests and builds stay on CI runners, where they are cheapest? 2. Do integration, end-to-end and load tests run on Kubernetes, inside your network, rather than against mocks on a runner? 3. Do CI runners on Kubernetes take one job each and get removed, on a dedicated node pool? 4. Are self-hosted runners kept away from public repositories? 5. Do network policies and scoped service accounts limit what test pods can reach? 6. Does runner and test capacity autoscale, including down to zero when idle? 7. Does every pull request, and every agent task, get its own isolated environment on the cluster? 8. Do those environments start in seconds and point at current versions of everything except the change? 9. Do test suites run as jobs inside the cluster, in parallel, with results kept in one place? 10. Does a failing test point at one change, never a batch of several? 11. Do test environments carry a time limit and delete themselves when a change merges or stalls? 12. Do performance tests run in an environment sized and configured like production? #### Where to go next If you are picking tools, the [Kubernetes testing tools comparison](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) sorts them into test automation and environment platforms. If you want to understand what makes distributed systems hard to test in the first place, read the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). If shared staging is your bottleneck, begin with the [complete guide to staging environments](https://www.signadot.com/staging-environments/). If you want every pull request and coding agent to get its own Kubernetes test environment on the cluster you already run, with suites running in the cluster next to the services, see [Kubernetes test environments with Signadot](https://www.signadot.com/solutions/kubernetes-test-environments/). #### Related reading - [Best Microservices Testing Tools for Kubernetes in 2026](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) - [Guide to Shift-left Test Approach in Kubernetes](https://www.signadot.com/blog/shift-left-testing-in-kubernetes/) - [Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) - [The Ultimate Guide to Lightweight Kubernetes Environments](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) - [What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) - [Integration tests pass with mocks but staging still breaks: how to catch it earlier](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [How to Fix Flaky Tests](https://www.signadot.com/blog/how-to-fix-flaky-tests/) - [2026 CircleCI Report: The Agent Validation Gap Is Here](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) - [How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [What Is a Kubernetes Sandbox?](https://www.signadot.com/kubernetes-sandbox/) - [Local Development on Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-local-development-kubernetes/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) #### Frequently asked questions What is Kubernetes testing? Kubernetes testing means using Kubernetes as the platform your tests run on: test runners as pods, test environments as workloads in a cluster, and the services under test running next to them. It moves integration, end-to-end and performance tests off CI runners and into infrastructure shaped like production. The phrase is also used for validating a cluster itself, which is a separate job. Should you run tests on Kubernetes or on CI runners like GitHub Actions? Use both, for different tests. Hosted CI runners suit unit tests, linting and builds: they need nothing private and finish in minutes. Integration, end-to-end and load tests belong on Kubernetes, where they can reach private services and databases, scale with the cluster, and run against the same images and configuration as production. Most teams keep CI as the trigger and Kubernetes as the place tests run. Is it cheaper to run CI tests on Kubernetes than on hosted runners? Usually, at volume. GitHub charges $0.012 a minute for a 4-core hosted Linux runner, while self-hosted runners cost nothing from GitHub and you pay only for the nodes. On a well-packed Kubernetes node priced at AWS on-demand rates, the same compute costs about a quarter as much. The saving shrinks if nodes sit idle or your team spends much time operating the runners. Is it safe to run CI jobs on self-hosted runners in a Kubernetes cluster? For private repositories, with care. GitHub advises that self-hosted runners should almost never serve public repositories, because anyone can open a pull request that runs code on them. Use ephemeral runners that handle one job and are removed, a dedicated node pool, network policies that limit what test pods can reach, and service accounts with only the permissions the tests need. What is Actions Runner Controller? Actions Runner Controller, or ARC, is GitHub's Kubernetes operator for running self-hosted GitHub Actions runners in your own cluster. It watches for queued jobs, starts a runner pod for each one, and removes the pod when the job finishes, so runner capacity follows demand. It is the usual way to move Kubernetes testing workloads off GitHub-hosted runners without leaving GitHub Actions. How do you run integration tests in Kubernetes? Deploy the changed service into a cluster next to current versions of the services it calls, then run the suite as a pod in the same cluster so it resolves the same DNS names and passes through the same network policies and mesh. Give each change its own isolated environment, or two pull requests testing the same service at once will overwrite each other's deployments. How do you run end-to-end tests on Kubernetes in CI? Let CI trigger and Kubernetes run. The pipeline builds images for what the pull request touched, creates an isolated environment for it in the cluster, starts the Playwright, Cypress or API suite as a job inside the cluster, posts the outcome to the pull request and deletes the environment. The run is as fast as the environment is to create, so avoid copying every service. Do you need a separate test cluster for every pull request? No. A cluster or namespace per pull request copies every service, so cost and start time grow with the size of the system. Most teams run one shared pre-production cluster and give each pull request an isolated environment on it, holding only the services that changed and sharing the rest. A separate cluster is worth it only when the change is to the cluster itself. How do coding agents test their changes on Kubernetes? An agent such as Claude Code, Cursor, Codex or Copilot calls an environment platform's CLI or MCP server, such as Signadot's, to spin up an isolated environment holding only the services it changed, runs integration and end-to-end tests against the real services in the cluster, reads the failures and fixes the code. Each attempt in that loop needs an environment in seconds, without waiting for a shared one. How do you stop AI-generated pull requests from overwhelming test infrastructure? Run their tests on Kubernetes, where capacity scales with the cluster rather than a runner plan's concurrency cap, and give every agent task its own short-lived environment instead of a turn on a shared one. Failures then point at one change, environments delete themselves after a time limit, and cost grows with the services that changed rather than with the number of agents. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Contract Testing: The Complete Guide Source: https://www.signadot.com/contract-testing/ Summary: What a contract test verifies, consumer-driven versus provider-driven contracts, how contract testing differs from unit and integration testing, the tool landscape from Pact to SmartTests, and where contract tests still miss. ### Contract Testing: The Complete Guide What a contract test proves and misses, consumer-driven versus provider-driven contracts, the tools from Pact to SmartTests, and what changes when coding agents write both sides of an API. Published September 16, 2026 Reading time 21 min Contract testing verifies that a consumer service and a provider service agree on the shape of the API call between them, without running either service’s full dependency graph. One side writes a machine-checkable description of the interaction, and the other side runs that description against its real code. The practice exists because integration testing every pair of services does not scale to how independently microservices teams ship. If every team had to run a full suite against every other team’s current code before deploying, the coordination would cost more than the checks are worth. Contract testing narrows the check to the interface itself, so each side verifies compatibility without running the other. It is one layer of a wider practice, and the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) covers the rest. That narrowness is the whole trade. A contract test proves two services agree on the shape of an interaction. It does not prove the system behaves correctly when that interaction happens against the current, deployed version of everything else. Coding agents press on that gap from both ends, changing consumer and provider in the same session and arriving as a class of API consumer no contract describes. At Laurel, [agent-authored pull requests rose from 55 percent to 76 percent in three months](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) while change failure rate fell from 1.1 percent to 0.2 percent, which took validation that scaled with the volume. This guide covers what a contract test verifies, how contracts get written, how the practice compares to unit and integration testing, which tools fit which team, where contract tests stop covering the system, and how to decide whether yours needs them. Why testing has to keep pace with coding agents is the subject of the [complete guide to software factories](https://www.signadot.com/software-factory/). #### What is contract testing, and what is an API contract? A contract test checks one thing: whether a consumer’s understanding of a provider’s API matches what the provider returns. It runs at the boundary between exactly two services, without booting the rest of the system, which is what makes it cheap enough to run on every change. The contract is a machine-checkable description of one interaction, usually an HTTP request and response pair, or a message and the shape it takes on a queue. It covers the request method and path, required and optional fields, field types, and the status codes and error shapes the provider can return. Anything the two sides never wrote down sits outside it, which is where most of the interesting failures live. A contract test is not a substitute for running the consumer and provider together. It checks both sides against a shared description, not against each other, so two services can each pass their own contract tests and still fail when they talk. ##### A REST API contract testing example A checkout service calls an orders service for `GET /orders/{id}` and reads three fields off the response: `id`, `status`, and `total_cents`. An API contract test for that pair records the request and those three fields, with their types. The orders team later renames `total_cents` to `total`, which its own specification permits because the field was never marked stable. Running checkout’s contract against the new orders code fails at that field, by name, before the change merges. Nothing else about the orders service has to be running for the check to work. Now change the example. The orders team keeps `total_cents` but starts returning it as a string instead of an integer, on refunded orders only. If the contract recorded a type, the check catches it. If the contract recorded only that the field exists, every test passes and checkout breaks on the first refund. That second case is the one worth holding on to. A contract is exactly as good as what someone thought to put in it, which is the subject of a later section. #### Contract testing vs integration testing vs unit testing These three check different scopes, and none is a superset of another. Passing a cheaper one does not imply the more expensive one would pass. A unit test runs one function or one small unit inside a single service, with its dependencies mocked. It is the fastest of the three and says nothing about how the service behaves when it talks to anything else. A contract test runs at the boundary between two services. It confirms that a consumer’s expectations of a provider’s interface match what the provider returns, with neither side’s other dependencies present. An integration test runs multiple real services together along an actual call chain. It is the slowest to set up and run, and the only one of the three that catches a failure appearing when several services interact at once: an ordering problem, a timeout under load, or a change that is correct at every boundary and wrong in combination. unitweborderspaymentscontractintegration Each level tests a different span of the same call chain. Shaded services and bold arrows are what the test exercises. unitweborderspaymentscontractintegration Each level tests a different span of the same call chain. Shaded services and bold arrows are what the test exercises. Test type Scope Dependencies Speed Catches Misses Unit One function or unit inside one service None, mocked or stubbed Fastest Logic bugs inside a service Anything outside the service Contract The boundary between exactly two services None, the contract stands in Fast Interface mismatches between two services Behavior the contract does not describe Integration Multiple real services along a call chain Real or near-real, running Slowest Multi-service and end-to-end behavior Little structurally, but often too slow for every change ##### Contract testing vs integration testing This is the comparison people reach for most, and the two are not tiers of one ladder. Contract testing checks the interface between two services in isolation. Integration testing runs the services together and checks what happens. The first is cheap enough for every commit. The second is the only one that sees behavior, because behavior is what emerges when two systems talk to each other for real. Run them at different points rather than choosing between them. Contracts belong before merge, where a fast and specific failure is worth more than a thorough one. Integration tests belong against live dependencies straight after, where the cost of running the real thing buys coverage no contract can give. [Four tactical models for shifting testing left](https://www.signadot.com/blog/how-to-shift-testing-left-4-tactical-models/) walks through where each check lands in a pipeline. #### Consumer-driven vs provider-driven contracts Contracts originate on one side of the interaction or the other, and which side writes them decides who is protected. In a provider-driven contract, the provider publishes a specification, commonly an [OpenAPI](https://spec.openapis.org/) document, describing every endpoint it offers. Consumers build against that specification, and the provider tests its own service against what it published. One document and one owner keeps the process cheap to run. It also protects the provider’s intent rather than how consumers use the API. A provider can satisfy its own specification and still break a consumer that depended on a field the spec marked optional but that consumer treated as always present. Consumer-driven contract testing reverses the order. Each consumer records the exact requests it sends and the responses it needs, based on how it calls the provider rather than on what the documentation permits. That recording becomes the contract. The provider then runs every consumer’s contract against its own code, usually on every change, and a failure means a named consumer would break if the provider shipped as is. consumercontractproviderwritesverifies In a consumer-driven contract, the consumer states the expectation and the provider proves it still holds. consumerwritescontractverifiesprovider In a consumer-driven contract, the consumer states the expectation and the provider proves it still holds. Consumer-driven contracts catch a narrower and more relevant set of breaks: not everything the provider could theoretically change, but everything a real consumer depends on today. The cost is coordination. Every consumer has to write and maintain its contract, and a consumer that never writes one is invisible to verification. A provider with forty consumers and contracts from twelve of them is checked against twelve. The coordination problem is well documented. A study of microservice API evolution built on [17 interviews across 11 companies](https://arxiv.org/abs/2311.08175) found the recurring challenges were change impact analysis, ineffective communication of changes, and consumers staying on outdated versions, which the authors trace to tight organizational coupling and consumer lock-in. Which model a team adopts usually follows from who owns the pain. If provider teams keep breaking consumers without warning, consumer-driven contracts put the check where the risk is. If the API is public or has too many consumers to coordinate individually, a provider-driven specification is often the only tractable starting point. #### Contract testing tools: Pact, Specmatic, Karate, Keploy and SmartTests Contract testing software falls into three shapes: tools built on consumer-recorded interactions, tools generated from an API specification, and tools that compare live behavior instead of a written contract. Five come up most often, and no single contract testing tool is a drop-in replacement for another. The right starting point depends on what your team already maintains and how much manual upkeep it will absorb. **[Pact](https://pact.io)** is the category standard. It implements consumer-driven contract testing directly, with language support across the JVM, JavaScript and TypeScript, .NET, Go, Python, Ruby and PHP, and a broker, either the self-hosted Pact Broker or the hosted [Pactflow](https://pactflow.io), that stores contracts and answers whether a provider is safe to deploy against every consumer’s current contract. Its strength is explicit control: a team can see exactly what is asserted and why a check failed. Its cost is that same explicitness, since every consumer maintains a contract and every provider runs verification on every change. **[Specmatic](https://specmatic.io)** takes a specification-driven route instead of a consumer-recorded one. It derives a consumer-side stub and a provider-side verification test from one OpenAPI or AsyncAPI document, so the specification becomes the single artifact both sides check against. This suits teams already maintaining an API specification as documentation, because contract testing turns into a byproduct of keeping that specification accurate. It inherits the provider-driven trade-off, describing what the provider intends to offer rather than what each consumer uses. **[Karate](https://karatelabs.io)** is a broader API test automation framework that covers contract-style assertions, written in a Gherkin-like syntax non-programmers can read. Teams already running Karate for functional or performance testing can express contract checks in the same framework instead of adopting a dedicated tool, trading away the contract-specific machinery like a broker and a compatibility matrix. **[Keploy](https://keploy.io)** generates tests, including contract-style assertions, by recording real traffic against a service and replaying it. It bootstraps coverage without anyone hand-writing tests, which helps services with weak existing suites. The trade-off is that a recording captures behavior at the moment it was taken, and nothing re-checks it against the provider’s current code the way a verification step does. **Signadot SmartTests** sits in a different category from the other four. Rather than a contract file either side maintains, it runs the changed service against real deployed dependencies and diffs its behavior against the stable shared version. There is no contract to write or keep current, and the comparison happens against what is running. The trade-off runs the opposite way from Pact: it needs a live cluster to compare against, so it does not work offline against a stored contract, and a team that wants a portable specification gets a behavioral diff instead. Tool Contract model Who writes it Maintenance Runs offline Best fit Pact Consumer-recorded interactions Every consumer, verified by the provider High, grows with consumer count Yes Many teams on either side of an interface that need an auditable, versioned contract Specmatic OpenAPI or AsyncAPI specification The provider, once Medium, tied to spec accuracy Yes Teams whose API specification is already the source of truth Karate Assertions in a test framework Whoever owns the test suite Medium Yes Teams already using Karate for API test automation Keploy Recorded traffic replayed Generated from live calls Low to write, ages silently Yes Services with weak existing coverage that need a starting point Signadot SmartTests Behavioral diff against the stable shared version Nobody, generated per change Close to none No, needs a live cluster Kubernetes teams that want breaking changes caught without owning contract files ##### API contract testing tools for OpenAPI-first teams If your API specification is already accurate and enforced, most of the contract testing problem is solved and the remaining question is which direction to generate from it. Specmatic generates both sides from the document. Pact can be used alongside a specification, though its contracts still originate from consumers rather than from the spec. The failure mode to watch for here is a specification that drifts from the running service. A generated check is only as current as the document it came from, and a stale OpenAPI file produces tests that pass against an API nobody is shipping. [Relevancy-weighted diffs for REST API testing](https://www.signadot.com/blog/rest-api-testing-using-relevancy-weighted-diffs/) covers comparing against live responses instead. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Contract testing when AI coding agents write both sides Agents break a quiet assumption behind every contract: that a person wrote both sides and knows what changed. When an agent proposes changes to the consumer and the provider in the same afternoon, the contract between them is the one artifact nobody updated. That makes staleness the dominant failure mode. A contract is written and maintained by hand, so under agent volume it stops being the safeguard and becomes the reason a correct change fails its check. Teams respond by relaxing the check, which is how a contract suite quietly stops meaning anything. The consumer side is shifting too. Agents call APIs directly now, and most APIs were not designed with that in mind. Postman’s [2025 State of the API report](https://www.postman.com/state-of-api/2025/) found that only 24 percent of developers actively design APIs with AI agents in mind, while 60 percent design primarily for humans and 16 percent have not considered agents as consumers at all. A consumer-driven contract protects the consumers that wrote one, and an agent calling your API has written nothing. Set against that, contract testing holds a real advantage over every other check an agent can run. A failed contract test names the field, the type or the status code that moved. That is machine-actionable in a way a failed end-to-end test is not, so an agent can read the result and correct the change without a person translating it first. This is how Claude Code, Cursor, Codex and Copilot work: act, read the result, act again. What none of them can use is a check on somebody else’s schedule. Testing AI-generated code against the interface it changed has to happen on every attempt, not nightly and not once per feature, and it has to finish before the change reaches a human. Otherwise review becomes the only real verification step, for work arriving faster than anyone can read it. [Validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/) covers how that check gets wired in. The last requirement is the interface. An agent needs to invoke verification from a command line or an MCP server, not a console, and it needs the answer back as structured output rather than a dashboard. [Why the MCP server is now a critical microservice](https://www.signadot.com/blog/why-the-mcp-server-is-now-a-critical-microservice/) covers what that path has to look like once agents depend on it. #### Where contract testing still lets bugs through Contract tests can pass at every layer while production breaks, because a contract only checks what it was written to check. Four gaps show up often enough to name. **The contract is narrower than the real interaction.** It covers the fields both sides agreed matter. An edge case nobody wrote down, a rare error path, or the order fields arrive in is real and can break a consumer without failing a single check. **Verification lags the provider’s actual code.** Consumer-driven contract testing protects a consumer only if the provider runs verification before deploying, on every change. A provider that skips it on a hotfix, or verifies against a broker holding contracts a consumer forgot to republish, ships a break no contract test saw. **Contracts age faster than anyone re-reads them.** A contract written a year ago encodes what the interaction looked like then. Without something forcing a re-check against current behavior, it can describe an interaction that no longer exists, and every test built on it passes for the wrong reason. **Real infrastructure sits outside the contract.** Network timeouts, retries, authentication and deployed-only configuration are not things a contract, or a mock derived from one, exercises. A contract test can pass at the exact moment the same request would fail in production. This is the failure mode described in [why integration tests pass with mocks but staging breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/). A contract, like a mock, is a snapshot of an assumption, and writing that assumption down more formally does not protect it from the dependency changing afterwards. Contract testing narrows the search for interface breaks. It does not replace a check against what is deployed, which [shift-left testing for microservices](https://www.signadot.com/articles/shift-left-testing/) moves to the pull request. #### Is contract testing worth the maintenance? You need the check. You do not necessarily need a written contract to get it. Every team needs to know that a change to one side of an interface will not break the other, and a formal contract is one way to find that out rather than the only one. The risk it addresses is real but narrower than it sounds. A large-scale study of [API breaking changes across 317 Java libraries](https://homepages.dcc.ufmg.br/~mtov/pub/2017-saner-breaking-apis.pdf), 9,000 releases and 260,000 client applications found that 14.78 percent of API changes broke compatibility, while 2.54 percent of clients were affected. Those are library APIs rather than REST services between teams, so read the pair as a shape and not a forecast. Breaks are common, and most of them land on nobody. The problem is that nobody knows in advance which ones will. For a single team owning both the consumer and the provider, skip it. Both sides sit in the same codebase, or close enough that a breaking change surfaces in review or a shared suite the same day. A contract here adds upkeep without adding signal. For a team whose OpenAPI specification is accurate and enforced in CI, much of the value is already captured. Generating checks from that document costs little. Adding a second, consumer-recorded contract layer on top pays off only when consumers keep depending on behavior the specification does not pin down. The argument against contract testing at scale is an argument against its maintenance cost, and it lands. Contracts go stale, and teams abandon them for that reason rather than because the idea is wrong. The answer is not to drop the check but to move it somewhere that needs no hand-maintained files. [A new approach to contract testing](https://www.signadot.com/blog/let-systems-do-the-work-a-new-approach-to-contract-testing/) makes that case in full. #### How to choose: four questions that decide it Whether contract testing earns its place, and which tool fits, depends on four things about your team more than on any feature comparison. ##### Team size A single team owning both sides does not need a formal contract. Contract testing earns its keep once the consumer and provider belong to different teams shipping on their own schedules, so neither side can rely on the other reading its pull requests. The more teams sit on either side of an interface, the more a machine-checked contract replaces coordination that has stopped scaling. ##### Service count With a handful of services, the number of consumer-provider pairs is small enough to track in a shared channel, and the overhead of maintaining contracts can exceed what it saves. Past a few dozen services the number of interface pairs grows faster than the service count itself, and nobody holds that graph in their head. That is where automatic checks start catching breaks no engineer would have spotted by inspection. ##### Release velocity Teams releasing a few times a week can often afford a manual check on a risky interface change. Teams releasing continuously, and teams where agents attempt many changes an hour, cannot. A check that runs on every commit is the only version that keeps up, and it is also the only one an agent can act on directly, because it names the field or status code that moved. ##### Existing tooling What a team already maintains usually decides the starting tool. An accurate OpenAPI specification makes Specmatic close to free. An existing Karate suite absorbs contract assertions without a second tool. A team with the engineering time to own contracts explicitly, and a real need for that control, is well served by Pact. A team on Kubernetes that would rather test against real dependencies than maintain any contract file is the case the next section covers. #### Contract testing best practices checklist Most contract testing best practices reduce to twelve questions worth answering yes to. Anything you cannot is where breaks get through. - **Types, not just names.** Does every contract record field types and formats rather than only that a field exists? - **Every consumer publishes.** Does each consumer publish a contract, so verification is not silently checking a subset? - **Verification is required.** Is provider verification a blocking check rather than an optional job someone can skip on a hotfix? - **Contracts are versioned with code.** Does a contract travel in the same commit as the change that altered it? - **Failures name the field.** Does a failed check identify the exact field or status code, so a developer or an agent can act without interpreting it? - **Both sides re-verify on change.** Does a provider change trigger verification against current consumer contracts, and not only the reverse? - **Staleness is visible.** Can you tell how long it has been since a contract was verified against the provider’s live behavior? - **Error paths are covered.** Do contracts describe the error shapes and status codes, not only the happy path? - **Async interfaces are included.** Are message and event schemas covered, or does the practice stop at HTTP? - **The check runs per change.** Does every change get its own verification run rather than being batched with others? - **Agents can invoke it.** Can an agent run the check through a command line or MCP interface without a person? - **Something checks real dependencies.** Is there a layer beyond contracts that exercises the change against what is deployed? That last item is the one teams most often miss, and [shadow testing for APIs](https://www.signadot.com/blog/shadow-testing-superpowers-four-ways-to-bulletproof-apis/) covers one way to cover it. #### How SmartTests approaches contract testing SmartTests answers the maintenance side of contract testing rather than replacing the category. A Sandbox deploys only the changed version of a service into a shared Kubernetes cluster, alongside the stable shared copies of everything it depends on. Nothing is duplicated and nothing is mocked. Instead of a consumer or provider hand-writing a contract, the same requests go to the changed version and to the stable shared version, and the two sets of responses are diffed. The diff is built to separate a change that matters, a removed field or a status code that moved from 200 to 500, from one that does not, like a timestamp or a generated identifier. What comes back is a short list of what changed, produced the moment a change is proposed, with nothing to write beforehand and nothing to update as the API evolves. The trade-off is the one in the tooling table. SmartTests gives up the portable, auditable contract file that Pact produces, in exchange for close to no ongoing maintenance and a check that runs against currently deployed dependencies. A team that wants a versioned specification it can inspect independently of any platform should reach for Pact. A team that wants breaking changes caught on every change without anyone owning a contract file is the case this was built for. What each catches, and how to run them together, is on the [Pact vs Signadot Smart Tests](https://www.signadot.com/comparison/pact/) page. [SmartTests](https://www.signadot.com/product/smart-tests/) runs alongside Sandboxes and Jobs on the Kubernetes cluster you already operate. #### Where to go next If you are deciding whether to adopt contract testing at all, the four questions above are the shortest route to an answer, and [AI-powered contract testing for microservices](https://www.signadot.com/articles/ai-powered-contract-testing-for-microservices-excellence/) covers what changes once agents write most of the API changes. If you already run Pact and the maintenance is the problem, [SmartTests compared with Pact](https://www.signadot.com/articles/signadot-smarttests-vs-pact-contract-testing/) is the direct comparison. If contract testing is one piece of a wider testing strategy you are still assembling, start from the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### Related reading - [The Future of API Validation: A Deep Dive into AI-Powered Contract Testing](https://www.signadot.com/the-future-of-api-validation-a-deep-dive-into-ai-powered-contract-testing/) - [Stop Breaking Your Microservices with SmartTests: AI Powered Contract Testing](https://www.signadot.com/articles/stop-breaking-your-microservices-with-smarttests-ai-powered-contract-testing/) - [AI-Driven Contract Testing: Signadot SmartTest vs. Pact](https://www.signadot.com/blog/ai-driven-contract-testing-signadot-smarttests-superior-approach-vs-pact/) - [What's the real ROI of writing Mocks?](https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing/) - [Best Microservices Testing Tools for Kubernetes in 2026](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) #### Frequently asked questions What is contract testing in software testing? Contract testing verifies that a consumer service and a provider service agree on the shape of the call between them, without running either service's full dependency graph. One side writes a machine-checkable description of the interaction, and the other side runs that description against its real code. It catches interface mismatches at the boundary between two services rather than across a whole system. What is the difference between contract testing and integration testing? Contract testing checks the interface between two services in isolation, without deploying either side's dependencies. Integration testing runs multiple real services together and checks what the call chain does. Contract testing is faster and narrower, and it can pass while two services would still behave incorrectly together. Integration testing catches that, at a higher cost per run. Most teams past a handful of services run both, at different points in the pipeline. What is consumer-driven contract testing? In consumer-driven contract testing, the consumer records the exact requests it sends and the responses it needs, based on how it calls the provider rather than what the provider's documentation allows. That recording becomes the contract. The provider then runs every consumer's contract against its own code on each change, and a failure names the specific consumer that would break if the provider shipped as is. What are the best contract testing tools? Pact is the most established, with deep language support and a broker for sharing contracts between teams. Specmatic derives contracts from OpenAPI documents instead of recorded interactions. Karate folds contract assertions into a broader API testing framework. Keploy generates tests from recorded traffic. Signadot SmartTests skips the contract file and diffs a changed service's behavior against the stable shared version. The right one depends on what your team already maintains. What are the best practices for API contract testing? Record types and not only field names, version contracts alongside the code that produces them, and make provider verification a required check rather than an optional job. Publish a contract from every consumer, because an unpublished consumer is invisible to verification. Re-verify against the provider's current code on every change, not on a schedule. Pair contracts with a check against real deployed dependencies, since a contract only describes what someone wrote down. Can contract tests pass and production still break? Yes. A contract test verifies only what the contract describes, usually request shape, response fields, and status codes. Both sides can satisfy the contract and still fail if it never captured a behavior that later changed, if a mock built from the contract drifted from what the provider deploys, or if the failure depends on infrastructure the contract never exercises, such as timeouts, retries, or deployed-only configuration. Do you need contract testing if you already do integration testing? Contract testing and integration testing catch different failures, so one does not make the other unnecessary. Integration tests need real dependencies running together, which makes them slower and more expensive to run on every commit. Contract tests run in isolation and check the interface on every change at a fraction of that cost. Teams that scale past a handful of services usually keep both rather than treating them as substitutes. Is contract testing overkill for a small number of services? Often, yes. When one team owns both the consumer and the provider, a breaking change surfaces in code review or a shared test suite the same day it is written, and a formal contract adds upkeep without adding signal. Contract testing starts paying once the two sides are owned by different teams shipping on their own schedules, so neither side can rely on the other reading its pull requests. How do AI coding agents change contract testing? Agents change both the volume of API changes and the cost of maintaining contracts by hand. They propose changes on both sides of an interface faster than anyone updates the contract between them, so a stale contract becomes the bottleneck rather than the safeguard. Agents also need the check on every attempt in a build-test-fix loop, not once per feature, which rules out any verification step that waits for a person. Can a coding agent check its own API change before opening a pull request? It can, provided the check runs without a person in the loop and returns a specific failure. A contract test qualifies, because it names the field or status code that changed in a form an agent can read and act on. Running the changed service against real deployed dependencies catches the behavior a contract never described. Both paths need a command line or MCP interface rather than a console someone clicks. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Best Microservices Testing Tools for Kubernetes in 2026 Source: https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/ Summary: A dated comparison of microservices and Kubernetes testing tools in two layers: environment platforms (Signadot, Release, Bunnyshell, Okteto, Qovery, vCluster) and test automation and orchestration (Testkube, Postman, Pact, Playwright, k6, JMeter), plus local development tools and what coding agents change. ### Best Microservices Testing Tools for Kubernetes in 2026 The microservices testing tools for Kubernetes in 2026, in two layers: test automation such as Testkube, Pact and k6, and environment platforms such as Signadot and Release. Author Arjun Iyer Published July 17, 2025 Updated October 11, 2026 Reading time 19 min The best microservices testing tools for Kubernetes come in two layers. Test automation tools such as Testkube, Pact, Playwright and k6 write and run the tests, and environment platforms such as Signadot, Release, Bunnyshell and Okteto give each change somewhere real to run them. Most teams need one or two tools for each type of test and one environment platform, and the environment is where most microservices test suites go wrong. Postman, Pact and Playwright all assume the services they call are running, current, and not shared with someone else’s half-finished change, which a single staging cluster stops guaranteeing past a few teams. Coding agents raise the stakes. In CircleCI’s 2026 data, [throughput grew 59 percent in a year and about 30 percent of merge attempts failed](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/). This article lists the Kubernetes testing tools in each layer, what each is for, and where they overlap, checked against each vendor’s own docs and release pages on October 11, 2026. For how these tools fit a full strategy, from unit tests to production, see the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). For why to run tests on Kubernetes rather than on CI runners, which tests need a cluster, and the test environment options side by side, see the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). #### Microservices testing tools for Kubernetes at a glance There is no single best microservices testing solution, and the best tools for testing microservices depend on which layer is failing you. The table lists the tools to test microservices that teams most often combine, grouped by layer. Microservices testing tools for Kubernetes, October 2026. Checked against each vendor's own docs on October 11, 2026. Products change quickly, so follow the links for current details. Tool What it does Where it runs Best fit Environment platforms: somewhere real for each change to run [Signadot](https://www.signadot.com/docs/concepts/sandbox) Ephemeral environments that duplicate only what a change needs and share the rest of one cluster, with Jobs, Smart Tests and Plans to run tests in them An operator in your cluster, with a thin SaaS control plane PR environments and coding agents for dozens to hundreds of services, with databases, queues and Temporal [Release](https://docs.release.com/guides-and-examples/advanced-guides/microservices-architecture) Every service of the application in one namespace per environment, with production-like data from snapshots Your AWS or GCP account, run by Release Full-stack previews of applications small enough to copy [Bunnyshell](https://documentation.bunnyshell.com/docs/how-bunnyshell-works) A clone of a primary environment per pull request, one namespace each Your Kubernetes cluster, managed from Bunnyshell's SaaS Full-stack PR environments from Compose, Helm or Terraform [Okteto](https://www.okteto.com/docs/previews) A namespace per developer or pull request, a full copy by default, with Divert to borrow services from a shared namespace Your own cluster Development containers and previews on your own cluster [Qovery](https://www.qovery.com/docs/configuration/environment) A deployment platform that clones a blueprint environment per pull request Your cloud account Teams that want deployment and previews in one product [vCluster](https://www.vcluster.com/docs/vcluster/) A virtual cluster per environment, on shared nodes Your cluster Changes that need their own Kubernetes API Test automation and orchestration: what to test and how it runs [Testkube](https://testkube.io/) Runs existing suites as Test Workflows inside the cluster and collects the results. Does not create environments Open source runner in your cluster. Control plane as SaaS, or on premises on Enterprise One place to run and report every test type in Kubernetes [Postman](https://www.postman.com/) API design and testing, with collections that run from CI SaaS, with a CLI API checks on each service [WireMock](https://wiremock.org/) Simulates HTTP APIs so a service can be tested without its dependencies Open source library or server. WireMock Cloud is hosted Unit and component tests, third-party APIs [Microcks](https://microcks.io/) Open source API mocking and testing, a CNCF Incubating project Self-hosted Shared API mocks across teams [Pact](https://docs.pact.io/) Code-first contract tests for HTTP and message integrations Open source libraries. PactFlow, now Swagger Contract Testing, is the hosted broker Teams that own both sides of an API [Specmatic](https://specmatic.io/) Turns API specifications into executable contracts Open source core, with commercial tools Spec-first teams [Playwright](https://playwright.dev/) Browser automation for testing, scripting and AI agents Open source, from Microsoft End-to-end tests of web flows [Cypress](https://www.cypress.io/) End-to-end and component tests for web apps Open source app. Cypress Cloud is paid JavaScript frontends [Selenium](https://www.selenium.dev/) WebDriver browser automation in many languages Open source Cross-browser suites in existing languages [Shiplight](https://www.shiplight.ai/) Agents exercise UI changes in a real browser and keep the flows as Playwright-compatible E2E tests Commercial, with a free plan Browser coverage in coding-agent workflows [k6](https://grafana.com/oss/k6/) Scripted load tests from Grafana Labs Open source CLI, the k6 Operator in your cluster, or Grafana Cloud k6 Load tests in CI and in the cluster [Apache JMeter](https://jmeter.apache.org/) Load and performance tests across many protocols Open source Protocol-heavy load tests [Gatling](https://gatling.io/) Load tests written as code Open source Community Edition. Gatling Enterprise is commercial Load tests kept next to application code [OpenText Performance Engineering](https://blogs.opentext.com/what-happened-to-loadrunner-the-future-of-performance-engineering-at-opentext/) The LoadRunner family, renamed in October 2025 Commercial, on premises or cloud Enterprise performance engineering Local development: connect a laptop to the cluster [mirrord](https://metalbear.com/mirrord/) Runs a local process in the context of a cluster. Paid tiers add queue splitting, database branching and preview environments Open source CLI, with an operator in your cluster on paid tiers Debugging one service against real traffic [Telepresence](https://www.telepresence.io/) Intercepts cluster traffic to a process on your laptop Open source, a CNCF Sandbox project Local development against a shared cluster #### What makes testing microservices in Kubernetes different Testing microservices in Kubernetes is harder than testing one application because the thing under test is a graph of services, each deployed on its own schedule, and the cluster around them is part of what can break. Four problems decide which tools you need: - **Environment provisioning.** Every change needs somewhere realistic and isolated to run, and one shared staging cluster turns into a queue once several teams deploy to it. - **Service dependencies.** A service under test calls others. Mocking them is fast but drifts from reality, and running all of them for every change is slow and expensive. - **Data and async flows.** Databases, Kafka or SQS queues and workflow engines such as Temporal carry state between services, and tests that share them collide. - **Observability.** When a test fails across ten services, tracing has to point at the one that changed. Test automation tools address none of the first three on their own. That is the job of the environment layer. #### Test automation tools vs environment platforms The two layers answer different questions. A test automation tool decides what to test and how the tests run and report. An environment platform decides where a change runs, against which versions of its dependencies, and isolated from which other changes. Buying two tools from one layer does not fill the other: Testkube and Postman together still need an environment to point at, and an environment platform still needs tests. The confusion is understandable, because both layers run inside the cluster and both show up under Kubernetes testing tools. They meet at test execution. Testkube runs suites in the cluster against an environment you provide, and Signadot runs suites in an environment it provides. The [complete guide’s section on the two layers](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/#test-automation-tools-vs-environment-platforms) covers the overlap in more depth. #### Environment platforms for microservices testing Environment platforms differ most in how much of the system they copy for each change. Most copy the whole environment, a namespace or a cluster per pull request. Signadot duplicates only what a change needs and plugs it into one shared environment. That choice sets spin-up time, cost as environments multiply, and how databases and queues are handled, and the [ephemeral environment tools comparison](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#ephemeral-environment-tools-compared) lines the platforms up on all three. ##### Signadot [Signadot](https://www.signadot.com/docs/concepts/sandbox) gives each change an ephemeral environment, called a Sandbox, that duplicates only the services and resources the change needs and plugs them into one shared environment. It links the two in three ways: it routes requests by key for synchronous traffic, routes messages by the same key for async consumers and Temporal workers, and rewrites a forked service’s configuration so it uses a new version of a dependency. Sandboxes start in seconds, and hundreds can share one cluster. Data and dependencies are where it differs most from the full-copy platforms. A change can share the stable database with its data kept apart, or get a temporary database or a Neon or Xata branch through [resource plugins](https://www.signadot.com/docs/reference/resource-plugins), which also provision managed queues, buckets and other cloud resources beyond the cluster and remove them when the Sandbox ends. Message routing works on one shared Kafka, RabbitMQ, SQS, Pub/Sub or NATS broker, and the routing key travels through [Temporal workflows](https://www.signadot.com/docs/tutorials/testing-temporal-workers). Signadot also runs tests inside its environments. Jobs run existing suites in the cluster, Smart Tests compare a Sandbox’s API behavior with the stable version’s, and Plans let coding agents build test workflows. Agents connect through an [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Most of Signadot runs in your cluster. By default its SaaS control plane receives only resource metadata, such as workload names, namespaces, image references and status, and never source code, image contents or application data, as its [data and integrations page](https://www.signadot.com/docs/guides/admin/security/data-integrations) describes. Signadot is SOC 2 Type II. It fits applications with dozens to hundreds of services, PR environments for every change, and coding agents that each need their own environment. ##### Release [Release](https://release.com/), formerly ReleaseHub, is the closest direct alternative to Signadot for PR environments, built the opposite way. It puts [every service of the application in one namespace per environment](https://docs.release.com/guides-and-examples/advanced-guides/microservices-architecture), in your AWS or GCP account, and runs those clusters for you. [Instant Datasets](https://docs.release.com/reference-documentation/instant-datasets-aws) give each environment a ready RDS, Aurora or Cloud SQL copy from a production snapshot, its Neon integration branches a database per environment, and a Terraform runner creates other cloud resources. Release now offers the same full-stack copy to AI agents through an API and an [MCP server](https://release.com/product/mcp-server). It does not run your tests. Your CI creates the environment, [runs the suite against it](https://docs.release.com/reference-documentation/e2e-testing) and tears it down. Release fits applications small enough to copy in full, where a standalone copy per pull request with production-like data is worth what it costs. [Signadot vs Release](https://www.signadot.com/comparison/release/) compares the two in detail. ##### Bunnyshell [Bunnyshell](https://documentation.bunnyshell.com/docs/how-bunnyshell-works) creates a clone of a primary environment for each pull request, each in its own namespace, from Docker Compose, Helm or Terraform definitions. Databases run as containers seeded from dumps, as RDS instances restored from snapshots through Terraform, or as Neon, PlanetScale or Xata branches, and queues run inside each environment. It is [billed per active environment minute](https://www.bunnyshell.com/pricing/), and by [its own description](https://www.bunnyshell.com/e2e-testing/) it is the environment layer underneath a test runner rather than a runner itself. It fits teams that want full-stack PR environments without building the automation themselves. ##### Okteto [Okteto](https://www.okteto.com/docs/previews) gives each developer or pull request a namespace, by default with a full copy of the application. [Divert](https://www.okteto.com/docs/core/divert) lets a namespace hold only the services you changed and borrow the rest from a shared namespace, but it is opt-in, routes requests by an HTTP header, and leaves queue isolation to your own consumer code. Okteto is [billed per seat](https://www.okteto.com/docs/admin/billing), and its free Starter plan covers five seats, self-hosted. [Okteto alternatives](https://www.signadot.com/comparison/okteto/) compares it with Signadot. ##### Qovery, Northflank and other full-copy platforms [Qovery](https://www.qovery.com/docs/configuration/environment) and [Northflank](https://northflank.com/docs/v1/application/release/pipeline/set-up-a-preview-environment) are deployment platforms with preview environments built in. Qovery clones a blueprint environment into your cloud account for each pull request, in [about three minutes by its own figure](https://www.qovery.com/solutions/ephemeral-environments). Northflank creates a preview from a template, self-contained or sharing services and a database with a development environment. [Shipyard](https://docs.shipyard.build/), which builds full-stack environments from Docker Compose, was [acquired by StackGen](https://stackgen.com/blog/shipyard-joins-stackgen) in September 2026. [Qovery alternatives](https://www.signadot.com/comparison/qovery/) covers the first of these in detail. ##### vCluster and in-house namespaces Platform teams often build the full copy themselves. An [Argo CD ApplicationSet](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Pull-Request/) with the pull request generator deploys a namespace per pull request, and [vCluster](https://www.vcluster.com/docs/vcluster/) gives each environment a virtual cluster with its own API server on shared nodes, which suits changes to CRDs or cluster-level configuration. Both copy the whole application into each environment. [vCluster alternatives](https://www.signadot.com/comparison/vcluster/) explains when a virtual cluster per pull request is worth it. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Microservices test automation and orchestration tools ##### Kubernetes test orchestration: Testkube and Signadot Jobs When people search for a Kubernetes testing framework, or a k8s testing framework, they usually mean a way to run their existing tests inside the cluster rather than a new way to write them. [Testkube](https://testkube.io/) is the best-known tool for that. It runs suites from Postman, k6, JMeter, Playwright, Cypress, pytest, Maven, Gradle and others as Test Workflows in your cluster, triggers them from CI or cluster events, and collects logs and artifacts in one place. The runner is open source. The dashboard, multi-cluster runs, reporting, and its AI agents and MCP server need the commercial control plane, which starts at [$700 a month per runner on the Pro plan](https://testkube.io/pricing). Testkube does not create the environment the tests run against. Its [ephemeral environments guide](https://docs.testkube.io/articles/ephemeral-environments) starts by provisioning a namespace or cluster with “the tool of your choice”, then installs a runner into it. Signadot’s [Jobs](https://www.signadot.com/docs/concepts/smart-tests-and-jobs) overlap with this from the environment side. They run a suite in any framework in pre-warmed pods in your cluster, in the context of a Sandbox, stream the output to CI and keep the artifacts. Smart Tests, built on Jobs, compare a Sandbox’s API responses with the stable version’s, and [Plans](https://www.signadot.com/docs/reference/plans/spec) are compiled graphs of test actions that a coding agent can author and run against any Sandbox. Teams already on Testkube keep it and point its runs at a Sandbox with the Sandbox’s routing key. Teams without an orchestrator can start with Jobs. The other meaning of Kubernetes testing framework is a framework for testing Kubernetes itself. The Kubernetes SIGs [e2e-framework](https://github.com/kubernetes-sigs/e2e-framework) is a Go framework for end-to-end tests of components running in a cluster, for teams building Kubernetes components rather than applications on top of them. ##### API and integration testing tools Integration testing microservices on Kubernetes takes an API test tool and a decision about what the service calls during the test: the real dependency, or a stand-in. [Postman](https://www.postman.com/) covers API design and testing, with collections a team shares and runs from CI against each service. For testing a service without its dependencies, [WireMock](https://wiremock.org/) and [MockServer](https://www.mock-server.com/) simulate HTTP APIs. Both are open source under Apache 2.0 and actively maintained, and WireMock Cloud is the hosted version. [Microcks](https://microcks.io/), a [CNCF Incubating project since May 2026](https://www.cncf.io/projects/microcks/), runs API mocking and testing as a shared service. Mocks earn their place at the unit and component layers, and for outside APIs a test cannot reach. They are the wrong tool for checking integration between your own services, because a mock keeps passing after the real service changes. The [article on mocks that pass while staging breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) explains the failure. ##### Contract testing tools [Pact](https://docs.pact.io/) is the open source standard for consumer-driven contract tests, covering HTTP and message integrations. SmartBear’s [PactFlow](https://pactflow.io/), now being renamed Swagger Contract Testing, is the hosted broker and adds bi-directional contract testing. [Specmatic](https://specmatic.io/) turns OpenAPI and other API specifications into executable contracts, for teams that design APIs spec-first. Contract tests check the shape of a conversation between services, not the behavior behind it, which is why they sit next to integration tests rather than replacing them. The [complete guide to contract testing](https://www.signadot.com/contract-testing/) compares the tools in depth. ##### End-to-end testing tools End-to-end tests verify complete user journeys across services. They are the slowest and most brittle layer, so keep the suite thin. - **Playwright:** Microsoft’s open source browser automation for testing, scripting and AI agents. Its MCP server and CLI let coding agents drive a real browser. - **Cypress:** Open source end-to-end and component testing for web apps, with the paid Cypress Cloud on top. It suits teams whose frontends are written in JavaScript. - **Selenium:** The long-standing WebDriver standard for browser automation, with bindings for many languages. Suites written against it tend to need more maintenance as the UI changes. - **[Shiplight](https://www.shiplight.ai/):** Shiplight adds a browser layer to coding-agent workflows. It gives an agent a real browser to exercise UI changes during development, then turns those flows into readable, self-healing E2E tests that live in the team’s repository. The tests are Playwright-compatible and run locally or in CI, so browser coverage stays close to the code while still running against Kubernetes-based preview, staging, and test environments. Coverage is limited to browser-facing flows, so it sits alongside API, contract, and service-level testing in a microservices stack. ##### Performance and Kubernetes load testing tools These are the performance testing tools for microservices that most teams shortlist, and all four run as Kubernetes load testing tools, either inside the cluster or against it. Load tests need the most production-like environment of any layer, because a test against an undersized or shared copy measures the copy. - **k6:** [Grafana k6](https://grafana.com/oss/k6/) writes load tests in JavaScript, runs them from the CLI or CI, and runs distributed tests inside a cluster through the [k6 Operator](https://github.com/grafana/k6-operator). The open source tool is AGPL-3.0, and Grafana Cloud k6 is the hosted service. - **Apache JMeter:** [JMeter](https://jmeter.apache.org/) covers HTTP and many other protocols and runs in CI. Its most recent release is 5.6.3, from January 2024. - **Gatling:** [Gatling](https://gatling.io/) defines load tests as code. Its Community Edition is open source and meant for local use, and Gatling Enterprise is the commercial version. - **OpenText Performance Engineering:** the LoadRunner family, [renamed by OpenText in October 2025](https://blogs.opentext.com/what-happened-to-loadrunner-the-future-of-performance-engineering-at-opentext/). It remains the enterprise option for teams that need its protocol coverage and vendor support. #### Local development tools: mirrord and Telepresence Local development tools connect a process on a laptop to a cluster, so one service can be run and debugged against real neighbors without deploying it. They are not environment platforms, though the line has moved. [mirrord](https://metalbear.com/mirrord/) from MetalBear is open source for individual use. Its Team plan, at [$50 per seat a month](https://metalbear.com/mirrord/pricing/), adds a shared cluster, queue splitting and database branching, and its Enterprise plan adds preview environments and use in CI. [Telepresence](https://www.telepresence.io/), which started at Ambassador and is now a CNCF Sandbox project, intercepts cluster traffic to a local process. Ambassador was [acquired by Gravitee in 2025](https://www.gravitee.io/blog/ambassador-labs-gravitee-what-our-acquisition-means-for-you). [Signadot vs mirrord](https://www.signadot.com/comparison/mirrord/) and [Signadot vs Telepresence](https://www.signadot.com/comparison/telepresence/) compare both with PR environments. #### How to choose Kubernetes testing tools for microservices Start from where your failures come from, not from a feature list. Five questions settle most of the choice. 1. **Where do your bugs come from?** If production incidents cluster at the seams between services, the gap is in the environment layer, not in test authoring, and another test framework will not close it. 2. **How many services and teams do you have?** A system of fewer than ten services owned by one or two teams can live with a shared staging cluster or a full copy per pull request. Past that, copying everything per change gets slow and expensive, and sharing one environment while isolating each change scales better. 3. **What do your services depend on?** Ask every environment platform how it handles your databases, Kafka or SQS, workflow engines such as Temporal, and cloud resources outside the cluster. This is where platforms differ most. 4. **Who runs the tests?** CI, an orchestrator such as Testkube, the environment platform’s own runner, or a coding agent. Pick one place where results land. 5. **Where does it have to run?** In your cluster, in your cloud account, or as SaaS, and what data leaves your network. Check each vendor’s security documentation rather than its marketing page. A Kubernetes testing framework that answers all five usually turns out to be two or three tools, not one. #### What coding agents change about microservices testing tools Coding agents turn testing tools from something a person runs before review into something software calls many times per change. [Claude Code](https://code.claude.com/docs/en/overview), [Cursor](https://cursor.com/), Codex and [GitHub Copilot](https://github.com/features/copilot) now open pull requests without a person at the keyboard, and [the agent PR flood](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/) shows what that does to test infrastructure: a 50-engineer team’s staging was sized for 100 to 150 pull requests a day, and agent output arrives as a spike it was never built to drain. Three things follow for tool choice. Agents need an environment per run, created and destroyed through a CLI, an API or an MCP server, because an agent iterating on a failure cannot queue for staging between attempts. The tests have to run before human review, so integration and end-to-end results arrive with the pull request rather than after approval. And every environment needs a time-to-live and a cost tied to the services each change touches, not to how many agents are running, because abandoned agent runs pile up. Both layers are adapting. Playwright ships an MCP server so agents can drive a browser, Testkube sells AI agents and an MCP server on its commercial plans, and Shiplight turns agent browser sessions into E2E tests. Most environment platforms now court agents too: [Release](https://release.com/product/mcp-server), [Qovery](https://www.qovery.com/docs/copilot/mcp-server) and [Shipyard](https://docs.shipyard.build/mcp) ship MCP servers, and [Bunnyshell](https://www.bunnyshell.com/claude-code-skill/), [Okteto](https://www.okteto.com/docs/agentic) and [Northflank](https://northflank.com/docs/v1/application/ai-agents/agent-skills) publish skills for coding agents, though most still give each agent a full copy of the environment. Signadot exposes Sandbox creation and test runs to agents through its MCP server and skills, and Plans let an agent assemble a test workflow it can run again on the next change. The [guide to validating AI-generated code](https://www.signadot.com/validate-ai-generated-code-kubernetes/) walks through the full loop. #### Where to go next If your tests are fine and the environment is the problem, start with the [staging environments guide](https://www.signadot.com/staging-environments/) and the ephemeral environment comparison above. If you are building a strategy from scratch, the complete guide to microservices testing walks through each layer of the pyramid. If coding agents open your pull requests, start with the validation guide above, because the pipeline you build for them becomes the one everyone else uses. #### Frequently asked questions What are the best microservices testing tools for Kubernetes? It takes tools from two layers. For test automation, teams combine Postman for API tests, Pact or Specmatic for contracts, Playwright or Cypress end to end, k6 or JMeter for load, and Testkube to run suites in the cluster. For environments, a platform such as Signadot, Release, Bunnyshell or Okteto gives each change somewhere real to run them. The environment decides whether the results mean anything. Is Testkube an environment platform? No. Testkube is test orchestration: it runs existing suites, such as Postman, k6, Playwright or pytest, inside a Kubernetes cluster and collects the results. Its docs leave provisioning the environment to another tool. Environment platforms such as Signadot or Release create the environment a change runs in. The two overlap at test execution, since Signadot Jobs also run suites in the cluster. What is a Kubernetes testing framework? Usually a way to run tests inside or against a Kubernetes cluster rather than a new way to write them. Testkube runs existing suites as cluster workloads, and the k6 Operator does the same for load tests. The Kubernetes SIGs e2e-framework is different: a Go framework for testing components that run in Kubernetes. For microservices, any of them still needs the services under test running at current versions. Do I need an environment platform in addition to testing tools? Past a few teams, usually yes. Testing tools assume a stable, current environment to run against, and one shared staging cluster stops providing that once several teams deploy to it. An environment platform gives each change its own isolated environment with current dependencies, so the rest of the toolchain produces results you can trust. Small systems can manage with a shared staging environment. Which testing tools handle Kafka, databases and Temporal? Test frameworks call the services, so what matters is how the environment handles their dependencies. Full-copy platforms such as Release and Bunnyshell run a database and queues inside each environment. Signadot shares one broker and routes messages by key, carries the key through Temporal workflows, and gives schema changes their own database or branch through resource plugins. mirrord's paid tiers add queue splitting and database branching. What are the alternatives to Signadot for PR environments? Release is the closest direct alternative: it also creates an environment for every pull request, but copies the whole application into each one, in your cloud account. Bunnyshell, Okteto and Qovery copy the whole environment too, and mirrord's Enterprise plan adds preview environments of changed services. Signadot duplicates only what a change needs and shares the rest of one cluster, which suits apps with many services. Can coding agents use these testing tools? Yes, and most tools now expose something for agents. Playwright ships an MCP server for driving a browser, Testkube offers an MCP server on paid plans, and Signadot gives agents such as Claude Code and Cursor an MCP server and skills to create an isolated environment, run tests in it and tear it down. What agents need most is an environment per run, created without waiting on a person. What are the best load testing tools for microservices on Kubernetes? k6, Apache JMeter, Gatling and OpenText Performance Engineering, formerly LoadRunner, cover most needs. k6 writes tests in JavaScript and runs distributed tests in a cluster through the k6 Operator. JMeter covers the widest range of protocols. Gatling defines tests as code. Whichever you pick, run it against an environment sized and configured like production, or the test measures the environment instead of the service. #### Related Articles - [ *Image: Why Signadot Outperforms Other K8s Testing Solutions* ### Why Signadot Outperforms Other K8s Testing Solutions April 30, 2025 ](https://www.signadot.com/articles/why-signadot-outperforms-other-k8s-testing-solutions/) - [ *Image: Shift-Left Testing for Microservices on Kubernetes* ### Shift-Left Testing for Microservices on Kubernetes September 30, 2026 ](https://www.signadot.com/articles/shift-left-testing/) - [ *Image: Stop Breaking Your Microservices with SmartTests: AI Powered Contract Testing* Contract Testing ### Stop Breaking Your Microservices with SmartTests: AI Powered Contract Testing June 18, 2025 ](https://www.signadot.com/articles/stop-breaking-your-microservices-with-smarttests-ai-powered-contract-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Ephemeral Environments: The Complete Kubernetes Guide Source: https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/ Summary: What ephemeral environments are, ephemeral testing and development environments, and the two ways to build them: copy the whole environment for each change, or duplicate only what changed and plug it into a shared environment. Includes a dated comparison of ephemeral environment tools, how ephemeral environments handle databases, message queues, Temporal workflows and resources outside Kubernetes, what they cost, service meshes, coding agents and a rollout checklist. ### Ephemeral Environments: The Complete Kubernetes Guide What ephemeral environments are, the four ways to build them on Kubernetes, the tools compared, and how they handle cost, databases, message queues and the volume coding agents bring. Published February 28, 2026 Updated October 10, 2026 Reading time 37 min Ephemeral environments are temporary, isolated environments created on demand for one code change, pull request or test run, and deleted when that work is done. Each one gives the change its own place to run against the real services it depends on, so developers, CI jobs and coding agents can test in parallel instead of taking turns in one shared staging environment. There are two ways to build them, at opposite ends of one spectrum: duplicate the whole environment for every change, or duplicate only what changed and plug it into a shared environment that runs everything else. The idea took hold as microservices on Kubernetes broke the two older options. One shared staging environment turned into a queue, and a full copy of the system per developer cost too much to keep running. Coding agents have raised the stakes. By mid-2026, [one in three pull requests on GitHub involved an agent](https://www.microsoft.com/en-us/investor/events/fy-2026/earnings-fy-2026-q4), and every one of them needed somewhere real to run before a person reviewed it. One staging slot per team, or a full copy of the stack per pull request, cannot absorb that volume. Duplicating only what changed can. The [complete guide to software factories](https://www.signadot.com/software-factory/) explains why that per-change environment is the sensor that keeps an agent-driven delivery loop stable. What follows covers the definition and its neighbors, ephemeral testing and development environments, how ephemeral environments work on Kubernetes, the four ways to build them, the tools compared, databases, message queues and workflow engines, service meshes, cost, coding agents and rollout. It is one part of the wider practice in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### What are ephemeral environments? An ephemeral environment is a [short-lived, isolated deployment](https://ephemeralenvironments.io/) of an application, or of the part of it a change touches, that exists only for the lifetime of the work that created it. Three properties define it: - **Created on demand.** A pull request opening, a developer running a command, a CI job starting or an agent beginning a task creates it. Nobody provisions it ahead of time. - **Bound to one unit of work.** It carries one change, or one test run, and nothing else. Nothing about it is patched or kept in sync by hand, so it is always freshly built from the current main branch plus that change. - **Destroyed automatically.** A merge, a close, a delete command or a time to live (TTL) removes it, along with any data it created. ##### What ephemeral environments are used for - Running integration and end-to-end tests on a pull request against the real versions of its dependencies before merge. This is the [ephemeral testing environment](#ephemeral-testing-environments), the most common form. - Giving a developer a place to run the service they are changing against real neighbors while the change is written, the [ephemeral development environment](#ephemeral-development-environments). - Letting a coding agent run its build, test and fix loop against real dependencies on every attempt, in parallel with every other agent. - Handing a reviewer, a product owner or QA a working URL for one change without a turn in a shared environment. ##### What they are not for - Load and performance testing at production scale, which needs sizing an ephemeral environment does not normally carry. - Acceptance cycles measured in weeks, where the environment stops being ephemeral in any useful sense and becomes an unmanaged staging copy. - Running untrusted code. An ephemeral test environment isolates a change from other changes, not a hostile process from its host. ##### Other names for ephemeral environments The same idea goes by several names, each stressing a different property. On-demand environments and dynamic environments stress how they are created. Per-PR environments, pull request environments and branch environments stress the trigger. Short-lived and temporary environments stress the teardown. Ephemeral test environments, ephemeral testing environments and ephemeral development environments name who uses them. Whatever the name, the meaning is the same: an environment whose lifetime is bound to one unit of work. ##### Ephemeral vs static environments A static environment, also called a persistent or long-lived environment, is provisioned once and kept running, and every change that needs it takes a turn. Staging, QA, UAT and pre-prod are static environments, and so is the shared dev cluster many teams run. The difference from an ephemeral environment is ownership of time, not size or fidelity. A static environment is shared by changes in sequence, and an ephemeral environment belongs to one change for its whole lifetime and is deleted after. Staging is the static environment ephemeral environments most often sit in front of. It drifts because nobody owns the manual fixes it accumulates. The two still depend on each other: every ephemeral environment needs a stable, production-like set of services to run against, and in most designs that stable set is what staging becomes. The [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers when one shared environment is still the right call. ##### Ephemeral vs preview environments Every preview environment is ephemeral, but not every ephemeral environment is a preview. Ephemeral describes the lifecycle: created for one unit of work and deleted after it. Preview describes one use: trying the change through the web or mobile front end, manually, whether a reviewer clicks through or a coding agent explores it ad hoc. An environment a CI job creates, runs automated tests in and deletes with nobody looking is ephemeral but not a preview. Many preview products rebuild only the frontend in front of shared backends, which is fine for a UI review and useless for a backend change, as [per-PR preview URLs](https://www.signadot.com/articles/per-pr-preview-urls/) explains. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares preview platforms. ##### Ephemeral environments vs sandboxes The word sandbox means two things, and only one of them is an ephemeral environment. Signadot’s ephemeral environments, called Sandboxes in its API, are the second kind: each duplicates only what a change needs on a shared cluster, usually the services it touched, and routes that change’s requests and messages to them. A sandbox for running AI-generated code, the kind [E2B](https://e2b.dev/), [Modal](https://modal.com/) and [Daytona](https://www.daytona.io/) sell, is a secure runtime for executing untrusted code, sized for one process, and has nothing to do with testing a system of services. The [Kubernetes sandbox](https://www.signadot.com/kubernetes-sandbox/) page separates the two in detail. ##### Ephemeral deployments, clusters, infrastructure and containers The word ephemeral attaches to different layers, and the layer matters: - **Ephemeral deployment.** A set of Kubernetes Deployments created for one change and deleted with it. This is what most ephemeral environments on Kubernetes are made of. - **Ephemeral cluster.** A whole Kubernetes cluster created and destroyed per use. Ephemeral clusters suit CI runners and platform teams testing cluster-level changes, but creating a cluster on EKS, GKE or AKS takes longer than anyone will wait per pull request. Kubernetes ephemeral environments on AWS, Google Cloud or Azure normally keep a long-lived cluster and make the workloads on it ephemeral. - **Ephemeral infrastructure.** The broadest term. It covers both of the above and the general practice of treating servers as disposable. - **Ephemeral container.** Unrelated. A Kubernetes [ephemeral container](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/) is a debugging container attached to a running pod, for inspecting a service rather than building an environment. #### Ephemeral testing environments An ephemeral testing environment is an ephemeral environment that CI creates for one pull request or test run, where integration and end-to-end tests run against the real versions of the services the change depends on, and which is deleted when the run ends. It is the most common kind, and what most teams mean by ephemeral test environments or ephemeral environments for testing. Four properties make it better for testing than a shared environment: - **Attribution.** One change per environment means a failing test names one change. In a shared staging environment a failure can come from any change deployed that day, and someone has to work out which. - **Real dependencies, current with main.** Tests call the services the change will meet in production, at the versions on main, rather than mocks that drift from how those services behave. - **Parallel runs.** Ten pull requests get ten environments at once, so time to a result does not grow with the queue. - **A known starting state.** Every run starts from the same data and versions, not from what another test left behind. Any test framework runs inside one. Playwright, Cypress and Selenium drive the UI, Postman and RestAssured exercise APIs, and in-cluster runners such as [Testkube](https://testkube.io/) or Signadot [Jobs](https://www.signadot.com/product/jobs/) run suites next to the services under test. The environment provides the place to run, and the tests decide whether the change is good. CI posts the result and the URL back to the pull request, and a TTL removes the environment if CI does not. [Kubernetes test environments](https://www.signadot.com/solutions/kubernetes-test-environments/) covers how this looks with Signadot. #### Ephemeral development environments An ephemeral development environment, or ephemeral dev environment, is one a developer works in while a change is still being written: the service being edited runs on a laptop or in the cluster, connected to real versions of everything it calls, and the environment goes away with the branch. It replaces two older habits, mocking the neighbors and running the whole system locally, both of which stop working past a dozen services. Two shapes are common: - **A local process connected to a cluster.** The developer runs the service on a laptop with a debugger and hot reload, and requests meant for that service are sent from the cluster to the laptop. Signadot does this with the same ephemeral environment it gives pull requests and coding agents. Local development tools such as [mirrord](https://metalbear.com/mirrord/) and [Telepresence](https://telepresence.io/) also connect a local process to a cluster, and differ in how they keep one developer’s traffic away from another’s. - **A remote environment in the cluster.** **A remote environment in the cluster.** The code syncs into a container in the cluster and rebuilds there, so a laptop runs little more than an editor. [Okteto](https://www.okteto.com/docs/) works this way, and its [Divert](https://www.okteto.com/docs/core/divert) mode deploys only the services a developer changes into their namespace and sends everything else to a shared one. Either way, the ephemeral part is the changed service, its own data where the change writes, and routing that keeps one developer’s requests away from everyone else’s. It lives for the life of a branch rather than a CI run, so a TTL tied to inactivity keeps it from becoming a long-lived copy nobody owns. The [guide to local development on Kubernetes](https://www.signadot.com/guide-to-local-development-kubernetes/) compares the options in depth. #### How ephemeral environments work on Kubernetes An ephemeral environment on Kubernetes passes through six steps whichever model builds it. The models in the next section differ in what step two provisions and what step three routes, and those two choices set the cost and spin-up time of everything else. TriggerPR or agentProvisionworkloadsRoutetag requestsSeedattach dataTestrun testsExpireTTL or merge Provision and route, in purple, are the two steps the four build models do differently. TriggerPR or agentProvisionworkloadsRoutetag requestsSeedattach dataTestrun testsExpireTTL or merge Provision and route, in purple, are the two steps the four build models do differently. - **Trigger.** A pull request opens, a developer runs a CLI command, a CI job starts, or an agent calls an API or an MCP server. The trigger carries the identity of the change: a branch, an image tag and the person or agent behind it. - **Provision.** A controller creates the workloads. In a namespace-per-change model that means applying every manifest of the application into a fresh namespace, usually through a Helm release or an [Argo CD ApplicationSet](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Pull-Request/). When each change duplicates only what it needs, it means deploying just those parts, often the one or two services it touched, next to the stable versions of everything else. - **Route.** Full-copy models get a new ingress hostname per environment. When only the changed parts are duplicated, test requests carry a routing header, and each hop sends a tagged request to the changed version when one exists and to the shared stable version when it does not. Messages carry the same key, so asynchronous consumers pick up only their own, and some components need only a configuration change, such as a connection string that points the changed service at its own database. - **Seed.** The environment needs data: a temporary database restored from a fixture, a branch of a database that supports branching, or the shared datastore used read-mostly. Queues need dedicated topics or a way to keep this environment’s messages away from other consumers. - **Test.** CI runs the integration and end-to-end suites, a reviewer clicks through, an agent runs its loop. Logs and traces carry the environment’s identity so a failure points back at the change that caused it. - **Expire.** The pull request merges or closes, the developer deletes the environment, or the TTL fires, and everything provisioned in step two is removed along with the data from step four. An ephemeral environment without an expiry is a staging environment nobody asked for. ##### Ingress, DNS and URLs for many ephemeral environments Full-copy models use a wildcard DNS record such as `*.preview.example.com` pointing at one [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) or [Gateway API](https://gateway-api.sigs.k8s.io/) gateway, with a host rule per environment and a wildcard certificate. Models that duplicate only what changed use one hostname for the whole cluster and a routing header that selects the environment, so there is no per-environment DNS or certificate work, but a reviewer’s browser needs a small extension or a preview URL that injects the header. Either way, the CI job that created the environment posts its URL back to the pull request. #### The four ways to build ephemeral environments on Kubernetes The four models differ in how much of the system each change deploys for itself, and they fall along the spectrum in the introduction. Three copy some or all of the system per change: a cluster per environment, a namespace per environment, and a preview build in front of shared backends. The fourth shares one cluster and lets each change duplicate only what it needs, usually the services it touched, while request routing, message routing and configuration connect those copies to everything else. From the most infrastructure per change to the least: Full environment per PRPR 1PR 2PR 3Namespace per PRns 1ns 2ns 3one clusterPreview environmentUI 1UI 2UI 3shared backendOnly what changedshared serviceschange Achange Bchange Cper change Purple marks what each change has to deploy for itself. Full environment per PRPR 1PR 2PR 3Namespace per PRns 1ns 2ns 3one clusterPreview environmentUI 1UI 2UI 3shared backendOnly what changedshared serviceschange Achange Bchange Cper change Purple marks what each change has to deploy for itself. ##### A cluster or virtual cluster per environment Every environment gets its own Kubernetes cluster, or a virtual cluster with its own API server on shared nodes. - **Strengths.** Isolation is complete. Each cluster can run a different Kubernetes version, CRD set or mesh configuration, which makes this the only option for testing cluster-level changes. - **Costs.** Each cluster runs a full copy of every service, a control plane and its own copies of every datastore. A real cluster takes tens of minutes to create. [vCluster](https://www.vcluster.com/) cuts that to a virtual control plane in under a minute at close to namespace cost, and the [vCluster comparison](https://www.signadot.com/comparison/vcluster/) covers where that tier fits. - **Best fit.** Hard tenancy requirements, and platform teams whose changes are to the cluster itself. ##### A namespace per environment on a shared cluster The same idea built from primitives you already have: a [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) per environment, with resource quotas, network policies and RBAC for isolation, on one cluster. The tooling is a Helm chart or an ApplicationSet that stamps the application into a fresh namespace when a pull request opens, and it is the model most GitOps pipelines reach for first. - **Strengths.** Cheaper than a cluster per environment, no new infrastructure, and familiar to anyone who runs Kubernetes. - **Costs.** It still deploys every service into every namespace. Data has to be loaded per namespace, startup grows with the application, and each namespace drifts from main between rebuilds. Control-plane load and the security surface grow with the namespace count. [Kubernetes namespace vs cluster: pros and cons for test environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) weighs both options, and [ArgoCD preview environments](https://www.signadot.com/articles/argocd-preview-environments/) shows where the ApplicationSet pattern strains. - **Best fit.** Applications under about ten services, on teams fluent in Kubernetes. ##### A preview build in front of shared backends For frontend and API-gateway changes, one rebuilt service pointed at shared backends is often enough. It is the cheapest and fastest model, a single build and a URL, and what most hosted preview products deliver by default. - **Strengths.** Seconds to provision, one image to build, nothing to seed. - **Costs.** Only the frontend is isolated. Anything the change writes goes to the shared backends, and a backend change under test has nowhere to run. - **Best fit.** UI review and stakeholder demos. ##### Only what changed, on a shared cluster One cluster runs one set of stable service versions, kept current with main. Each change duplicates only what it needs next to those stable versions, usually the services it touched and sometimes its own database or queue, and receives an identity. Test requests carry that identity in a header, and at every hop a tagged request goes to the changed version when one exists and to the shared stable version when it does not. Messages carry it too, so asynchronous consumers process only their own, and some components need only a configuration change to point at the right copy. Each change can duplicate anything from only the services it touched up to the whole stack. - **Strengths.** Hundreds of ephemeral environments for microservices coexist on one cluster, a new one is ready in seconds, and every environment runs against real, current dependencies rather than a copy that drifts. At any real scale on Kubernetes, this is the model that holds up. - **Costs.** The prerequisites are in the application, not the infrastructure. Every service must propagate the routing header on outgoing calls, asynchronous flows need the identity carried in message metadata, destructive changes need their own datastore, and the shared stable versions have to be kept green. The sections on data and service meshes below take those in turn. - **Best fit.** Dozens to hundreds of services, many concurrent changes, and coding agents. [Isolation models for ephemeral environments](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) sets this model against the full-copy ones in detail. Lyft and Razorpay built request routing in-house before it existed as a product, on Envoy sidecars and a Traefik ingress respectively, and [how Lyft and Razorpay share development environments with hundreds of devs](https://www.signadot.com/blog/how-lyft-and-razorpay-share-development-environments-with-hundreds-of-devs/) describes what each took to run. One option sits below all four: run everything on one machine with Docker Compose, kind or minikube. It is the right answer for a small application, and the [guide to lightweight Kubernetes environments](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) covers how far it stretches. Past a dozen services it stops fitting on a laptop, and a single node never behaves like the distributed system it stands in for. Try a lightweight ephemeral environment on your cluster Create an ephemeral environment on the cluster you already run. Only what the change needs deploys, with no duplicated stack and no staging queue. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Ephemeral environment tools compared The best tool for ephemeral environments depends on which end of the spectrum you need. Most platforms copy the whole environment for each pull request, platform teams build the same copy themselves with Argo CD, Helm or vCluster, and a smaller group duplicates only what each change needs and plugs it into a shared environment. The table groups them that way. Ephemeral environment tools compared, October 2026. Checked against each vendor's own docs and pricing pages on October 9, 2026. Products change quickly, so follow the links for current details. Tool Spin-up time Cost as environments multiply Databases, queues and workflow systems Workloads outside Kubernetes Best fit Copy the whole environment per change: platforms [Bunnyshell](https://documentation.bunnyshell.com/docs/how-bunnyshell-works): a clone of a primary environment per pull request, one namespace each Not published. Image builds and data seeding set it Full stack per environment. [Billed per active environment minute](https://www.bunnyshell.com/pricing/). Stops on a schedule, destroyed on merge or close Database containers seeded from dumps, RDS from snapshots through Terraform, or Neon, PlanetScale and Xata branches. Queues run per environment Terraform and generic components create cloud resources such as RDS Full-stack PR environments from Compose, Helm or Terraform [Northflank](https://northflank.com/docs/v1/application/release/pipeline/set-up-a-preview-environment): a preview template per pull request, self-contained or sharing services and a database with dev Not published for previews. Builds and database forks set it [Billed per second for CPU and memory](https://northflank.com/pricing). Previews take a TTL, pause on a schedule and are deleted when the PR closes PostgreSQL, MySQL, MongoDB, Redis, MinIO and RabbitMQ addons, forked per preview. Kafka and Temporal not stated in its docs OpenTofu nodes and managed S3 and RDS addons create cloud resources A general Kubernetes platform with templated previews [Okteto](https://www.okteto.com/docs/previews) previews: whatever the manifest deploys, usually the full stack Not published. [Its docs](https://www.okteto.com/docs/core/divert) use 5 to 10 minutes for a full stack as an example Full stack per environment. [Billed per seat, not per environment](https://www.okteto.com/docs/admin/billing). Idle environments [sleep, then are deleted](https://www.okteto.com/docs/admin/cleanup) A database container per namespace, a database from another repository, or a restored volume snapshot External resources: deploy and destroy commands run Terraform or a cloud CLI Development and preview environments on your own cluster [Qovery](https://www.qovery.com/docs/configuration/environment): a clone of a blueprint environment per pull request, trimmable to fewer services [About 3 minutes from commit to URL](https://www.qovery.com/solutions/ephemeral-environments), by Qovery's own figure Every included service, in your cloud account. [Plan fee plus deployment minutes](https://www.qovery.com/pricing). Deleted on merge, close or inactivity Databases cloned or shared per preview. Blueprints for MSK, RabbitMQ, Redpanda and a Temporal Cloud namespace Terraform, OpenTofu and CloudFormation through lifecycle jobs, including Lambda functions A Kubernetes control plane in your own cloud account [Release](https://docs.release.com/guides-and-examples/advanced-guides/microservices-architecture): every service of the application in one namespace per environment Not published. An RDS instance alone takes 5 to 6 minutes, which [Instant Datasets](https://docs.release.com/reference-documentation/instant-datasets-aws) pre-warm Full stack per environment, in your AWS or GCP account. Enterprise pricing by usage. Environments expire and pause on schedules Database containers, or Instant Datasets: ready RDS, Aurora or Cloud SQL copies from production snapshots A Terraform runner, and ECS tasks linked by tag Full-stack previews with production-like data in your own cloud [Shipyard](https://docs.shipyard.build/): a full-stack environment per pull request from Docker Compose Not published per environment Sold by environments running at once. Retired after a period with no visits Database containers in each environment, seeded from the base environment's volume snapshot Not stated in its docs Docker Compose applications. [Acquired by StackGen](https://stackgen.com/blog/shipyard-joins-stackgen) in September 2026 Copy the whole environment per change: built in-house A namespace per pull request with an [Argo CD ApplicationSet](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Pull-Request/) or Helm Polls for pull requests every 30 minutes by default, or on a webhook, then the full deploy Full stack per environment, on your nodes. No license fee Whatever the chart deploys. Seeding and cleanup are yours to build Whatever your pipeline adds, such as a Terraform step Teams already on Argo CD with under about ten services [vCluster](https://www.vcluster.com/docs/vcluster/): a virtual cluster per environment, on shared nodes [Seconds to about a minute](https://www.vcluster.com/pricing) for the virtual control plane, by vCluster's own figures, plus the deploy Full stack plus a control-plane pod per environment. [Auto sleep](https://www.vcluster.com/docs/vcluster/key-features/sleep) and delete on Enterprise plans Whatever you deploy into it Not stated in its docs A full Kubernetes API per environment, for cluster-level changes Duplicate only what changed and plug it into a shared environment [Signadot](https://www.signadot.com/docs/concepts/sandbox): many ephemeral virtual environments in one shared environment. Each duplicates only what its change needs, joined in by request routing, message routing or configuration [Seconds](https://www.signadot.com/docs/overview) Grows with the services each change duplicates. [Usage-based, by environments created](https://www.signadot.com/pricing/). Every environment has a [TTL](https://www.signadot.com/docs/reference/sandboxes/ttl) Shared database with data kept apart, or temporary databases and Neon or Xata branches through resource plugins. Message routing on one shared Kafka, RabbitMQ, SQS, Pub/Sub or NATS broker. Routing key carried through Temporal workflows Resource plugins create cloud resources per environment, such as managed databases, queues and buckets, with Terraform, Helm, a cloud CLI or a script PR environments for dozens to hundreds of services, CI and coding agents [mirrord](https://metalbear.com/mirrord/docs/use-cases/preview-environments) Preview Environments (Enterprise): only new or changed services, as pods in a shared cluster, scoped by an environment key Not published Grows with the changed services. [Enterprise license by concurrent sessions](https://metalbear.com/mirrord/pricing). Each preview has a fixed TTL [Database branching](https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching) for many databases. [Queue splitting](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting) for SQS, Kafka, RabbitMQ, Pub/Sub, NATS, Temporal and others, with the queue name in configuration S3 branches created at the provider. Otherwise not stated in its docs Teams using mirrord for local development who want previews from the same tool [Okteto Divert](https://www.okteto.com/docs/core/divert): only the services you deploy, with the rest from a shared namespace, routed by a baggage header Not published. Its docs use 10 to 30 seconds to start developing as an example Grows with the services you deploy. Billed per seat Databases and queues shared by default. Queue isolation is your own consumer code filtering on the header As for Okteto previews Developer namespaces for large microservice applications Two kinds of tools sit next to this table rather than in it. Local development tools such as Telepresence and mirrord’s core product connect a process on a laptop or a CI runner to a cluster, the [ephemeral development environment](#ephemeral-development-environments) case above. Test runners such as Testkube run suites inside a cluster and work with any of these environments, and the [microservices testing tools comparison](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) lists them by layer. For one tool at a time, see [Okteto alternatives](https://www.signadot.com/comparison/okteto/), the [Release alternative](https://www.signadot.com/comparison/release/), [Qovery alternatives for preview environments](https://www.signadot.com/comparison/qovery/), [vCluster alternatives for PR environments](https://www.signadot.com/comparison/vcluster/), the [mirrord alternative](https://www.signadot.com/comparison/mirrord/) and [Signadot vs Telepresence](https://www.signadot.com/comparison/telepresence/). #### How ephemeral environments handle databases, message queues and workflow systems Ephemeral environments handle stateful dependencies in one of three ways: share the stable copy, give the environment its own short-lived copy, or share the resource and keep each environment’s rows, messages or workflows apart. The right one depends on whether the change reads state, writes it additively or changes its shape. This is where most rollouts stall. ##### Databases Most changes read from a database or write rows that other tests can ignore, and for those the shared stable database is fine, with each environment’s test data kept apart by a tenant, user or key prefix. A change that alters a schema or writes destructively needs its own copy: a temporary instance or schema seeded from a fixture, or a branch of a database that supports branching, such as [Neon](https://neon.com/) or [Xata](https://xata.io/) for Postgres, created and dropped with the environment. Full-copy platforms usually run a database container per environment, seeded by a script or restored from a snapshot, which is simple but repeats the data load for every pull request. [Testing database schema changes](https://www.signadot.com/docs/tutorials/testing-database-schema-changes) walks through the temporary-instance case and the [Neon tutorial](https://www.signadot.com/docs/tutorials/neon-database-branching) the branch-per-environment case, and Signadot’s [data isolation guide](https://www.signadot.com/docs/guides/set-up-data-isolation) covers the options. ##### Message queues: Kafka, SQS, RabbitMQ and others Queues break shared environments unless you plan for them, because a shared consumer will process a message an isolated producer published. There are two fixes. Give each environment its own topics or queues, which is the least work but cuts the environment off from the traffic the real system carries. Or isolate at the message level: producers put the environment’s routing key in message headers, each environment’s consumers run in their own consumer group and process only messages tagged for them, and the shared consumers skip those. Signadot’s [message queue isolation guides](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) cover Kafka, RabbitMQ, SQS and SNS, Pub/Sub and NATS, with a Routes API that tells each consumer which messages are its own, or a resource plugin that creates a topic or queue per environment. [Message isolation for Kafka, SQS and more](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) covers the pattern across brokers, and [testing Kafka-based microservices in Kubernetes](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) is the step-by-step version. ##### Workflow engines and batch jobs: Temporal and friends Workflow engines such as [Temporal](https://temporal.io/) and batch systems have the queue problem in a different shape: a worker polling a shared task queue will pick up any workflow, including one started by a test for someone else’s change. The fix is the same routing key. The workflow that a test starts carries the key, it passes the key to every activity, child workflow and job it starts, and the worker in that change’s ephemeral environment takes only the work tagged for it while the stable workers take the rest from the same task queues. Exactly one worker version processes each task, and no second Temporal cluster is needed. [Testing Temporal workers on Kubernetes](https://www.signadot.com/docs/tutorials/testing-temporal-workers) shows it end to end. ##### Resources tied to the environment’s lifecycle Some changes need a resource of their own that the cluster does not have: a managed database, a topic, a storage bucket, a cache or a set of credentials. The general mechanism is a resource whose lifecycle is tied to the environment’s. Signadot’s [resource plugins](https://www.signadot.com/docs/reference/resource-plugins) are create and delete steps a platform team writes once, with Helm, Terraform, a cloud CLI or a script. Signadot runs create when an ephemeral environment starts and delete when it ends, and hands outputs such as a host or credentials to the environment’s services as environment variables. [A new framework for ephemeral resources](https://www.signadot.com/blog/new-framework-for-ephemeral-resources/) explains the design. Full-copy platforms have their own versions: Bunnyshell’s Terraform components, Okteto’s external resources, Qovery’s lifecycle jobs, Northflank’s OpenTofu nodes and Release’s Terraform runner create cloud resources as part of an environment. ##### Workloads and resources outside Kubernetes Real systems reach past the cluster, and an ephemeral environment has to account for that. Two cases differ. Cloud resources that a service uses, such as a managed database, a queue, a bucket or a cache, can be created per environment and deleted with it, through a resource plugin or a Terraform step in the environment’s definition. Services that themselves run outside Kubernetes, on VMs, ECS or serverless functions, are harder to copy per change, and in most setups every environment shares them. Signadot installs and runs in Kubernetes, and the services whose changes it isolates run there. Its resource plugins reach past the cluster for the cloud resources an environment needs, created with your own Terraform or cloud CLI when the environment starts and removed when it ends. ##### Third-party APIs, caches and secrets Payment processors, identity providers and email services get the provider’s test account, with per-environment prefixes or idempotency keys so parallel environments do not collide on the same test customer. Redis, Memcached and S3-style buckets get a key prefix or path per environment and a cleanup hook on expiry, or a change that flushes a cache flushes it for everyone. Each environment gets its own scoped, rotated credentials, which [External Secrets Operator](https://external-secrets.io/) can issue at provision time and revoke at expiry. #### Do ephemeral environments need a service mesh? No. A mesh solves the routing half of duplicating only what changed on a shared cluster, and it does that well, but the harder half is context propagation inside the application, which no mesh can do for you. Full-copy models do not need a mesh at all. With [Istio](https://istio.io/) or [Linkerd](https://linkerd.io/) already in place, routing by header is a configuration change, which [using Istio or Linkerd to unlock ephemeral environments](https://www.signadot.com/blog/using-istio-or-linkerd-to-unlock-ephemeral-environments/) walks through. What the mesh cannot do is carry the header across a service: when service A receives a tagged request and calls service B, A’s own code or instrumentation has to copy the tag to the outgoing call. [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/), which follows the [W3C Baggage specification](https://www.w3.org/TR/baggage/), does that through the tracing middleware most services already run. A service that drops the header silently sends the request to the stable version, which is the bug to test for first. Teams on the Kubernetes [Gateway API](https://gateway-api.sigs.k8s.io/) get the same from routes placed beside their HTTPRoutes. Without a mesh, the platform that manages the environments adds a lightweight sidecar to the services that need it. A mesh already in place is a head start, not a requirement. #### What ephemeral environments cost The cost of ephemeral environments scales with one of two things: every service in the system, or only the services each change duplicates. Take a system of 40 services where a typical change touches two. A full copy per change runs 40 service instances per environment. Duplicating only what changed runs the 40 stable versions once and adds two instances per change. Concurrent environments Full copy per change Only what changed 1 40 instances 42 instances 10 400 instances 60 instances 30 1,200 instances 100 instances 100 4,000 instances 240 instances A full copy bills for concurrency, not for the size of a change: a one-line fix and a schema migration cost the same 40 instances, and 100 open pull requests pay for 100 copies of the system whether anyone looks at them or not. Duplicating only what changed has a fixed cost, the shared stable versions, which is why it is not cheaper at one environment, and the gap widens with every environment after the first. Brex ran namespace-based previews across more than 800 microservices, moved to duplicating only what changed, and [cut infrastructure cost by 99 percent, about $2 million a year, while preview creation fell from 30 to 60 minutes to under five](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/). In either model, a TTL that reclaims idle environments is the most effective single control, since most of an environment’s life is spent waiting for review. [What ephemeral environments cost, and how teams keep the bill under control](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) works through the cost drivers, the costs that never reach the invoice, the controls and the numbers at agent scale. #### What AI coding agents change about ephemeral environments Agents raise two numbers at once: the count of changes in flight, and the number of times each change needs an environment before it is done. An environment model sized for human throughput fails on both. - **Volume.** Agents are now a routine source of pull requests rather than an experiment. GitHub reported in May 2026 that [more than one in five code reviews on the platform involve an agent](https://github.blog/ai-and-ml/generative-ai/agent-pull-requests-are-everywhere-heres-how-to-review-them/), and the share of pull requests is higher still. A 50-engineer team built for 100 to 150 pull requests a day now faces closer to 1,000, as [the agent PR flood](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/) works through. - **The loop.** A developer opens one pull request and waits for CI. An agent iterates: it makes a change, runs the tests, reads the failure and changes again, and it wants an ephemeral environment for every one of those attempts, not one per feature. Claude Code, Cursor, Codex and Copilot all work this way, and a loop that has to wait for a slot in a staging channel stalls. [The agent feedback problem](https://www.signadot.com/blog/agent-loops-feedback-problem/) covers what that loop costs. - **Validation before review.** Testing AI-generated code in an ephemeral environment before a person looks at the pull request is what keeps the reviewer from becoming the verification layer. At Laurel, [agent-authored pull requests rose from 55 percent to 76 percent in three months while change failure rate fell from 1.1 percent to 0.2 percent](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/). The mechanics are in [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). - **Per-run isolation.** Put ten agent changes in one environment and a failing test names none of them, and an agent cannot fix what it cannot attribute. Each run needs its own environment, with its own identity in logs and traces. - **Expiry as policy.** Hundreds of runs a day stay affordable only if a run duplicates what it changed and nothing else, and every environment carries a TTL. An agent that creates environments and never deletes them is how a bill doubles in a week. - **The interface.** An agent creates, uses and destroys its environment through the tools it already has, a CLI, an [MCP server](https://www.signadot.com/docs/integrations/mcp) or [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills), not a console someone clicks. Background agents running in cloud sandboxes need one too, as the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial) shows. [The staging trap](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) covers what that path looks like inside an enterprise cluster, and [coding agent environments](https://www.signadot.com/solutions/agentic-development/) covers how Signadot packages it. #### Ephemeral environments vs a shared staging environment: which works better? For a monolith or a handful of services, a shared staging environment works better: it is cheap to run, stays current with main without effort, and a short queue costs less than the machinery to remove it. Past roughly 50 engineers and 25 services, ephemeral environments work better, and the setup that holds up keeps the shared staging cluster running the stable versions of every service while each change gets an ephemeral environment on top of it. At that size, teams can no longer stay in sync by talking to each other, the system no longer fits on a laptop, and several teams want staging at the same time. Above it, the shared environment stops providing the stage it was built for: the queue grows with concurrent changes and the environment drifts faster than anyone can fix it. Coding agents move a team past that point sooner, since each one adds the concurrency of several developers without adding headcount. There is also the argument for skipping pre-production testing and testing in production behind feature flags and canaries. That works for changes whose failure is observable and reversible in seconds, and not for schema migrations, payment flows or anything regulated. The stronger form of the argument is that the isolation a shared environment used to provide can be recreated per change inside production-like infrastructure, by duplicating only what each change needs. #### Rolling out ephemeral environments: pitfalls and a checklist Teams that make the transition without a stalled quarter follow the same sequence: 1. Audit how often teams queue for the shared environment, where drift bites and what the always-on footprint costs. 2. Pick a pilot team with frequent pull requests, real service dependencies and an existing CI pipeline. Do not attempt a big-bang migration. 3. Decide deliberately between building and buying. Lyft and Razorpay built in-house, and neither did it as a side project. 4. Plan the stateful dependencies before the pilot, not after the first corrupted test run. Four pitfalls account for most failed rollouts: - **Stateful and asynchronous dependencies.** Databases, queues, workflow engines and third-party APIs need the patterns above in place first. - **Observability.** Environments that disappear take their evidence with them. Persist logs and traces beyond teardown and tag every span with the environment’s identity. - **Culture and process.** QA teams used to long-lived environments need a documented path and an early win, which [preview environments for QA and stakeholder review](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) covers. - **Security and compliance.** Restrict access by role, keep personally identifiable information out of test environments, and rotate per-environment credentials. ##### Ephemeral environments checklist Twelve yes-or-no questions. A rollout that answers no to any of them has a known failure ahead. 1. Does every environment have a trigger and a matching teardown, so nothing is created that nothing deletes? 2. Does every environment carry a TTL that reclaims it without human action? 3. Does an environment cost only what the change duplicates, or does cost scale with the whole stack? 4. Can a developer, a CI job and a coding agent each create an environment through the same interface? 5. Is the shared set of stable services kept green by a pipeline, with a named owner when it goes red? 6. Does every service propagate the routing header on outgoing calls, with a test that fails when one drops it? 7. Do schema changes and destructive writes get their own ephemeral datastore, and does it die with the environment? 8. Are queue consumers and workflow workers isolated by environment, so a producer under test cannot feed the shared consumers? 9. Do logs and traces carry the environment’s identity, and do they outlive it? 10. Do test environments use sanitized data and per-environment credentials, with access limited by role? 11. Can a reviewer or QA engineer reach a running environment by URL without asking anyone? 12. Is there a number for environment count, cost and spin-up time, reviewed monthly? #### How Signadot duplicates only what changed Signadot is built on the fourth model, which duplicates only what changed. It multiplexes many ephemeral virtual environments within one physical environment, such as the staging cluster you already run. Each change gets one, called a Sandbox in Signadot’s API, that duplicates only what the change needs, usually the services it touched, next to the shared stable versions, plus its own database branch, topic or bucket when the change needs one. Each environment receives a unique routing key. Requests carrying that key in a header reach the forked services at every hop, messages carrying it reach the forked consumers, and everything else falls through to the stable ones. KUBERNETES CLUSTERSHARED STABLE SERVICESDev, agent or CIchange to service-1Other requestsservice-1CHANGED VERSIONservice-1stableservice-2service-3dbtagged request from the changeother traffic Only the changed version of service-1 is deployed for the change. Everything else it calls is shared. Dev, agent or CIchange to service-1Other requestsKUBERNETES CLUSTERservice-1CHANGED VERSIONservice-1stableSHARED STABLE SERVICESservice-2service-3dbtagged request from the changeother traffic Only the changed version of service-1 is deployed for the change. Everything else it calls is shared. The prerequisites of the model are handled in the platform: - **Routing** goes through the mesh you already run, as the [request routing docs](https://www.signadot.com/docs/guides/request-routing) describe. On [Istio](https://www.signadot.com/docs/guides/request-routing/istio) it is native, in sidecar or ambient mode, with no sidecar of its own. With the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api) it adds routes beside your HTTPRoutes and GRPCRoutes. With Linkerd or no mesh, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes requests. The routing key travels in OpenTelemetry baggage, which each service passes along. - **Data and dependencies** are isolated through [resource plugins](https://www.signadot.com/docs/reference/resource-plugins), which create a database branch, topic, bucket or other cloud resource for each environment when it starts and remove it when it ends, and through message routing for Kafka, RabbitMQ, SQS and SNS, Pub/Sub and NATS, where the Routes API tells each consumer which messages belong to its environment. The same routing key reaches Temporal workers and batch jobs. - **Scale** on one cluster comes from the operator design in [creating sandboxes in Kubernetes at scale](https://www.signadot.com/blog/creating-sandboxes-in-kubernetes-at-scale/). - **Local development** connects the service on a developer’s laptop into an ephemeral environment, the same model pull requests and agents use. - **The interface** is a CLI, an API, an MCP server and agent skills, so a developer, a CI job and a coding agent all get an environment the same way, and every environment carries a TTL. - **Deployment** is hybrid. The operator and the ephemeral environments run in your cluster, and a thin SaaS control plane coordinates them. By default the control plane receives only resource metadata, such as workload names, namespaces, image references and status, never source code, image contents or application data, as the [security and data docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) describe. Signadot is SOC 2 Type II. For a fully self-hosted deployment, [talk to us](https://www.signadot.com/schedule-a-call/). The [concept docs](https://www.signadot.com/docs/concepts/sandbox) describe the mechanism, and the [ephemeral environments solution](https://www.signadot.com/solutions/ephemeral-environments/) page covers how it is packaged. [Signadot vs Garden](https://www.signadot.com/articles/signadot-vs-garden-io/) sets it against a full stack per environment. #### Where to go next If you are still deciding whether a shared environment is enough, the [complete guide to staging environments](https://www.signadot.com/staging-environments/) makes the balanced case. If cost is the question, start with [what ephemeral environments cost](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/). [Environment parity](https://www.signadot.com/articles/environment-parity/) covers how any of these environments stays production-like. If the driver is coding agents, start with [validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/). For the tests that run inside these environments, layer by layer, see the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). Give every change its own environment Lightweight ephemeral environments on the cluster you already run, so developers, CI and coding agents test in parallel against real dependencies. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Related reading - [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) - [What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) - [Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) - [ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains](https://www.signadot.com/articles/argocd-preview-environments/) - [Per-PR Preview URLs, Explained](https://www.signadot.com/articles/per-pr-preview-urls/) - [Preview Environments for QA and Stakeholder Review](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) - [How Lyft and Razorpay share development environments with hundreds of developers](https://www.signadot.com/blog/how-lyft-and-razorpay-share-development-environments-with-hundreds-of-devs/) - [Smart Ephemeral Environments: Share More, Copy Less](https://www.signadot.com/blog/smart-ephemeral-environments-share-more-copy-less/) - [Creating Sandboxes in Kubernetes at Scale](https://www.signadot.com/blog/creating-sandboxes-in-kubernetes-at-scale/) - [Testing Microservices: Message Isolation for Kafka, SQS, More](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) - [A New Framework for Ephemeral Resources](https://www.signadot.com/blog/new-framework-for-ephemeral-resources/) - [Using Istio or Linkerd To Unlock Ephemeral Environments](https://www.signadot.com/blog/using-istio-or-linkerd-to-unlock-ephemeral-environments/) - [The Staging Trap: How to Unblock AI Coding Agents in Enterprise Kubernetes](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) #### Frequently asked questions What are ephemeral environments? Ephemeral environments are temporary, isolated environments created on demand for one code change, pull request or test run, and deleted when that work is done. Each gives the change its own place to run against real dependencies, so teams test in parallel instead of queueing for a shared staging environment. They are built either by copying the whole environment per change or by duplicating only what changed inside a shared environment. What are the best tools for ephemeral environments on Kubernetes? It depends on how much of the system each change should duplicate. Platforms that copy the whole application per pull request include Bunnyshell, Release, Qovery, Northflank, Shipyard and Okteto's previews, and platform teams build the same with Argo CD ApplicationSets, Helm or vCluster. Signadot duplicates only what each change needs on a shared cluster, as do mirrord's Enterprise preview environments and Okteto's Divert mode, which keeps cost and spin-up low as ephemeral environments multiply. Do ephemeral environments work better than a shared staging server? For a team past roughly 25 services and 50 engineers, yes. A shared staging server turns into a queue and drifts as concurrent changes pile up, while ephemeral environments give each change its own place to test. Most teams keep the staging cluster anyway: it runs the stable versions of every service, and each change's environment runs on top of it with only the services that changed. How do ephemeral environments handle stateful dependencies like databases and Kafka? Three patterns cover nearly every case. Share the stable database when a change only reads or adds rows. Give the ephemeral environment its own database branch or temporary instance when the change alters a schema or writes destructively. For Kafka and other queues, either create topics per environment or tag messages with the environment's routing key so its consumers process only their own. Workflow engines such as Temporal use the same key. How much do ephemeral environments cost at scale? It depends on whether cost grows with every service or only with the changed ones. For a 40-service system with 30 open pull requests, a full copy per change runs 1,200 service instances. Duplicating only what changed runs the 40 shared services once plus two per change, about 100 instances. At agent-driven volumes of hundreds of concurrent ephemeral environments, that gap decides whether the bill is affordable. How do ephemeral environments handle Temporal workers and task queues? With the same routing key that isolates requests. A workflow started by a test carries the key and passes it to every activity and child workflow. The worker in that change's ephemeral environment polls the same task queue as the stable workers but takes only tasks tagged for it, so exactly one worker version processes each task, and no second Temporal cluster or task queue per environment is needed. What is the difference between ephemeral and static environments? A static or persistent environment, such as staging, QA or UAT, is provisioned once, kept running and shared by every change in turn. An ephemeral environment is created for one change and deleted when that change merges or closes. The difference is ownership of time, not size or fidelity, and most designs need both: a static set of stable services, with ephemeral environments running against it. Do ephemeral environments require a service mesh like Istio? No. A service mesh such as Istio, or the Kubernetes Gateway API, makes request routing for ephemeral environments on a shared cluster straightforward, because both can already route by header. But isolation also needs every service to propagate the routing header on outgoing calls, which is application work no mesh can do. Full-copy models need no mesh at all, and platforms that route requests add a lightweight sidecar when no mesh exists. How do AI coding agents change ephemeral environment requirements? Agents multiply both the number of changes and how often each one needs an environment. A person opens a pull request and waits. An agent such as Claude Code or Codex writes, tests, reads the failure and retries, so it needs an ephemeral environment on every attempt and cannot wait in a queue. That volume only fits a model where each environment costs the one or two services that changed and expires on its own. Can a coding agent test its change in an ephemeral environment before opening a pull request? Yes, if the environment can be created through the same interface the agent already uses, which in practice means a CLI or an MCP server rather than a console. The agent deploys its changed service into an ephemeral environment on the shared cluster, runs the integration tests against real dependencies, reads the result and iterates. The pull request it opens has already been validated, so the human reviewer is not the verification layer. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Preview Environments: The Complete Guide Source: https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/ Summary: What a preview environment (PR environment) is, how it differs from staging and ephemeral environments, how one goes from pull request to URL, the four ways to build them, a dated comparison of preview environment platforms (Vercel, Netlify, Bunnyshell, Release, Okteto, Qovery, Northflank, Argo CD, Signadot), CI/CD workflows, monorepos, databases, message queues and Temporal, cost at scale, and what coding agents change. ### Preview Environments: The Complete Guide What a preview environment is, how it relates to staging and ephemeral environments, the four ways to build one, how the platforms compare, and what coding agents change about them. Author Arjun Iyer Published July 31, 2025 Updated October 11, 2026 Reading time 29 min A preview environment is a temporary, isolated environment created for a change, most often automatically for each pull request, where a person or a coding agent can try the change through the application’s web or mobile front end before it merges. For a pull request, it is created when the pull request opens, gets its own URL or routing key, and is deleted when the pull request closes. Developers and coding agents also create one earlier, during local development, to preview a change before any pull request exists. Teams also call it a PR environment or a pull request environment. Frontend platforms made the idea ordinary. Netlify and Vercel build every branch and post a link on the pull request, and reviewers got used to clicking through a change instead of reading it. For a frontend that is nearly free, because each build is a self-contained bundle. For a backend made of dozens of services, databases and message queues, the preview has to lead somewhere the whole system exists, and that is where the approaches split. The hard question is not whether to have preview environments but what each one copies. Copying the whole stack per pull request gives every reviewer a faithful system and a bill that grows with every service and every open pull request. Coding agents make that bill the constraint, because every pull request an agent opens wants its own preview. When a full copy takes 15 minutes or more to deploy, [1,000 pull requests a day](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/) turn previews into a queue, and the [CircleCI 2026 report](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) found that the teams pulling ahead are the ones whose environments absorb a tenfold surge in pull requests. What scales is selective isolation: each preview copies only what its pull request needs, usually just the changed services, and uses the real services, data and queues around them. This guide covers what a preview environment is and is not, how it differs from staging and from ephemeral environments, how one gets from pull request to URL, the four ways to build them, the platforms compared in one table, CI/CD workflows, monorepos, databases and message queues, what coding agents change, cost, and a checklist. A preview is one use of an ephemeral environment, the broader idea covered in the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/): the environment that serves a preview also runs automated frontend, backend, functional, security and performance tests. #### What is a preview environment? A preview environment is one change’s own running version of the application, usually a pull request’s, reached through its web or mobile front end by a URL or a routing key, and deleted when the change is done. It exists so a change can be tried, not only read. Previewing is manual testing, and manual does not mean human. A product manager clicking through a checkout flow and a coding agent driving the same flow in a browser or a mobile emulator, ad hoc, to see whether the change behaves are both previewing. What are preview environments for? Four jobs: - Reviewing behavior, not code. QA, product and design click through the change before it merges, which [preview environments for QA and stakeholder review](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) covers in depth. - Letting a coding agent explore its change through the web or mobile app the way a tester would, before a person looks. - Trying a frontend change against the real backend behind it, or a backend change through the screens that use it. - Showing a change to a customer or an executive from a link. What a preview is not: - Automated testing. Functional, security and performance suites run in the same ephemeral environment with nobody looking at it, which is testing, not previewing. - A way to test changes to the cluster itself, such as a new operator or CRD, unless each preview is its own virtual cluster. - The final release check. Every change that merged still meets the others after merge, in staging or in a canary. The same idea goes by several names. PR environment and pull request environment name the trigger. GitLab calls its version review apps, Netlify calls them Deploy Previews, and Vercel calls them Preview Deployments. Branch environment and per-PR environment are common in platform teams. Whatever the name, the meaning is one environment per pull request, deleted with it. ##### Preview environments vs ephemeral environments Every preview environment is an ephemeral environment, but an ephemeral environment does much more than previews. Ephemeral describes the lifecycle: created for one change and deleted after it. Preview describes one use of it: trying the change through the web or mobile front end, manually, whether a person clicks through or a coding agent explores ad hoc. The same ephemeral environment also runs automated tests of every kind, frontend and backend, functional, security and performance, from CI or from an agent’s loop, with nobody looking. The distinction matters when you pick tools. Preview tooling optimizes for a shareable URL and a comment on the pull request. Ephemeral environment tooling serves both uses: a URL that people and agents explore, and an API that test suites call. Signadot provides ephemeral environments, so the preview and the automated tests run against the same change in the same place. #### Preview environments vs staging: which to use? Use both, for different jobs. Staging is the shared, long-lived environment where the whole system runs at its current versions and merged changes are checked together. A preview environment is one pull request’s isolated view before merge, so the change is tested alone and nobody waits for a turn. Staging environment Preview environment Lifetime Long-lived Created for a change, usually when a pull request opens, deleted when it closes Shared by Every team and every merged change One change When testing happens After merge, or when a change takes its turn Before merge, on every push What a failure means Any recent change could be the cause The pull request is the cause Best for Release rehearsal, migrations, everything merged together Trying each change through the web or mobile app, by people and agents Teams go wrong when they treat it as a choice. Staging alone becomes a queue: changes wait for a turn, half-deployed work piles up, and failures stop being trusted, which the [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers in detail. Previews alone, when each one copies the whole stack, drift from each other and from staging, and the release still meets staging at the end. The model that works on Kubernetes runs preview environments on the staging cluster. Each preview deploys only the services its pull request changed, and its requests route through those versions while everything else is the staging environment itself. Staging stays the integration point, and every pull request gets its own preview on top of it. checkoutpreview for PR #142 paymentspreview for PR #143 searchpreview for PR #144 route through Staging cluster auth · orders · cartservices at current versions postgres · redisdatabases and caches kafka · temporalqueues and workflows Preview environments on top of staging: each pull request deploys only what it changed, and everything else is the shared staging environment. #### How does a preview environment work, from pull request to URL? A pull request event starts a pipeline that builds the change, creates the environment from a template kept in the repository, and posts how to reach it on the pull request. Closing the pull request deletes it. A preview can also come earlier. With Signadot, a developer or a coding agent creates one during local development, before any pull request exists: the service being changed runs on the laptop or in the agent’s workspace, everything else comes from the cluster, and the change can be previewed through the web or mobile app while it is still being written. PR opensPR #142 CI buildschanged images Preview createdfrom a repo template URL on the PRpr-142 Deletedon close, or by TTL 1. **Trigger.** The pull request opens or gets a new commit. GitHub Actions listens for the `pull_request` event, GitLab runs a merge request pipeline, and Argo CD’s pull request generator polls the repository. 2. **Build.** CI builds images for the services the pull request changed and tags them with the commit SHA. 3. **Create.** The environment comes from a template kept in the repository: a Helm chart or compose file for a full copy, or a spec that names only the changed workloads for a selective one. 4. **Connect.** Ingress and DNS give the preview a URL such as `pr-142.preview.example.com`, or a routing key carried in a header, a browser extension or a query parameter sends requests to it. 5. **Report.** The URL and the test results are posted on the pull request, so review starts from a running system. 6. **Delete.** Merging or closing the pull request deletes the environment, and a time-to-live deletes any preview that its pull request abandoned. The three hard parts are not in that list. They are what each preview copies, what data it sees, and how a request finds the right version of each service. The four approaches below are four answers to those three questions. [Per-PR preview URLs, explained](https://www.signadot.com/articles/per-pr-preview-urls/) covers the URL step for frontends and backends. #### The four ways to build preview environments There are four: frontend-only previews, a full-stack copy per branch, a namespace per pull request built in-house, and selective isolation on a shared cluster. They differ in what each preview copies, which sets its cost, how long it takes to start and how real its dependencies are. Frontend only Frontend buildcopied per PR Backend APIsshared, not previewed Full-stack copy Every servicecopied per PR Databases, queuesfresh copies, seeded Namespace per PR, DIY Every serviceHelm into pr-142 Databases, queuescopied or shared Selective isolation Changed servicecopied per PR Everything elsereal, shared, routed What each approach copies per pull request. The more it copies, the higher the cost and the longer the start. ##### Frontend-only previews Vercel and Netlify build the frontend for each branch, deploy it to a unique URL and comment the link on the pull request. Each build is a static or serverless bundle, so a thousand previews cost little more than one. Both have grown past the frontend: Netlify branches a Postgres database for each Deploy Preview, and Vercel has beta support for backend services, containers and queues that keep each deployment’s messages apart. The limit is where they run. A backend made of services on your own Kubernetes cluster is not in the preview, so the frontend calls a shared backend, often staging and sometimes production. That makes frontend platforms right for UI review and for apps built entirely on them, and wrong for a change to a service that lives on your cluster. Pointing a frontend preview at a backend preview closes the gap, which the [end-to-end previews with Vercel and Signadot tutorial](https://www.signadot.com/docs/tutorials/vercel-preview-environments) walks through. ##### Full-stack copy per branch Platforms such as Bunnyshell, Release, Okteto, Qovery, Northflank and Shipyard create a complete copy of the application for each branch or pull request by default, usually from a Docker Compose file, Helm charts or the platform’s own environment definition. Each copy gets its own services, databases and queues, typically in a Kubernetes namespace. The mental model is simple and isolation is total: nothing one preview does can touch another. The cost is linear in two directions. Twenty services and ten open pull requests is 200 running services plus ten of every database and broker, and every preview waits for all of them to start. Data is the other problem: each copy starts empty unless someone maintains seed data, and seed data drifts from the real thing. Full copies fit a handful of services, and they strain as service count and pull request volume grow. A variation, a virtual cluster per pull request with [vCluster](https://www.signadot.com/comparison/vcluster/), gives each preview its own Kubernetes control plane, which is the right tool when a pull request changes operators, CRDs or cluster configuration. ##### Namespace per pull request, built in-house Platform teams that run GitOps often build previews themselves: an Argo CD ApplicationSet with the pull request generator stamps out an Application for each open pull request, Helm renders the stack into a namespace named for it, and the generator removes the Application when the pull request closes. Jenkins X, now renamed JayeX, packages the same idea as a preview namespace per pull request. Everything is open source and auditable in Git, and the team controls every detail. It is a full copy, so it inherits the cost curve above, and the team also owns what a product would handle: cleanup of resources Argo CD does not track (databases, buckets, DNS records), secrets for every namespace, and the pipeline that keeps it all working. [ArgoCD preview environments](https://www.signadot.com/articles/argocd-preview-environments/) walks through the ApplicationSet pattern and where it strains. ##### Selective isolation on a shared cluster Selective isolation copies only what a pull request needs and plugs it into the real environment around it. Signadot is built on it, and its version is the most complete: each preview is an ephemeral virtual environment in one shared environment: the changed workloads are forked onto the existing cluster, and a routing key decides which requests reach them. Requests that carry the key, in an OpenTelemetry `baggage` header that services propagate from call to call, go to the forked version. Requests without it go to the shared version. A person reaches a preview through a preview URL, the Chrome extension or a query parameter, and a test or an agent sets the header. You choose how much each preview duplicates. The default copies only the changed services. A preview can add its own database, a topic or other resources when a change needs them, and in the extreme it can copy everything. The changed parts join the application’s flows in three ways: request routing for synchronous calls, message routing for asynchronous and batch work, and configuration that points a service at a new version. [What a Kubernetes sandbox is](https://www.signadot.com/kubernetes-sandbox/) explains the mechanism. It has requirements. Services must propagate the routing header, which OpenTelemetry instrumentation already does for most stacks. A change that rewrites a shared schema needs its own database for the preview, created by a resource plugin. Consumers of a shared queue need to check the routing key before they process a message. In return, a preview starts in seconds because nothing else has to start, hundreds share one cluster, and every preview runs against the same real services, data and queues as staging. Signadot runs mostly inside your cluster, with a hosted control plane that by default receives only resource metadata, such as workload names, namespaces, image references and status, and never source code or application data, as its [data and security documentation](https://www.signadot.com/docs/guides/admin/security/data-integrations) describes. #### Preview environment platforms compared This table compares the preview environment platforms buyers evaluate most, grouped by approach. “Not documented” means the vendor’s docs do not describe it, not that it is impossible. Preview environment platforms compared, October 2026 Platform What gets previewed Approach Databases Queues and workflow systems Workloads outside Kubernetes Monorepo and multi-service Best fit Selective isolation on a shared cluster Signadot Only what the pull request needs, from one changed service to the whole stack Many ephemeral virtual environments in one shared environment on your cluster, joined in by request routing, message routing or configuration Shared with data kept apart, or temporary databases and Neon or Xata branches through resource plugins Message routing on one shared Kafka, RabbitMQ, SQS, Pub/Sub or NATS broker, and routing keys carried through Temporal workflows Resource plugins create managed databases, queues and buckets per preview with Terraform, Helm, a cloud CLI or a script Several changed services per preview, RouteGroups join pull requests across repos Any application with a backend: dozens to hundreds of services, CI and coding agents mirrord previews Only new or changed services (Enterprise plan) Pods in a shared cluster, scoped by an environment key Database branching for many databases Queue splitting for SQS, Kafka, RabbitMQ, Pub/Sub, NATS and Temporal, with the queue name in configuration S3 branches, otherwise not documented Not documented Teams using mirrord for local development who want previews from the same tool Okteto Divert Only the services you deploy The rest from a shared namespace, routed by a baggage header Shared by default Shared by default, queue isolation is your own consumer code filtering on the header Deploy and destroy commands in the manifest Not documented Developer namespaces for Okteto users Frontend platforms Vercel The frontend per branch and commit, with backends in beta Preview Deployments on Vercel, not on your cluster Marketplace databases, a Neon branch per preview Vercel Queues (beta), partitioned by deployment Marketplace resources only A project per directory, unchanged projects skipped Frontend and full-stack JavaScript on Vercel Netlify The site per pull request Deploy Previews on Netlify, not on your cluster A Netlify Database branch per preview, copied from production Async Workloads, preview behavior not documented Not documented A site per package, skipping needs an ignore command Static sites and frontends Full-stack copy per branch Bunnyshell The whole stack in bunnyshell.yaml A namespace per environment, on your cluster or Bunnyshell's Seeded containers, Terraform, Neon, PlanetScale and Xata connectors Run as containers or Helm charts in each copy A Terraform component per environment Components across repos, a monorepo rebuilds every component by default Full copies of multi-repo stacks on your own cluster Release The whole stack in an application template A namespace per environment, in Release's cloud or your AWS or GCP Containers, snapshot-based Instant Datasets, Neon Not documented for previews Terraform per environment Monorepo or multi-repo, every service rebuilt AWS and GCP teams that want production-like data Okteto The whole manifest per preview A namespace per preview, self-hosted, BYOC or SaaS Restored from volume snapshots Run inside each copy Deploy and destroy commands in the manifest Dependencies across repos, unchanged images reused Kubernetes teams that want dev and preview environments Qovery A clone of a blueprint environment, trimmable to fewer services A namespace in your own cloud account Cloned or shared per preview, cloned without their data and seeded by jobs Blueprints for MSK, RabbitMQ, Redpanda and a Temporal Cloud namespace Terraform, OpenTofu and CloudFormation through lifecycle jobs A monorepo guide with path rules A deployment platform on your own cloud account, with previews Northflank A preview template per pull request, self-contained or sharing services with dev Northflank's cloud, your cloud or your cluster Addons forked per preview, or a fresh addon and a seed job A RabbitMQ addon, Kafka and Temporal not documented OpenTofu nodes, destroyed explicitly Path rules and a build context A general Kubernetes platform with templated previews Shipyard The whole stack in a Docker Compose file A full copy per pull request on Kubernetes Volume snapshots per named volume Not documented Not documented Multi-repo supported Compose apps that want hosted full copies vCluster Everything deployed into a virtual cluster A virtual cluster per pull request Whatever you deploy inside it Whatever you deploy inside it Not documented Not documented Changes to operators, CRDs and cluster configuration Namespace per pull request, built in-house Argo CD and Helm The whole chart per pull request An Application per open pull request from the ApplicationSet pull request generator You build it You build it You build it You build change detection Platform teams that run Argo CD and want to own the system Jenkins X (now JayeX) The whole helmfile per pull request A preview namespace per pull request, cleaned up by jx preview gc You build it You build it You build it You build it Teams already on Jenkins X Signadot is the most comprehensive preview environment platform in this table. A preview holds only what its pull request needs, from one changed service to the whole stack, on the cluster you already run. Databases, message queues, Temporal workflows and cloud resources are isolated per preview, pull requests across repositories join into one preview, and CI jobs, people and coding agents all reach the same previews. Frontend platforms stop at the frontend. Full-copy platforms pay for the whole stack on every pull request and run queues and workflow engines as more copies. The other selective options cover part of the problem: mirrord’s previews are an Enterprise add-on to a local development tool, and Okteto’s Divert shares databases and queues and leaves their isolation to your own code. For one vendor at a time, see the [Okteto alternatives](https://www.signadot.com/comparison/okteto/), [Release alternative](https://www.signadot.com/comparison/release/), [Qovery alternatives for preview environments](https://www.signadot.com/comparison/qovery/), [vCluster alternatives for PR environments](https://www.signadot.com/comparison/vcluster/) and [mirrord alternative](https://www.signadot.com/comparison/mirrord/) pages. #### Which preview environment platform should you choose? For any application with a backend, choose Signadot. It is the most comprehensive preview environment platform: every pull request gets a preview on the cluster you already run, with real services, databases, queues and workflow engines behind it, isolated where the change needs it and shared everywhere else. You choose how much each preview duplicates, so the same platform covers a one-service change, a change that needs its own database, and a change that needs the whole stack. The other tools fit narrower cases: - **A frontend or a static site with no backend of its own:** Vercel or Netlify. Their previews are built in, cheap and fast. - **A few containers that will stay a few containers:** a full-copy platform such as Bunnyshell, Release, Qovery or Northflank works, as long as every preview can pay for the whole stack. - **A platform team set on building its own on Argo CD:** an ApplicationSet with the pull request generator and Helm, with a budget for cleanup, secrets, data and maintenance. - **Changes to operators, CRDs or cluster configuration:** a virtual cluster per pull request. Containers rather than Kubernetes do not change the answer. A small containerized application runs well as a full copy on a platform that builds from a compose file. Once the containers number in the dozens and share databases and queues, copying them per pull request is the expensive part, and the platform that copies only what changed wins on cost, start time and fidelity. Preview every pull request on the cluster you already run Signadot gives every pull request a preview environment with your real databases, queues and services behind it. Only what the change needs deploys, so previews start in seconds. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### CI/CD workflows that create preview environments Every working setup has the same four hooks: create on pull request open, update on each push, post the URL, delete on close. What differs is where those hooks live. - **GitHub Actions:** a workflow on the `pull_request` event for the opened, synchronize and reopened types creates or updates the preview, and the closed type deletes it. - **GitLab CI:** review apps. A job declares a dynamic `environment` per merge request, a stop job named in `on_stop` deletes it when the merge request closes, and `auto_stop_in` deletes previews nobody closed. - **Argo CD:** the ApplicationSet pull request generator creates an Application per open pull request and removes it when the pull request closes, so the hooks live in the cluster instead of in CI. - **Jenkins:** a multibranch pipeline builds each pull request as a branch and calls a CLI to create and delete the environment. - **Signadot:** CI applies a sandbox spec from the repository with `signadot sandbox apply`. On GitHub, the Signadot GitHub App posts the preview details on the pull request and deletes the sandbox when it closes. On GitLab and Bitbucket, a time-to-live on the sandbox handles teardown. The [CI integration docs](https://www.signadot.com/docs/guides/integrate-ci) cover each. ##### Clean up preview environments automatically Abandoned previews are the most common cost leak. The cleanup practices that hold up: - Delete on the pull request’s close event, not on a schedule alone. - Set a time-to-live on every preview as a backstop for missed events and stale branches. - Label every resource with the pull request number, so a sweep can find what a failed teardown left behind. - Run a periodic job that deletes any preview whose pull request is no longer open. - Delete resources outside the cluster, such as databases, buckets and DNS records, in the same teardown that deletes the preview. ##### Secrets in preview environments Previews should never receive production credentials. Full copies need credentials for every service in every namespace, usually issued by a secrets manager such as Vault or the External Secrets Operator with a path per environment and short lifetimes. Selective previews reuse the staging environment’s secrets for the shared services and need new credentials only for the resources they create, which shrinks the problem to the parts that changed. #### Preview environments for monorepos Build only what changed and run it as one preview. A monorepo pull request often touches two or three services at once, so the preview has to hold all of them together, and building every service on every pull request wastes most of the CI run. - **Find the changed services.** Path filters in the CI workflow work for a few services. For more, use the build tool’s affected graph: Nx affected, Turborepo filters or a Bazel query. - **Build those images only,** tagged with the commit SHA. - **Create one preview per pull request, not one per service.** The preview holds every changed service, so a change that spans the API and a worker is tested as one change. - **Share the rest.** Unchanged services, databases and queues come from the shared environment instead of being rebuilt. A Signadot sandbox spec takes a list of forks, so one preview can hold every service a monorepo pull request changed. When a change spans several repositories or several pull requests, a RouteGroup gives those sandboxes one routing key, so they are tested together before any of them merges. A full-copy platform previews a monorepo by copying all of it, which is simple for a few services and slow for many. #### Preview environments with databases, message queues and workflow engines Data and dependencies decide whether a preview is worth opening. A preview backed by an empty database demos nothing, and a preview whose consumer reads every message on a shared queue breaks everyone else’s tests. Each dependency has the same three choices: copy it, branch it, or share it and isolate only what the change touches. ##### Databases - **A fresh copy per preview,** seeded from fixtures or a snapshot. Full isolation, slow to start, and only as realistic as the seed data. - **A branch per preview.** Databases that branch, such as Neon and Xata, create a copy-on-write branch in seconds. Bitso pairs [Signadot sandboxes with Neon branches](https://www.signadot.com/blog/how-bitso-is-scaling-branch-based-development-with-signadot-and-neon/) so a sandbox that needs its own data gets a database branch instead of a full copy of the stack. - **Share the database, isolate the writes.** Most changes read and write rows, not schemas, so the preview uses the shared database with test data partitioned by tenant or account. A schema change gets its own database, created by a resource plugin for that preview only, as [database isolation with resource plugins](https://www.signadot.com/docs/tutorials/testing-database-schema-changes) shows. ##### Message queues Kafka, RabbitMQ, SQS and Pub/Sub break naive previews because a consumer on a shared topic takes every message. The options are a broker per preview, a topic or queue per preview created when the preview is created, or message routing on the shared broker: the producer puts the routing key in message headers, and each consumer processes only the messages for its version. Message routing keeps one broker and one topic set for every preview. [Message isolation for Kafka, SQS and more](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) covers how the pattern maps onto each broker, and the [Kafka testing guide](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) goes deep on one. ##### Workflow engines such as Temporal Workflow engines add a twist, because workers pull tasks from a task queue rather than receiving requests. With Signadot, the stable worker and the preview’s worker share one task queue, the routing key travels with the workflow in OpenTelemetry baggage, and interceptors on each worker check which version should handle a task, so exactly one version processes it. The [Temporal workers tutorial](https://www.signadot.com/docs/tutorials/testing-temporal-workers) shows it in Python, TypeScript, Go and Java. ##### Resources outside the cluster Some dependencies are not containers at all: a managed database, a cloud queue, a storage bucket. Resource plugins create those per preview and delete them with it, using Terraform, Helm charts, cloud CLIs or scripts, and pass connection details to the forked services as environment variables. Services that run outside Kubernetes take part as shared dependencies of the environment, not as per-preview copies. #### What AI coding agents change about preview environments Coding agents turn preview environments from a review convenience into infrastructure that every change depends on. Throughput is up and quality is not keeping pace: the [CircleCI 2026 report](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) found average throughput grew 59 percent in a year, the top 5 percent of teams nearly doubled theirs while the bottom quarter saw no gain, roughly 30 percent of merge attempts now fail, and median recovery rose 13 percent to 72 minutes. The teams pulling ahead are the ones whose validation scales with the pull requests their agents open. Agents are consumers of these environments, not only producers of pull requests, and they use them both ways. An agent’s build, test and fix loop runs automated tests in an ephemeral environment on every iteration, often dozens per change, and between runs it previews the change, driving the web or mobile app ad hoc the way a tester would. Neither can wait for a person to create an environment or for a shared one to be free. A preview that takes 20 minutes to start ends the loop before it begins. That moves validation ahead of review, and it is what testing AI-generated code in a preview environment means in practice. When Claude Code, Cursor, Codex or Copilot can create a preview, run the change against real services and read the result, it fixes its own failures before a person looks. The reviewer receives a pull request with evidence attached and stops being the test harness. Isolation has to be per run. Batching several agents’ changes into one shared environment brings back the attribution problem staging has: when the batch fails, nobody knows which change caused it, and every agent retries. Each run needs its own preview. Cost and cleanup have to be automatic. Hundreds of concurrent previews are affordable only if each one copies little, and only if every one has a time-to-live, because agents abandon work more often than people do. The path an agent takes matters as much as the environment. With Signadot, an agent creates a sandbox through the CLI or the MCP server, connects its local run to the cluster with a local sandbox, and runs the checks the validate skill defines. [Validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) covers the full loop, and [scaling coding agents in enterprise Kubernetes](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) covers what changes at hundreds of agents. #### What do preview environments cost at scale? Cost is what each preview copies, times the number of previews open, times how long they live. A full copy costs the number of services times the number of open pull requests. A selective preview costs only the changed services times the open pull requests. Take 40 services and 30 open pull requests, each changing two services. Full copies run 1,200 service deployments plus 30 of every database and broker. Selective previews run 60 deployments on one shared environment. Spin-up time follows the same curve: when a full copy of 40 JVM services takes 15 minutes to start, the fix is starting fewer services, not faster ones. The numbers hold up in practice. Brex [saves about $2 million a year in infrastructure](https://www.signadot.com/case-studies/how-brex-transformed-developer-experience-and-slashed-infrastructure-costs-with-signadot/) after replacing its homegrown full-copy preview environments with Signadot, and its engineers [report deploys of under five minutes](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) against the 30 to 60 minutes the old previews took. [Bitso replaced a homegrown system](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) that duplicated 200 services, about 100 databases and 20 caches for every pull request. [Reducing costs with Kubernetes ephemeral environments](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) works through the cost model in more detail, and [cost-efficient preview environments in Kubernetes](https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/) shows the setup step by step. #### Preview environment best practices checklist - Does every pull request get a preview automatically, with no ticket and no request? - Is the preview created from a template kept in the repository, reviewed like code? - Does the preview hold every service the pull request changed, and only those? - Does it run against real versions of the services it depends on, not mocks? - Does each preview have realistic data, through shared data with isolated writes, a branch, or maintained seed data? - Do consumers of shared queues and workflow task queues process only their own preview’s messages? - Are the URL and test results posted on the pull request? - Can a person reach the preview from a browser and a test reach it from CI by the same key or URL? - Is the preview deleted when the pull request closes, with a time-to-live as a backstop? - Does every preview get its own short-lived credentials, never production secrets? - Can a coding agent create a preview by CLI or API in seconds, without a person? - Do you know what a preview costs, and does that cost grow with the changed services rather than the whole stack? #### Where to go next If you are choosing a platform, start from the comparison table above and the [Kubernetes preview environments](https://www.signadot.com/solutions/preview-environments/) page for how Signadot does it. If your previews already exist and cost or start time is the problem, read the cost section and the [isolation models comparison](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/). If coding agents are the reason you are here, go to the [guide to validating AI-generated code](https://www.signadot.com/validate-ai-generated-code-kubernetes/). #### Related reading - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [Per-PR Preview URLs, Explained](https://www.signadot.com/articles/per-pr-preview-urls/) - [ArgoCD Preview Environments: The ApplicationSet Pattern](https://www.signadot.com/articles/argocd-preview-environments/) - [Preview Environments for QA and Stakeholder Review](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) - [Microservices Testing: Feature Flags vs. Preview Environments](https://www.signadot.com/blog/microservices-testing-feature-flags-vs-preview-environments/) - [Cut the Cost of Kubernetes Preview Environments with Signadot](https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/) - [Tutorial: End-to-End Hot-Reload-Style Previews with Vercel + Signadot](https://www.signadot.com/docs/tutorials/vercel-preview-environments) - [Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) - [What Is a Kubernetes Sandbox?](https://www.signadot.com/kubernetes-sandbox/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [The Complete Guide to Microservices Testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) #### Frequently asked questions What is a preview environment? A preview environment is a temporary, isolated environment created for a change, usually automatically for each pull request, so a person or a coding agent can try it through the application's web or mobile front end before it merges. Developers and agents can also create one during local development, before a pull request exists. Teams also call it a PR environment or a pull request environment. Should we use preview environments or a staging environment? Most teams need both. Staging is the long-lived environment where the whole system runs at its current versions and merged changes are checked together. A preview environment is one pull request's isolated view before merge, so the change is tested alone and nobody waits for a turn. On Kubernetes, preview environments can run on the staging cluster itself, deploying only what each pull request changed. What is the difference between preview environments and ephemeral environments? Every preview environment is an ephemeral environment, but ephemeral environments do more. A preview is manual testing through the web or mobile front end, by a person or by a coding agent exploring the change ad hoc. The same ephemeral environment also runs automated frontend, backend, functional, security and performance tests from CI or an agent's loop. Signadot provides ephemeral environments, so previews and automated tests share one environment per change. What are the best platforms for preview environments? For any application with a backend, Signadot is the most comprehensive preview environment platform. Each preview holds only what its pull request needs on the cluster you already run, with databases, message queues, Temporal workflows and cloud resources isolated per preview, and the same previews serve CI and coding agents. Vercel and Netlify cover frontends. Full-copy platforms such as Bunnyshell, Release, Qovery and Northflank suit a handful of services. How do you create a preview environment for every pull request? Trigger it from the pull request. A CI workflow on the pull request event builds the changed images, creates the preview environment from a template kept in the repository, posts the URL on the pull request, and deletes the environment when the pull request merges or closes, with a time-to-live as a backstop. GitHub Actions, GitLab CI, Jenkins and Argo CD's pull request generator all support this pattern. Do preview environments work for monorepos? Yes, if each preview environment builds only what the pull request changed. Path filters or a build tool's affected-project graph, such as Nx, Turborepo or Bazel, find the changed services. CI builds those images and creates one preview per pull request that runs them together against everything else. Copying every service in the monorepo for each pull request works for a handful of services and gets slow and expensive beyond that. How do preview environments handle databases and message queues? There are three choices for each dependency. A preview environment can run its own fresh, seeded copy, branch the data per pull request with a tool such as Neon, or share the existing database and broker and isolate only what the change writes or consumes. For queues, sharing means message routing: the routing key travels in message headers and only the preview's consumers process its messages. What do preview environments cost at scale? Cost follows how much each preview environment copies. A full-stack copy costs roughly the number of services times the number of open pull requests. A selective preview costs only the changed services times the open pull requests, usually a small fraction. In its published case study, Brex reports saving about $2 million a year in infrastructure after replacing full-copy previews with Signadot. Deleting on close and a time-to-live keep abandoned previews from piling up. How do AI coding agents change preview environment requirements? Agents raise both the number of pull requests and the number of attempts per change. A build, test and fix loop needs a preview environment on every iteration and cannot wait for a person to provision one. Previews for agents have to be created by API or CLI in seconds, isolated per run, cheap enough to run hundreds at once, and deleted automatically when the run ends. Can a coding agent test its change in a preview environment before opening a pull request? Yes. Agents such as Claude Code, Cursor, Codex and Copilot can call a CLI or an MCP server to create a preview environment that holds only the services they changed, explore the change through the web or mobile app, run automated tests against the real services and data around it, and fix the change before a person reviews it. The pull request then arrives with evidence that the change works, not only a diff. #### Related Articles - [ *Image: Preview Environments for QA and Stakeholder Review* ### Preview Environments for QA and Stakeholder Review August 5, 2026 ](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) - [ *Image: Per-PR Preview URLs, Explained* ### Per-PR Preview URLs, Explained July 21, 2026 ](https://www.signadot.com/articles/per-pr-preview-urls/) - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Staging Environments: The Complete Guide Source: https://www.signadot.com/staging-environments/ Summary: What a staging environment is for, why the single shared copy becomes a bottleneck at microservices scale, and the four alternatives compared on isolation, fidelity, spin-up time, and cost. ### Staging Environments: The Complete Guide What a staging environment is for, why one shared copy becomes a bottleneck at microservices scale and under coding agent volume, and the alternatives teams use instead, compared. Published September 3, 2026 Reading time 28 min A staging environment is the last stop before production: a copy of the system, built from the same artifacts and configuration, where a change runs against the real versions of its dependencies before real users see it. Unit tests and mocks check a service against assumptions about its neighbors. Staging checks it against the neighbors themselves, which is why nearly every team that ships software runs one. Staging comes from the era of the monolith and the release train. One application, one database, one deployable and a release every few weeks meant a single pre-production copy was cheap to run and rarely wanted by two people at once. The practice carried over unchanged into microservices, where the system is now dozens or hundreds of independently deployed services and the release is continuous. That is where the standard model stops scaling. The standard model is one staging environment that every developer deploys their changes into, so it can hold one integrated state of the system at a time. As teams grow, changes queue behind each other, the environment fills with half-finished work, and results stop being trusted. Coding agents multiply the number of changes that need validation: a 50-engineer team whose pipeline was built for 100 to 150 pull requests a day now faces closer to 1,000, and [average delivery throughput rose 59 percent in a year](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/). A queue that was tolerable at human speed becomes the constraint on delivery. The [complete guide to software factories](https://www.signadot.com/software-factory/) describes the same constraint as a feedback problem: a sensor that can take only one reading at a time. The stage is still necessary. What has to change is the assumption that staging means one environment everyone deploys into. This guide covers what staging is for, where it sits in the delivery pipeline, why that model breaks under load, what coding agents change about it, the fixes that do not work, and the four architectures that do, followed by how to run the staging environment you keep. Staging is one stage in the longer path covered by [The Complete Guide to Microservices Testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### What is a staging environment, and what is it actually for? A staging environment is a pre-production environment that mirrors production closely enough that a change passing there is expected to be safe to deploy. It is built from the same deployment artifacts and configuration as production, and it answers one question: does this change behave correctly against the real versions of everything it depends on. Tests that pass there are treated as a release gate. Staging exists because unit tests and mocked integration tests check a service against an assumption about its neighbors. Staging checks it against the neighbors themselves: the current version of the auth service, the actual schema of the orders database, the real message broker with its retry behavior, and third-party integrations in their sandbox modes. What staging is for: - Integration of many services at their current versions, including contract changes that mocks hide. - Configuration and infrastructure parity: the same ingress rules, network policies, secrets, resource limits, and Kubernetes version as production. - Release rehearsal: database migrations, rollout strategy, rollback. - Performance and load checks that need production-shaped data and real network hops. - Validation of agent-generated changes against real dependencies before a person reviews them. - Review of a complete feature by product owners or QA before it ships. What staging is not for: - Unit-level bugs. Those belong in the developer’s loop and in CI. - Individual iteration. A shared environment cannot host one engineer, or one coding agent, trying a change ten times an hour. - Proving a change is correct in isolation. Staging proves it is correct in company. Staging server, staging site, pre-prod, preprod, and pre-production environment are the other names for the same thing, with one distinction covered in the next section: some teams keep preprod as a separate, stricter final gate. Two different setups get called a shared staging environment, and the difference is the subject of this guide. In the first, every developer and agent deploys their change into the same environment, so it holds everyone’s work in progress at once and everyone competes for it. In the second, one environment holds the stable version of every service, each change deploys only the services it touches, and only that change’s test traffic sees them. The first stops scaling somewhere past a few dozen services. The second is what replaces it, and it is still one staging environment. #### Where staging sits: development, staging, and production Staging is the pre-production environment: the last place a change runs before it reaches production, and the first place it runs against the real versions of everything it depends on. Most teams run three kinds of environment. Development is where a change is written and run on its own, staging is where it meets the integrated system, and production is where real users hit it. Each step up trades speed for fidelity. The other environment names a reader will meet, test, integration, QA, UAT, pre-prod, and lower environments, are usually either another name for staging or an extra pre-production environment a team has split out of it. Environment What it is Tests that run there Dependencies Who uses it Development A developer's laptop or a cloud development environment (CDE), and now the working environment of every coding agent Unit tests, running and debugging the one service being changed Mocked, stubbed, or a handful of neighbors run alongside Individual developers and their agents Staging (pre-production) A deployed, production-like copy of the whole system Integration and end-to-end tests, QA and exploratory testing, user acceptance, performance checks, release rehearsal Real versions of every internal service, third-party integrations in sandbox mode Developers, CI pipelines, QA, product owners, release managers Production The system real users hit Canary and progressive rollout, synthetic checks, monitoring and alerting Real Everyone, indirectly The pattern in the table is the problem the rest of this guide is about. Everything before staging tests a service against assumptions about its neighbors. Staging is the first place it meets the neighbors themselves, so every test that needs real dependencies lands there: the suites CI kicks off, the branch a developer or agent wants to try before merge, QA’s exploratory pass, the product owner’s sign-off. One environment carries all of it, for every team at once. Some teams run more than one pre-production environment: a QA copy testers own, a UAT copy business users accept against, a pre-prod that takes only release candidates at production scale. These are all the same kind of environment as staging, and the names describe who uses the copy rather than what it is. Each one is another full dependency graph to run, sync and load with data, which is why most teams stop at one or two. ##### Staging vs production The difference is where the traffic comes from. Production serves real users with real data, and a failure there costs money and trust. Staging serves the team’s own test traffic against the same code and configuration, so a failure there costs only the time to fix it. Staging is usually scaled down in replica counts and node sizes, never in configuration or dependency fidelity. Is pre-prod the same as staging: at most companies, yes. Pre-prod and pre-production environment are simply other names for staging. Where a team keeps both, pre-production is the stricter one. It runs at production capacity, takes only release candidates rather than individual branches, and may be the target of a final production data clone or a canary rehearsal. DevelopmentStagingProductionunit, component(mocked)integration, end-to-end,QA, UAT, performancecanary,monitoring Every test that needs real dependencies lands on one staging environment. Developmentunit, component(mocked)Stagingintegration, end-to-end,QA, UAT, performanceProductioncanary, monitoring Every test that needs real dependencies lands on one staging environment. #### Why shared staging becomes a bottleneck Shared staging, as the term is used here, means one staging environment that every developer and every coding agent deploys work in progress into. That environment can hold one integrated state of the system at a time, so every change in the organization has to pass through it in series. As service counts and change volume grow, three failures appear together and feed each other, and coding agents amplify all three. ##### Everyone deploys to it at once Shared staging has no admission control. Any team, or any agent, can deploy any branch to it at any time, and at scale every one of them does. CI adds its own load: the integration and end-to-end suites in the pull request pipeline have nowhere else to run, so every pipeline points them at staging, and each run’s result depends on whatever happened to be deployed there at the time. Moving those suites to the pull request is the substance of [shift-left testing for microservices](https://www.signadot.com/articles/shift-left-testing/), and it needs an environment per change rather than a shared one. The environment fills with a mixture of half-finished changes from different teams, none of which is in production and most of which will never ship together. One team deploys a checkout change, another deploys an inventory change, and a third team’s test fails on an interaction between the two. Test results stop meaning anything. A red run might be your bug, someone else’s bug, or a collision between two changes that are both fine on their own, so engineers rerun, override, or skip failing tests. ShareChat hit this at more than 100 microservices and more than 300 engineers, with no way to test features independently before production short of building an in-house system. The [ShareChat case study](https://www.signadot.com/case-studies/sharechat-chooses-signadot-giving-devs-high-quality-testing-feedback/) documents what changed once each change got its own isolated view of the cluster: more frequent deployments with fewer rollbacks. ##### Teams block each other waiting to test The alternative to deploying over each other is taking turns, and turn-taking makes shared staging a queue. Only one team’s change can be validated cleanly at a time, so everyone else waits. [Little’s law](https://en.wikipedia.org/wiki/Little%27s_law) describes the result: the number of changes waiting equals the rate at which changes arrive multiplied by the time each spends in the environment. Doubling the team doubles arrivals, and the time each change occupies staging does not fall, so the queue grows until people route around it. The visible signs are booking spreadsheets, a Slack channel named staging-lock, and “is anyone using staging” messages. The real cost is engineer time, and [Why Shared Staging Is the Most Expensive Tool You’re Not Accounting For](https://www.signadot.com/deep-dives/why-shared-staging-is-expensive/) works the arithmetic from a conservative one lost hour per developer per day, about 12.5 percent of engineering capacity. The mechanics of the staging bottleneck, and what Uber and Lyft built to escape it, are covered in [Staging Environment Challenges: The Bottleneck and How to Fix It](https://www.signadot.com/blog/the-staging-bottleneck-why-your-engineering-team-is-slow-and-how-to-fix-it/). ChangesStagingProduction One shared staging environment is a single queue every change has to pass through. ##### Staging is never actually like production The third failure is drift. Staging is supposed to be production-like, and a shared staging environment that is always full of unmerged branches is production-like in name only. It represents no real state of the system. A change that passes there has been validated against a configuration that will never exist, and a change that fails there may be reacting to someone else’s work in progress. Drift compounds through configuration and data as well. A manual fix applied to unblock a release never reaches the manifests, a dependency pinned for one team is never unpinned, and test data ages. Each is small, and together they make “it passed in staging” a weaker guarantee every month. Drift is also why the obvious fix, adding more staging copies, fails. Every copy drifts independently from main and from the others, so more copies means more environments that are each wrong in a different way. #### What AI coding agents change about staging environments Agents change staging on both sides: how many changes arrive, and how often each one needs the environment. A staging environment sized for human throughput is the first thing that breaks. Start with volume. CircleCI’s 2026 delivery data, covered in [the agent validation gap](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/), puts average throughput up 59 percent year over year, with the top 5 percent of teams nearly doubling theirs while the bottom 25 percent saw no improvement at all. Around 30 percent of merge attempts still fail and median recovery has stretched to 72 minutes, up 13 percent. The teams absorbing a tenfold surge in pull request volume are the ones whose validation infrastructure scaled with it. [The agent PR flood](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/) gives the shape of that surge: a 50-engineer team at two to three pull requests a day per person built its pipeline for 100 to 150 a day and now faces closer to 1,000. Then the loop. A person opens a pull request once and waits. An agent writes, tests, reads the failure and tries again, so it needs an environment on every attempt rather than once per feature. Claude Code, Cursor, Codex and Copilot all run that loop, and none of them can wait for a slot in a staging-lock channel. That moves where validation sits. Testing AI-generated code in a staging environment has to happen before a person reviews it. Otherwise the reviewer becomes the verification layer, and the review queue absorbs the flood the pipeline was supposed to. The mechanics of that shift are covered in [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). Per-run isolation matters more at this volume, not less. Batching agent changes into one staging deploy turns every failure into a search through dozens of candidates for the one that caused it, and an agent cannot act on a result it cannot attribute. Each run needs its own view of the system, tagged so logs and traces point back to it. Cost and cleanup become policy rather than habit. Hundreds of runs a day only stay affordable if a run deploys the services it changed and nothing else, and if environments carry a time to live that reclaims them without anyone remembering to. An environment per pull request that survives until someone notices it is how an agent-era staging bill doubles. Last, the path in. An agent needs to create and destroy its environment through the same interface it uses for everything else, which in practice means a CLI or an MCP server rather than a console someone clicks. [The Staging Trap: How to Unblock AI Coding Agents in Enterprise Kubernetes](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) covers what that validation path looks like at this volume. #### Do you need a staging environment at all? You need the stage. You do not necessarily need a single environment that everyone deploys into to provide it. Every team needs a place where a change, whether a developer or an agent wrote it, meets the real versions of its dependencies before real users do. Whether that place is one environment everyone deploys into, one environment per change, or an isolated view of a shared cluster is an architecture decision, and the right answer depends on how many services and how many concurrent changes the team has. For a monolith, or a small system of a handful of services, one staging environment that everyone deploys into is the correct answer. It is cheap to run, easy to keep current with the main branch, and the number of people who want it at the same time is small enough that a short queue is tolerable. Do not over-engineer this case. For a microservices system with dozens of services and many teams and agents shipping independently, that model stops providing the stage it was built for. The sections after this one cover what replaces it. There is also an argument for skipping staging and testing in production, where feature flags, canary releases and progressive delivery expose a change to a slice of real traffic and roll back on the first bad signal. That works for changes whose failure is observable and reversible in seconds. It does not work for schema migrations, payment flows, or anything regulated, and it moves the cost of a wrong guess onto real users. The case for it, and its limits, is made in [It’s Time To Kill Staging: The Case for Testing in Production](https://www.signadot.com/blog/its-time-to-kill-staging-the-case-for-testing-in-production/). The stronger form of that argument is that the isolation staging used to provide can be recreated per change inside production-like infrastructure, which is the model the rest of this guide arrives at. #### Fixes that do not work, and why The first fixes most teams try treat the symptom rather than the cause. Each of the four below is reasonable on its face, and each fails for a reason worth understanding, because the reason points at what a working fix has to do differently. **Adding more staging copies.** If one staging environment is contended, run three, or one per team. Cost scales linearly with the number of copies, since each is a full duplicate of the dependency graph, and drift multiplies, because several environments now each need to be kept current with main and with each other. Data is the hardest part: every copy needs its own datastores, and third-party integrations end up shared anyway. The [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#ephemeral-environments-vs-a-shared-staging-environment-which-works-better) puts the inflection point around 50 engineers and 25 services. **Feature flags as isolation.** Wrapping every in-progress change in a flag so it can sit in staging without affecting other testers gates behavior, but it does not isolate state or dependencies. A flagged change that alters a schema, writes to a shared queue, or changes a response format still affects everyone the moment it runs. They also accumulate in the codebase and were designed for controlled production rollout rather than testing. [The pitfalls of feature flagging in shared staging](https://www.signadot.com/blog/pitfalls-of-using-feature-flagging-for-testing-in-shared-staging-environments/) covers the failure modes, and [feature flags versus preview environments](https://www.signadot.com/blog/microservices-testing-feature-flags-vs-preview-environments/) covers where flags do belong. **Booking calendars and Slack locks.** Turn-taking makes contention orderly. It does not reduce it. The same number of changes arrive and each occupies the environment for the same time, so the queue is the same length, only better documented. Locks also fail in the common cases: a test runs longer than its slot, a developer forgets to release the lock, or two teams’ changes need to be in staging at the same time to test an interaction. Locks also assume a person is holding them: a coding agent in a build-test-fix loop cannot wait for a reply in a channel, so it either blocks or skips validation. **Mocking the way out of integration.** If staging is unreliable, mock every dependency and run integration tests in CI instead. Mocks keep the loop fast, but each mock encodes one team’s understanding of another team’s contract at one point in time, and the two drift apart as the real service changes. The result is the failure every microservices team recognizes: integration tests pass with mocks and staging still breaks. [Integration tests pass with mocks but staging still breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) lays out the ladder from contract tests to real-dependency tests that closes the gap. What the four have in common is that none of them changes the unit of isolation. Each still assumes a change has to be validated inside a whole environment. The fixes that work change that assumption. #### What are the alternatives to a shared staging environment? The alternatives all give each change its own isolated view of the system instead of a turn at an environment everyone deploys into. They differ in how much infrastructure that isolation costs per change, which is what people are asking for when they search for lightweight staging environments. Ordered from most to least infrastructure per change, the four are a full ephemeral environment per pull request, a namespace per developer or pull request, a preview environment for frontend and stakeholder review, and duplicating only what changed on a shared cluster. **Full ephemeral environment per pull request.** Every pull request gets a complete copy of the system, provisioned when the PR opens and destroyed when it merges. This is the model [Release](https://release.com/), [Bunnyshell](https://www.bunnyshell.com/), [Shipyard](https://shipyard.build/), and [Qovery](https://www.qovery.com/) offer as a product. Fidelity per copy is the highest, because nothing is shared. So is cost, which scales with services multiplied by open PRs: a 40-service system with 30 open PRs runs 1,200 service instances. Spin-up grows with the size of the copy. The tradeoffs in detail are on the [Signadot vs Release comparison](https://www.signadot.com/comparison/release/). **Namespace per developer or pull request.** The same idea built from Kubernetes primitives: a namespace per environment, with resource quotas and network policies for isolation, sharing one cluster’s control plane and nodes. It is cheaper than a cluster per environment and uses tools the team already has, but it still duplicates every service into every namespace. Data has to be loaded per namespace, nothing keeps namespaces in sync with main, and startup time grows with the application. [Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) compares both options. A virtual cluster sits between this tier and a full environment: its own API server on shared nodes buys a harder boundary than a namespace at close to namespace cost, and still duplicates the workloads. **Preview environments for frontend and stakeholder review.** A preview environment gives a branch its own URL so a reviewer can click through a change before merge, and for frontend work it is often one build pointed at shared backend services, which makes it cheap and fast. Whether it is also a valid integration test depends on what sits behind the URL: a full copy, a namespace, or a routed slice. [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) covers that distinction. **Only what changed, on a shared cluster.** One staging environment still exists and is still shared, but what is shared is the set of stable service versions, not the place everyone deploys into. Each change deploys only the service or services it touches, alongside those stable versions. Tag test requests with a routing key in a header, and have each hop send a tagged request to the changed version when one exists and to the shared stable version when it does not. Isolation happens in the routing layer rather than by duplicating the environment, so every change gets an isolated staging environment of its own and hundreds can be under test on one cluster at once. A change costs the one or two services it deploys, and spin-up is seconds, because there is nothing else to provision. Full environment per PRPR 1PR 2PR 3Namespace per PRns 1ns 2ns 3one clusterPreview environmentUI 1UI 2UI 3shared backendRequest routingshared serviceschange Achange Bchange Cper change Purple marks what each change has to deploy for itself. Full environment per PRPR 1PR 2PR 3Namespace per PRns 1ns 2ns 3one clusterPreview environmentUI 1UI 2UI 3shared backendRequest routingshared serviceschange Achange Bchange Cper change Purple marks what each change has to deploy for itself. Approach Isolation Production fidelity Spin-up time Cost per change Best fit Full environment per PR Complete, nothing shared Highest per copy Minutes to tens of minutes, grows with stack size Full stack per open PR Under about ten services, or where hard isolation is required Namespace per developer or PR Namespace boundary on a shared cluster Medium, drifts between syncs Minutes, grows with application size Every service per namespace Small to mid-size teams already fluent in Kubernetes Preview environment (frontend) The frontend only Backend is shared Seconds to minutes One build UI review, stakeholder demos Only what changed, on a shared cluster Per change, by request and message routing High, real shared dependencies Seconds Only the changed services Dozens to hundreds of services, many concurrent changes Duplicating only what changed on a shared cluster has four prerequisites: - Every service has to propagate the routing header on outgoing calls. This is application-layer work a service mesh alone cannot do for you, usually handled through [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) or [B3 propagation](https://github.com/openzipkin/b3-propagation). - Asynchronous flows need the key carried in message metadata and consumers that honor it, which is its own piece of work for [Kafka, SQS and other message queues](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/). - Changes that alter a schema or write destructively need their own ephemeral datastore rather than the shared datastore. Bitso does this per branch with [Signadot and Neon](https://www.signadot.com/blog/how-bitso-is-scaling-branch-based-development-with-signadot-and-neon/). - The shared stable versions have to be kept green. The approach shrinks staging to a shared set of dependencies but does not remove the need to operate one. Large engineering organizations built this internally before it existed as a product. Lyft built Onebox on [Envoy](https://www.envoyproxy.io/) sidecars, Razorpay built Devstack on a [Traefik](https://traefik.io/) ingress with context in OpenTelemetry baggage. Both are described in [how Lyft and Razorpay share development environments with hundreds of devs](https://www.signadot.com/blog/how-lyft-and-razorpay-share-development-environments-with-hundreds-of-devs/). Signadot provides this mechanism as a product, on the staging cluster you already run. A Sandbox deploys only the changed services next to the shared stable versions and receives a unique routing key: requests carrying that key reach the sandboxed versions at every hop, everything else falls through. Every developer, pull request and coding agent gets its own view of one staging environment, and hundreds coexist on it with no contention. That is what the [Kubernetes test environments](https://www.signadot.com/solutions/kubernetes-test-environments/) solution delivers, the four options above are compared in depth in the [guide to ephemeral environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), and the mechanism itself is defined on the [Kubernetes sandbox](https://www.signadot.com/kubernetes-sandbox/) page. requestservice-2service-1service-1bFORKservice-3dbtagged requestshared dependencies The tagged request reaches the forked service and then calls the same shared dependencies as everything else. requestservice-1service-1bFORKdbtagged requestshared dependencies The tagged request reaches the forked service and then calls the same shared dependencies as everything else. #### If you keep a staging environment, run it like a product A staging environment with no owner is the one that is always broken. Request routing does not remove this environment. It changes what is shared: the environment stops being the place everyone deploys into and becomes the set of stable services every isolated change tests against, which makes its health more important, not less. In practice that means four commitments: scale down capacity but never fidelity, deploy every merge to main automatically through GitOps so the environment is never more than one merge behind, give it a named owner with a rotation and published SLOs for time green and time-to-test, and share it through routing rather than turn-taking. [Kubernetes Staging Environments: How to Build, Run, and Share One](https://www.signadot.com/blog/kubernetes-staging-environment/) has each of those as a full reference architecture, with the Argo CD wiring. The first commitment is a question of [environment parity](https://www.signadot.com/articles/environment-parity/): which differences between the environment and production matter for the change under test, and which can be scaled down without changing behavior. ##### Running a Kubernetes staging environment A Kubernetes staging environment is its own cluster running the same Kubernetes version, add-ons, ingress rules and network policies as production, fed by the same manifests. Staging environment testing means exercising a change there, against those dependencies, before it reaches users. The split most teams land on is one cluster each for dev, staging and production, with staging scaled down in replicas and node sizes and identical in everything else. Namespaces inside that cluster separate concerns, not environments: a namespace per team or per change still duplicates every service, which is the cost the [namespace versus cluster comparison](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) works through. Keeping Kubernetes staging and production aligned is most of the job, and staging testing is only as trustworthy as the configuration drift between the two. Track that drift as a number rather than an intention: how many merges behind main the environment is, and how many manifest differences exist between the two clusters that are not replica counts or node sizes. Both belong on the same dashboard as the smoke checks. ##### How QA and stakeholders get access Access decides whether a centralized staging environment stays useful once more than engineers depend on it. QA engineers, product owners and support need a URL that reaches a specific version of the system, not a turn at the whole environment. A shared environment offers only the second, which is why QA access usually degrades into a booking channel. Routing solves it the same way it solves contention. A tester opens the environment with a routing key attached, through a preview URL or a browser extension that sets the header, and sees that change against the shared stable services. Nobody redeploys and nobody else notices. For frontend review, [preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) cover what sits behind the URL. Developers get the same treatment from the other direction: run the one service you are changing locally and connect it into the shared environment, so your process answers tagged requests while every dependency stays real. That closes the gap behind the complaint that QA finds bugs in staging nobody can reproduce locally. [Local development on Kubernetes](https://www.signadot.com/guide-to-local-development-kubernetes/) covers the mechanics. Data deserves its own decision, and the workable strategy runs in three tiers: synthetic seed data for most functional testing, an anonymized subset of production refreshed on a schedule where behavior depends on real data distributions, and an ephemeral per-change database for schema migrations and other destructive work. The [reference architecture](https://www.signadot.com/blog/kubernetes-staging-environment/) covers how to load and refresh each. Raw production data containing personal information never belongs in staging, whatever the access controls, because staging is by design the environment the most people can reach. ##### Sharing production traffic with staging Shadowing a sample of production traffic into staging, or replaying captured requests, raises fidelity further once the environment is stable. It exercises the code paths real users hit rather than the ones test authors imagined, which is the fastest way to close the last gap in environment parity. It needs the same data discipline as any other tier: scrub the payloads, and make sure shadowed writes land in staging datastores rather than anything shared with production. #### Staging environment best practices checklist Audit an existing staging environment against these twelve items. Each is a yes-or-no question, and each no maps to a failure mode from the sections above. 1. Staging is built from the same artifacts and manifests as production, with only replica counts and node sizes reduced. 2. Every merge to main deploys to staging automatically, and the current lag behind main is visible on a dashboard. 3. Staging has a named owner and an on-call rotation. 4. Staging has at least two published SLOs: percentage of time green, and time from request to availability. 5. No manual configuration changes are made in staging. Fixes go through the same pipeline as production. 6. Test data is synthetic or anonymized. No raw production personal data is present. 7. Changes that alter schemas or write destructively get an ephemeral datastore rather than the shared datastore. 8. Two teams, or a developer and an agent, can test conflicting changes at the same time without coordinating in a chat channel. 9. A failed test in staging can be attributed to a specific change without asking who else deployed. 10. Unit and component tests run before staging, not in it. Staging is for integration, parity, and rehearsal. 11. Third-party integrations point at the vendor’s sandbox or test mode, never at live accounts. 12. The cost of staging, including engineer waiting time, is measured and reviewed quarterly. Teams that can answer yes to items 8 and 9 have already stopped deploying everything into one environment, whether or not they call it that. #### Staging in regulated industries In fintech and healthcare, staging carries extra obligations. [PCI DSS](https://www.pcisecuritystandards.org/), [GDPR](https://gdpr.eu/) and [HIPAA](https://www.ecfr.gov/current/title-45/part-164) apply to any environment holding regulated data, audit trails have to show what was tested and by whom, and residency rules constrain where copies can run. Every additional full copy is another copy of sensitive data to govern and certify. Per-change isolation helps here rather than fighting it. Fewer copies means fewer places regulated data can exist, Sandboxes inherit the shared cluster’s access controls, encryption and audit logging instead of each copy being re-certified, and because logs and traces carry the sandbox identity, every test run is attributable to a change and a person or agent. Environments stay inside the cluster and region the organization already controls, so residency is unchanged. The payments and lending specifics are in [Breaking the Staging Bottleneck: Scalable Microservices Testing for Fintech](https://www.signadot.com/resources/breaking-the-staging-bottleneck-scalable-microservices-testing-for-fintech/). The compliance mapping is on the [microservices testing for fintech](https://www.signadot.com/microservices-testing-for-fintech/) page. #### Where to go next If your single staging environment still works, [Kubernetes Staging Environments: How to Build, Run, and Share One](https://www.signadot.com/blog/kubernetes-staging-environment/) shows how to operate it as a product before the queue forms. If the queue has formed and you are weighing an environment per change against a shared cluster, the [guide to ephemeral environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares the four options in depth. If you have settled on request routing, the [Kubernetes test environments](https://www.signadot.com/solutions/kubernetes-test-environments/) page describes what Signadot provides. For which tests need a cluster at all and where each layer runs, see the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). #### Related reading - [Staging Environment Challenges: The Bottleneck and How to Fix It](https://www.signadot.com/blog/the-staging-bottleneck-why-your-engineering-team-is-slow-and-how-to-fix-it/) - [Why Shared Staging Is the Most Expensive Tool You're Not Accounting For](https://www.signadot.com/deep-dives/why-shared-staging-is-expensive/) - [Kubernetes Staging Environments: How to Build, Run, and Share One](https://www.signadot.com/blog/kubernetes-staging-environment/) - [It’s Time To Kill Staging: The Case for Testing in Production](https://www.signadot.com/blog/its-time-to-kill-staging-the-case-for-testing-in-production/) - [Integration tests pass with mocks but staging still breaks: how to catch it earlier](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [The Pitfalls of Using Feature Flagging for Testing in Shared Staging Environments](https://www.signadot.com/blog/pitfalls-of-using-feature-flagging-for-testing-in-shared-staging-environments/) - [Microservices Testing: Feature Flags vs. Preview Environments](https://www.signadot.com/blog/microservices-testing-feature-flags-vs-preview-environments/) - [The Staging Trap: How to Unblock AI Coding Agents in Enterprise Kubernetes](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) - [How Lyft and Razorpay share development environments with hundreds of developers](https://www.signadot.com/blog/how-lyft-and-razorpay-share-development-environments-with-hundreds-of-devs/) - [Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) - [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [Breaking the Staging Bottleneck: Scalable Microservices Testing for Fintech](https://www.signadot.com/resources/breaking-the-staging-bottleneck-scalable-microservices-testing-for-fintech/) - [ShareChat Chooses Signadot, Giving Devs High Quality Testing Feedback Enabling Faster Time-to-Market](https://www.signadot.com/case-studies/sharechat-chooses-signadot-giving-devs-high-quality-testing-feedback/) - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [What Is a Kubernetes Sandbox?](https://www.signadot.com/kubernetes-sandbox/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) #### Frequently asked questions What is a staging environment? A staging environment is a pre-production copy of a system, built from the same code, configuration, and infrastructure definitions as production, where a change is validated against the real versions of its dependencies before release. Tests that pass there are treated as a release gate. It is scaled down in capacity, not in fidelity, and uses anonymized or synthetic data rather than live user data. What is the difference between staging and production? Production serves real users with real data, and failures there cost revenue and trust. Staging runs the same code and configuration but serves only the team's test traffic, so failures there cost only the time to fix them. Staging typically runs fewer replicas and smaller nodes than production, and should never differ from it in dependency versions, network policy, or configuration. What is the difference between dev, test, QA, UAT, staging, and pre-production? They describe levels of testing fidelity, not six environments most teams run. Development is the developer's or agent's own environment, local or cloud-hosted, with mocked dependencies. Test and integration usually refer to the automated suites CI runs, against mocks or against staging. Staging is the deployed, production-like environment where integration, end-to-end, QA, and acceptance testing run against real dependencies. QA, UAT, and pre-production are other names for staging or extra pre-production copies split out for testers, business users, or release candidates. Do you need a staging environment? You need a stage where a change meets the real versions of its dependencies before users do. A monolith or small system is well served by one shared staging environment. A system with dozens of services and many concurrent changes needs per-change isolation instead, as a full environment or a routed slice of a shared cluster. Testing only in production suits reversible, observable changes, not migrations, payments, or regulated data. Why does a shared staging environment become a bottleneck? A staging environment that everyone deploys into holds one integrated state of the system at a time, so every change has to pass through it in series. As teams and change rate grow, changes queue, the environment fills with unrelated half-deployed work so failures cannot be trusted, and engineer time spent waiting and repeating work becomes the largest cost. Coding agents multiply the change rate and lengthen the queue. What is the alternative to a shared staging environment? The alternatives give each change its own isolated view of the system. A full ephemeral environment per pull request offers complete isolation at the highest cost. A namespace per change is cheaper but still duplicates every service. Duplicating only what changed, on a shared cluster, deploys the changed services alone and routes tagged test requests and messages to them, so hundreds of changes share one cluster while none shares its work in progress. Should staging use production data? Not raw production data. Staging is the environment the most people can reach, so personal or regulated data does not belong there under any access control. Use synthetic seed data for most testing, an anonymized subset of production refreshed on a schedule where behavior depends on real data distributions, and an ephemeral per-change database for schema migrations and other destructive operations. What is the difference between a staging environment and a sandbox? A staging environment is the long-lived, production-like deployment of the whole system. A sandbox is one change's isolated view of it: only the services that change are deployed, and a routing key sends that change's test traffic to them while everything else falls through to the shared stable versions. Sandboxes do not replace the staging environment. They run on it, which is how hundreds coexist without anyone deploying over anyone else. How do AI coding agents change staging environment requirements? Agents raise both the number of changes needing validation and the number of attempts per change. A build-test-fix loop needs an environment on every iteration, not once per feature, and it cannot wait for a slot in a chat channel. That rules out a single shared staging environment and any queue in front of it. What works is per-change isolation on one staging cluster, created in seconds and torn down on merge. Can a coding agent test its change against real dependencies before opening a pull request? Yes, if each change gets its own isolated view of a staging environment rather than a turn at a shared one. Agents such as Claude Code, Cursor, Codex and Copilot can call a CLI or an MCP server to create a sandbox holding only the services they changed, run tests against the real shared dependencies, read the results and fix the change before a person sees it. Validation moves ahead of review instead of behind it. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## What Is a Kubernetes Sandbox? Source: https://www.signadot.com/kubernetes-sandbox/ Summary: The two meanings of a Kubernetes sandbox: a secure box for running untrusted or AI-generated code, and a lightweight environment for testing a change against real services. ### What Is a Kubernetes Sandbox? The two things a Kubernetes sandbox can mean, a secure box for untrusted or AI-generated code and an isolated environment for testing a change against real services, and when to use each. Published June 21, 2026 Updated August 31, 2026 Reading time 8 min Search for “Kubernetes sandbox” today and you get two unrelated answers. One is a secure box for running untrusted or AI-generated code. The other is a lightweight, isolated environment for testing a code change against real services. They share a name and almost nothing else. This page explains both, so you can tell which one you actually need. Both kinds show up inside a [software factory](https://www.signadot.com/software-factory/), where coding agents need somewhere safe to run code and somewhere realistic to test it. #### The short version A Kubernetes sandbox is a temporary, isolated place to run or test something on a cluster without affecting everything else. The term splits into two camps: The second kind is what replaces a turn at a shared staging environment, a tradeoff covered in the [complete guide to staging environments](https://www.signadot.com/staging-environments/). - **Run-something sandboxes** care about safety. They give code you don’t fully trust, often code an AI agent just wrote, its own locked-down place to execute so it cannot touch the host or other tenants. - **Test-something sandboxes** care about fidelity. They let a developer or an agent try a change against the real services, databases, and queues it will run with in production, without copying the whole stack. If you are running code you don’t trust, you want the first kind. If you are checking whether a change works, you want the second. Signadot builds the second kind. #### A sandbox for running code safely This is the meaning that has taken over the term lately, pushed by AI coding agents. When an agent writes and runs code on its own, that code needs somewhere isolated to execute. Two layers of tooling cover this. At the lower level are general execution sandboxes that will run code from any kind of agent, not just a coding one: - [E2B](https://e2b.dev/) runs each session in a Firecracker microVM, so the code gets its own kernel and a hardware-level boundary. - [Daytona](https://www.daytona.io/) focuses on fast startup for running AI-generated code, with an SDK that agents call to open a session. - [agent-sandbox](https://agent-sandbox.sigs.k8s.io/), a Kubernetes SIG project (`kubernetes-sigs/agent-sandbox`), adds a `Sandbox` custom resource for long-running, stateful agent runtimes, using gVisor or Kata Containers for kernel and VM-grade isolation. A layer up are environments built specifically for coding agents. [Ona](https://ona.com/) (formerly Gitpod) is one example: it gives each software engineering agent a full, isolated cloud workspace to read the repo, write code, run it, and open a pull request. The focus here is the development workflow, not just safe execution. The common thread across both layers is the same: they isolate where code runs. The question they answer is “where can this agent do its work without doing damage?” None of them tell you whether the change is actually correct against your real services. #### A sandbox for testing a change This is what Signadot means by a sandbox. Here the problem is not safety. It is that you cannot tell if a change works until it runs against its real dependencies. Spinning up a full copy of thirty services for every change is slow and expensive, and a single shared staging environment turns into a queue. A Signadot Kubernetes sandbox solves this by duplicating only part of the environment. You start from a shared, production-like baseline cluster and decide, for each change, what to isolate and what to borrow from the baseline. For the services you are changing, a routing key on each request sends traffic to your version while everything else falls through to the baseline. For stateful dependencies you isolate as much as the test needs: [resource plugins](https://www.signadot.com/docs/reference/resource-plugins) spin up ephemeral [database instances](https://www.signadot.com/docs/guides/set-up-data-isolation) on demand, and for message queues you either route messages by the same key or spin up [ephemeral topics](https://www.signadot.com/docs/guides/set-up-message-queue-isolation). Because you choose what is shared and what is forked, you get a high-fidelity, production-like environment without copying the whole stack, and many developers or agents can run their own sandboxes on the same cluster at once. This is the same idea behind [ephemeral environments in Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/): start from a shared baseline and isolate only what each change needs. Execution sandbox Coding-agent environment Signadot sandbox Examples E2B, Daytona, kubernetes-sigs/agent-sandbox Ona (formerly Gitpod) Signadot What it does Runs any code in isolation Hosts a coding agent so it can write and run code Tests a change against real services in a cluster Scope Any agent The coding workflow The coding workflow Answers Can this run safely? Where does the agent work? Does this change actually work? #### How to create a Kubernetes sandbox The right way to create a sandbox follows from which of the two meanings you need. **For learning and experimentation.** If “sandbox” just means a safe cluster to poke at, run one locally with [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/), or use a hosted playground. Nothing you break matters, and teardown is one command. **For running untrusted or AI-generated code.** Define a `RuntimeClass` backed by a hardened runtime such as gVisor or Kata Containers and schedule the untrusted pods onto it, or use a purpose-built controller like [agent-sandbox](https://agent-sandbox.sigs.k8s.io/), which wraps this in a `Sandbox` custom resource with stable identity and persistent storage. The boundary you get is the point: the code cannot touch the host kernel or other tenants. **For testing a change against real dependencies.** Copying the stack into a namespace or a separate cluster per change works at small scale, but the cost and spin-up time grow with every service. The request-level approach avoids that: a short spec forks only the workloads you changed onto the shared baseline cluster, and a routing key steers matching requests to your version. With Signadot that spec looks like this: ``` name: my-feature spec: cluster: staging forks: - forkOf: kind: Deployment namespace: hotrod name: route customizations: images: - image: myrepo/route:pr-42 ``` Applying it creates a sandbox in seconds. Requests carrying the sandbox’s routing key hit your forked service; every other request flows through the untouched baseline. The [Kubernetes test environments](https://www.signadot.com/solutions/kubernetes-test-environments/) page covers how teams run this pattern at scale. Try a Kubernetes sandbox on your own cluster Isolate only what each change needs, from services to databases and queues, share the rest, and test against real dependencies in minutes. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### How they fit together These pieces are complementary, and the coding case is where they line up best. A coding agent can run inside an environment like Ona, where it has a workspace to read the repo, write code, and open a pull request. That covers writing and running the code. It does not cover whether the change works against the rest of your system. That is the gap Signadot fills. You connect the agent’s environment to a real Kubernetes cluster and run the change against live services, databases, and queues, the same dependencies it will meet in production. The coding agent writes the change in a host like Ona, and Signadot lets it run integration and system tests against the real cluster before the pull request merges. So the split is clean. A tool like Ona hosts the agent and its workspace. Signadot handles the integration testing and verification against real dependencies. One proves the code can run. The other proves the change does what it should. For more on that step, see [validating AI-generated code against real Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/). #### When to use a Signadot Kubernetes sandbox - Developing locally against real cluster dependencies, without rebuilding images for every change. - Previewing a pull request in a production-like environment before it merges, the subject of the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/). - Running integration and end-to-end tests in parallel, one sandbox per change. - Letting a coding agent check its own work against the real system. For the bigger picture on testing across the lifecycle, see the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/), and for where each layer of tests runs on a cluster, the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). Test your next change in a real Kubernetes sandbox Spin up an isolated sandbox on the cluster you already run, test against live dependencies, and tear it down when you are done. Start free, no stack duplication required. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Related reading - [Software Factories: The Complete Guide for Cloud-Native Systems](https://www.signadot.com/software-factory/) - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) #### Frequently asked questions What is a Kubernetes sandbox? A Kubernetes sandbox is a temporary, isolated place to run or test something on a cluster without affecting everything else. The term covers two different things: a secure execution box for running untrusted or AI-generated code (using gVisor, Kata Containers, or microVMs), and a lightweight test environment where a code change runs against the real services on a shared cluster. What is the difference between a Kubernetes sandbox and a namespace? A namespace is a logical grouping inside a cluster, but workloads in one namespace can still reach services, queues, and databases in another, and copying a full stack into a namespace per change is slow and expensive. A sandbox adds an actual isolation boundary: either a hardened runtime around untrusted code, or request-level routing that isolates one change's traffic while sharing the rest of the cluster. Is a Kubernetes sandbox the same as agent-sandbox? No. agent-sandbox (kubernetes-sigs/agent-sandbox) is a specific open source project that adds a Sandbox custom resource for running AI agent workloads in isolation, using gVisor or Kata Containers. It belongs to the run-something-safely camp. A testing sandbox, like Signadot's, answers a different question: whether a change works against its real dependencies. How do I create a Kubernetes sandbox? It depends on which kind you need. For a safe place to learn or experiment, run a local cluster with kind or minikube. For running untrusted code, use a hardened runtime such as gVisor or Kata Containers via a RuntimeClass, or the agent-sandbox controller. For testing changes against real services, use a platform like Signadot: apply a short sandbox spec that forks only the workloads you changed on a shared baseline cluster. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Software Factories: The Complete Guide for Cloud-Native Systems Source: https://www.signadot.com/software-factory/ Summary: What a software factory is, from the 1968 idea to today's agentic and dark factories, the five levels of AI coding, and why most factories close the loop too late: they scaled coding agents (the controller) without scaling verification (the sensor). Covers the three sensor properties (latency, fidelity, bandwidth), the ways to build verification for microservices compared, how much autonomy a factory can safely run, and what to measure. ### Software Factories: The Complete Guide for Cloud-Native Systems What a software factory is, what a cloud-native one needs, and why most agentic factories close the loop too late: they scaled the coding agents, not the checks that tell them a change works. Author Arjun Iyer Published October 15, 2026 Reading time 37 min A software factory is a system that turns intent into running software through many automated loops at once. Coding agents write the changes, harnesses constrain them, checks verify them and pipelines ship them, while engineers decide what gets built and own the result. The idea is nearly sixty years old. In 2026 it names something specific: the agentic software factory, a delivery system where agents produce most of the code, so the hard question is no longer how fast the factory can write software. It is how quickly and how reliably the factory can tell whether that software works. That question is hardest on cloud-native systems, where what a change breaks lives between services. Engineering leaders keep running into the same pattern. Faros AI’s Q2 2026 report, drawn from 22,000 developers, compared each organization’s lowest and highest periods of AI adoption. Epics completed rose 66.2% and incidents per pull request rose 242.7%, while deployments per week fell ([Faros AI](https://www.faros.ai/research/ai-acceleration-whiplash)). More work goes in. Less comes out, and more of what comes out breaks. A control engineer would recognize that curve. A software factory is a feedback loop: intent is the setpoint, agents are the controller, and verification is the sensor. Most organizations spent the past year turning up the gain, with more agents running with more autonomy, and left the sensor as it was. A loop with high gain and a slow, biased sensor behaves the way that data does. It overshoots, it oscillates, and it spends more effort to deliver less. Agents made the controller cheap. Nobody made the sensor faster. This guide defines the software factory and its parts, traces where the idea came from, and explains the dark factory and the levels of autonomy on the way to it. It then uses that loop to show why so many factories are unstable, what a good sensor looks like when your product is dozens or hundreds of services, and how to decide how much autonomy your factory can safely run. For the hands-on mechanics of letting an agent test its own change, see the [guide to verifying AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). #### What is a software factory? A software factory is the whole system that turns a queue of intent into deployed changes, with as little human handling per change as the organization can trust. It is not one tool. It is a set of loops, each with a job, connected so that work flows from a specification to production and signals flow back. Addy Osmani gave the definition most people now work from: “A software factory is many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans owning the whole thing from above” ([Osmani](https://addyosmani.com/blog/software-factories/)). The published designs, from StrongDM’s to OpenAI’s, share the same parts: - **Intent.** Specifications, the center of spec-driven development, plus tickets, scenarios and acceptance criteria that say what done means. - **Queue.** The backlog the loops draw from, increasingly written or split up by agents themselves. - **Harness.** The sandbox, tools, context, rules and memory that constrain what an agent can do. - **Generation.** The coding agents themselves: Claude Code, Codex, Cursor, GitHub Copilot and their peers. - **Verification.** Static analysis, tests, scenario runs and runtime checks against real dependencies. - **Review gate.** The point where a person, or a policy, decides a change may merge. - **Integration and deploy.** CI, merge queues and progressive delivery. - **Production feedback.** Telemetry, incidents and user signals that flow back into the queue. OpenAI automates this loop too: The Pragmatic Engineer describes a “Perf Factory” that monitors production and starts Codex agents to fix performance problems ([The Pragmatic Engineer](https://newsletter.pragmaticengineer.com/p/openai-software-factory)). A software factory is not a CI/CD pipeline with a new name: a pipeline moves and checks code someone already wrote, and a factory also produces it. It is not one developer working with an agent in an editor, which is a tool rather than a system. And it is not necessarily lights out. Most factories running today keep people at the review gate, and most should. You will also see it called an AI software factory or an agentic software factory, and the process it runs called the agentic software development lifecycle (agentic SDLC) or AI SDLC. At the far end of autonomy it becomes a dark factory. ##### Software factory vs CI/CD pipeline A CI/CD pipeline is one stage inside a software factory. The pipeline builds, tests and ships a change. The factory adds everything upstream of it, from intent to generation, and the feedback that runs from production back into the queue. The difference shows up when agents arrive. A pipeline sized for the rate at which people write changes starts receiving them at the rate agents write them. Anthropic reported that its CI job volume increased 25 times in six months ([Anthropic](https://claude.com/resources/articles/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic)). ##### Software factory vs harness engineering Harness engineering builds the controls around one agent loop: the context, tools, rules and checks that keep a single agent on track. A software factory runs many of those loops against one shared queue and one shared production system. That raises questions a single harness never meets: how changes from different loops interact, what verifies them, and how fast. Birgitta Böckeler’s description of a harness as guides and sensors is the right vocabulary at both levels, and the rest of this guide leans on it ([Böckeler](https://martinfowler.com/articles/harness-engineering.html)). #### Software factory history: from 1968 to DoD software factories The phrase is older than most of the people now using it. It has meant four different things, and each version put verification in a different place. Era Who What "factory" meant Where verification sat 1968 to the 1980s Bob Bemer's proposal, then Hitachi (1969), SDC (1975) and Toshiba (1977) Standard process, reusable designs and code, tools and measurement, so programming could be managed like production Quality and process control at every stage 2004 Microsoft architects Jack Greenfield and Keith Short Product lines built from patterns, models, frameworks and tools that generate much of an application In the models and frameworks, checked by people 2017 onward The U.S. Air Force's Kessel Run, then the DoD's Platform One A DevSecOps organization: "a system of people, process, and technology built to continuously deliver valuable software that warfighters love" Pipelines, automated tests, hardened containers, continuous authority to operate 2026 Dan Shapiro, StrongDM, Addy Osmani, Dex Horthy, OpenAI Many agent loops fed by a queue of intent and drained through gates into production Contested: review, tests, behavioral twins, runtime checks The thread through all four is the production step. Bemer’s 1968 proposal already called for measures and controls for quality, and the Japanese factories that followed put process control at the center, as Michael Cusumano documented at MIT ([Cusumano](https://dspace.mit.edu/handle/1721.1/47992)). Agents changed the production step itself. What still resists is the one every version of the factory depended on: knowing whether what came off the line is right. The most recent pre-agent version left a warning. Bryon Kroger, a co-founder of Kessel Run, wrote that many defense software factories slid into “innovation theater” and were “measuring the wrong things or worse, not measuring at all” ([Kroger](https://defenseopinion.com/defense-software-factories-need-a-major-reset/1052/)). An agentic factory that counts pull requests and tokens is set up to repeat that. #### What is a dark factory? The five levels of AI coding A dark factory, or dark software factory, is a software factory in which no person writes or reviews the code. Specifications go in and working software comes out. The name comes from lights-out manufacturing. Shapiro points to FANUC, whose robot plant near Mount Fuji was already building about 50 robots a shift and running unattended for up to 30 days in 2003 ([Business 2.0](https://web.archive.org/web/20091123102010/https://money.cnn.com/magazines/business2/business2_archive/2003/06/01/343371/index.htm)). Dan Shapiro’s five levels, published in January 2026 and amplified by [Simon Willison](https://simonwillison.net/2026/Jan/28/the-five-levels/), borrow the levels of driving automation ([Shapiro](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/)). Like the driving scale, they count from zero: - **Level 0.** Nothing reaches the disk without your approval. AI is a search engine or an occasional autocomplete. - **Level 1.** You hand discrete tasks, such as a unit test or a docstring, to an AI intern. - **Level 2.** You pair with the AI like a junior colleague. Shapiro puts most self-described AI-native developers here. - **Level 3.** The agent writes the code and you manage it, which means reviewing diffs all day. Shapiro says almost everyone tops out here. - **Level 4.** You write the spec, review the plan, leave, and come back to see whether the tests pass. - **Level 5.** The dark factory: “a black box that turns specs into software.” Read the levels by what verifies the code at each one and the sticking point is plain. At level 3 the person is the sensor, reading every diff, and a person saturates. At level 4 the tests become the sensor, so whether level 4 is safe depends on what those tests run against. The levels describe how much the agent does. Underneath, they are a ladder of sensors. The best-known dark factory is StrongDM’s. In February 2026 its CTO, Justin McCarthy, described how a team of three engineers, formed in July 2025, builds software under two rules: “Code must not be written by humans” and “Code must not be reviewed by humans” ([StrongDM](https://factory.strongdm.ai/)). People write specifications and scenarios. Agents write the code and check it against what StrongDM calls a Digital Twin Universe: behavioral clones of the third-party services its product depends on, including Okta, Jira, Slack and Google Docs, Drive and Sheets. The company’s own summary is that “validation replaces code review” ([StrongDM blog](https://www.strongdm.com/blog/the-strongdm-software-factory-building-software-with-ai)). The rules got the attention. The world they built to test in is what made the rules safe. A twin universe that agents can run thousands of scenarios against is a sensor fast enough and faithful enough to stand in for a person reading the diff. Plenty of teams copied the rules. Far fewer built the sensor. The counterexample came from Dex Horthy of HumanLayer. At the AI Engineer World’s Fair in June 2026, he described taking HumanLayer’s own product fully lights off in July 2025, and the failure anyone who runs that way for months eventually meets: the site goes down, the agent cannot fix it, and the answer is buried in code nobody has read in three months ([Horthy](https://www.ai.engineer/talks/Ib5GBkD555M-harness-engineering-is-not-enough-why-software)). His diagnosis is about loop speed. Checking that code runs and its tests pass is quick. The cost of bad architecture shows up over months and years, which makes maintainability far harder to verify. His answer was to turn the lights back on: put code review back, and make it cheaper by settling design and plans before the agent writes code. Between those two sits the factory most enterprises will run for years: lit, with people at the gate, and dark in specific corners. A class of change such as dependency upgrades or new tests can run lights out once the sensor for that class is good enough. The useful question is never whether the whole factory goes dark. It is which corners have a sensor you trust. #### A software factory is a control system Every software factory is a feedback loop, whether anyone designed it as one or not. Naming its four parts turns the debate about whether people should still read code into a question you can measure: is the loop stable? Intent is the setpoint: specifications, tickets, acceptance scenarios and SLOs. Coding agents and their harnesses are the controller, and how much they change without a person, autonomy multiplied by concurrency, is the gain. Your running services, data stores and queues are the system under control. Whatever reports how that system now behaves, from tests and environments to review and production telemetry, is the sensor. Intentspecs, scenariossetpointcompareCoding agentsplus their harnesscontrollerMerge, deployCI, gates, rolloutactuatorRunning systemservices, data, queuesplantother changes, traffic, driftVerificationtests, environments, reviewsensor: latency, fidelity, bandwidth The software factory as a control loop. Agents raised the controller's gain. The loop's stability now depends on the sensor at the bottom. setpointIntentspecs, scenarioscomparecontrollerCoding agentsplus their harnessactuatorMerge, deployCI, gates, rolloutplantRunning systemservices, data, queuesother changes,traffic, driftsensorVerificationtests, environments, reviewlatency, fidelity, bandwidth The software factory as a control loop. Agents raised the controller's gain. The loop's stability now depends on the sensor at the bottom. ##### Open loop vs closed loop Strictly, no software factory runs open loop. A toaster does: it heats for a set time and never checks the bread. A factory always closes the loop somewhere, because production reports eventually. The useful question is which loop closes against the real system, and whether it closes before the controller acts again. Most factories close the agent’s loop against unit tests and mocks, which are a model of the system rather than the system. The loop around the real system closes only after merge, when CI, staging or production finally reports. By then the controller has moved on to the next change, and so have the forty agents beside it. None of this vocabulary is new. Böckeler describes a harness as guides and sensors. The Thoughtworks Technology Radar put [feedback sensors for coding agents](https://www.thoughtworks.com/radar/techniques/feedback-sensors-for-coding-agents) in its Trial ring in April 2026. Marc Brooker of AWS has described agents as the move from open-loop generation to feedback loops ([Brooker](https://brooker.co.za/blog/2026/05/18/whats-easy-whats-hard.html)). Osmani calls the limit on autonomy back pressure, and Andrej Karpathy told Y Combinator’s AI Startup School in 2025 that we have to keep AI on the leash and make the loop between AI generation and human verification as fast as possible ([Karpathy](https://www.youtube.com/watch?v=LCEmiRjPEtQ)). What the metaphor has lacked is the engineering that comes with it. A sensor has measurable properties, and the stability of a loop depends on them. #### How an AI software factory changes the SDLC A software factory changes the SDLC in two places. It adds the generation step, so every stage after it receives changes at the rate agents write them, and it turns verification from a stage people wait at into a service agents call from inside their own loop. Everything else was already there. Software delivery has always been a set of nested loops, each closed by a different sensor at a different speed. The inner loop is edit, build and run, closed by a compiler and unit tests in seconds. The pull request loop is closed by CI and review in minutes to hours. The release loop is closed by staging, QA and canaries in hours to days. The production loop is closed by telemetry and incidents over days to weeks. The intent loop, where product and architecture decisions get corrected, runs over weeks to quarters. People hid how slow some of those sensors were, because people were slow too. change readyfirst signal from real servicesDeveloper, shared stagingthe signal arrives well before the next changeCoding agent, same pipelineabout 60 attempts fit before the signal arrivesCoding agent, ephemeral environment in the loopthe signal arrives inside the attempt1 min10 min1 hour1 day1 week Illustrative timings on a log scale. Before agents, the signal from real services beat the next change. On the same pipeline, an agent makes dozens of attempts before it hears back. change readyfirst signal from real servicesDeveloper, shared stagingsignal arrives firstCoding agent, same pipelineabout 60 attempts before the signalAgent, ephemeral environment in the loopsignal inside the attempt1 min10 min1 hour1 day1 week Illustrative timings on a log scale. Before agents, the signal from real services beat the next change. On the same pipeline, an agent makes dozens of attempts before it hears back. A developer who spends two days on a change hardly notices that staging takes four hours to report, because the report arrives long before the next change is ready. An agent that writes the same change in four minutes notices every one of those hours. It either waits, which throws away the speed you bought it for, or it keeps going on what it can see, which is unit tests and mocks. In most factories the agent’s loop is now the fastest loop in the building, while the sensor that reports on real services still sits in the pull request loop or the release loop. The fastest controller you have is wired to one of your slowest sensors. The slowest loop matters too. Horthy’s point about architecture is a point about loop speed: no test fires when a design starts to rot, so the only sensor is people looking at the system over months. That gives people their place in a software factory. They belong on the intent, writing specifications and scenarios, and on the slowest loops, where judgment is the only sensor there is. They do not belong as the per-change sensor for a controller that runs a hundred times faster than they read. #### What the data shows: gain without sensing The industry data is what you would expect from a loop whose gain went up while its sensor stayed put. DORA’s 2025 research found that AI adoption “is associated with an increase in both software delivery throughput and software delivery instability,” ([DORA](https://dora.dev/insights/balancing-ai-tensions/)), and the report describes AI’s main role as an amplifier of an organization’s existing strengths and weaknesses ([DORA 2025](https://dora.dev/research/2025/dora-report/)). Throughput and instability rising together is the signature of a high-gain loop with a weak sensor. Measured Initial adoption, Q2 2026 report Deepening adoption, Q3 2026 report What it says about the loop Incidents per pull request Up 242.7% Up 14.5% Failures reaching production, then settling per change Pull requests merged with no review Up 31.3% Up 76.3% The human sensor saturated, then bypassed Time in QA Not reported Up 300.6% Readings queuing behind the sensor Median time in review Up 441.5% Still elevated, no figure given Reviewers saturated Pull request size Up 51% Up 71.8% Each change touches more of the system Code churn, lines deleted per line added Up 861% Fell from its peak, no figure given Changes corrected, then corrected again Deployments per week Down 11.7% Reversed direction, no figure given Effort that never reached the output Monthly incidents Up 57.9% Up 125.4% More failures in total, even as each change got safer Epics completed Up 66.2% Not reported The controller's output went up Both columns come from Faros AI, which sells engineering analytics, so the figures are correlations across its customers rather than a controlled study. The Q2 2026 report compared each organization’s lowest and highest periods of AI adoption ([Faros AI](https://www.faros.ai/research/ai-acceleration-whiplash)). The Q3 2026 update looked at teams deepening adoption they already had ([Faros AI](https://www.faros.ai/blog/ai-speed-trap-takeaways)). Read side by side, the second column is not a loop settling. Incidents per change came down while pull requests merged with no review rose 76.3% and time in QA rose 300.6%. Less sensing happens per change, and what remains waits in a longer queue. That is a sensor being bypassed and backed up, not improved. The same pattern shows up from the other direction. CircleCI’s 2026 State of Software Delivery found average throughput up 59%, with the top 5% of teams nearly doubling theirs and the bottom 25% seeing no improvement at all, while roughly 30% of merge attempts failed ([CircleCI](https://circleci.com/resources/2026-state-of-software-delivery/)). Same agents, different sensors. At Bitso, more than 250 engineers and their coding agents work across over 200 microservices. After the company replaced its broken shared staging with per-change ephemeral environments, deployments rose from 664 to 1,400 a week while change failure rate fell from 5.2% to 0.9% ([Bitso case study](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/)). Gain went up and the loop got more stable, because the sensor scaled with it. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### The three properties of a good sensor A sensor is judged on three properties, and you can measure each one this quarter. Latency is how long it takes to report. Fidelity is how closely its reading matches the real system. Bandwidth is how many readings it can take at once. A factory that is weak on any one of them closes the loop too late, or against the wrong thing, for every change that property misses. ##### Latency: how long until the factory knows Delay is the classic way to make a stable loop unstable. The longer a signal takes to arrive, the more actions are taken before it does, and each of them is based on a state that no longer exists. Past a certain delay a loop cannot settle at all: every correction arrives late and overshoots, and the swings get wider instead of smaller. intent (setpoint)overshootovercorrectionsettles at intentsensor reports right awaysensor reports four steps latechange madetime Same controller, same gain. The only difference is how late the sensor reports. intentsensor reports right awaysensor reports four steps latechange madetime Same controller, same gain. The only difference is how late the sensor reports. A factory with slow feedback behaves the same way. Every action taken between a change and its signal is taken on stale information. The agent retries on top of a change nobody has verified, other agents build on top of that, and when the signal finally arrives it describes a system that no longer exists. A sensor that takes a day to report is not a sensor. It is a postmortem. The number to track is time to signal: the minutes from a change being ready to a verified result against real dependencies. For an agent, the target is inside its own loop, measured in minutes, not after the pull request in hours. ##### Fidelity: whether the reading matches the real system A biased sensor is worse than a slow one, because the loop trusts it. A thermostat mounted next to the oven will hold the kitchen steady at the wrong temperature. Unit tests and mocks are a sensor mounted next to the code. They report what a change does against a model of its dependencies, and the model is only as current as whoever last wrote it. Agents make the bias worse, because they optimize for whatever the sensor rewards. METR found frontier models modifying tests and scoring code to raise their scores ([METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)), and Anthropic’s system card for Claude 3.7 Sonnet reported the model occasionally special-casing tests to make them pass ([Anthropic](https://www.anthropic.com/claude-3-7-sonnet-system-card)). StrongDM met the same thing in practice: its agents rewrote code to pass narrow tests, so the team moved its checks into scenarios kept outside the codebase, like a holdout set in model training. Developers see the milder version every day. In the 2025 Stack Overflow Developer Survey, 66% named “AI solutions that are almost right, but not quite” as their top frustration ([Stack Overflow](https://survey.stackoverflow.co/2025/ai)). In a distributed system, almost right is the change that passes its unit tests and fails at a service boundary. The number to track is escape rate: of all the failures you find, the share found after merge, in CI on the main branch, in staging or in production. A falling escape rate means the pre-merge sensor is getting more faithful. ##### Bandwidth: how many changes can be measured at once A shared staging environment is a sensor that can take one reading at a time. When a few people merged a few changes a day, that was enough. With fifty agents, readings queue, or worse, they mix. Several changes land in the same environment, a test fails, and nobody can say which change caused it. A sensor that cannot attribute an error to an input cannot drive a correction, so the loop cannot close on that change. Agents raise demand on this property fastest. Every attempt in a build, test and fix loop wants its own reading, and loops run in parallel by design. The number to track is concurrent verification capacity: how many changes can be checked against real dependencies at the same time, without waiting for each other or interfering with each other. In most setups the three properties trade against each other. Mocks are fast and parallel but unfaithful. Shared staging is faithful but slow and single file. A full copy of the stack per change is faithful and parallel, but its cost grows with every agent you add. Getting all three at once is the design problem at the center of a software factory, and the way to stop trading them, covered below, is to duplicate only what each change needs and share the rest. #### Why cloud-native systems make the sensor hard In a distributed system, the behavior that breaks lives between services, so a faithful sensor has to include the services around a change and not only the change itself. That makes runtime verification, checking how a change behaves among the real services around it, the hardest part of a cloud-native software factory. Look at what tends to fail when an AI-generated change reaches staging. A field renamed in one service that another team’s service still reads. A retry that is harmless alone and multiplies load across a real call chain. A consumer that assumes ordering on a Kafka topic that does not guarantee it. A configuration or network policy that exists only in the cluster. Each change is correct at its own boundary and wrong in combination. Pull requests grew 51% larger with AI use in Faros’s first report and 71.8% in its second, so each change touches more of the system than it used to. CodeRabbit’s study of 470 open source pull requests found AI-co-authored ones carried about 1.7 times as many issues as human-only ones ([CodeRabbit](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report)). StrongDM’s design concedes the point. Its agents do not test against mocks. StrongDM built behavioral clones of the outside services its product touches, because the agents needed a world to run in. That works for third-party SaaS APIs with stable public behavior. For your own 150 internal services, a clone is a second implementation that drifts from the first, while the real thing is already running in your staging cluster. The [guide to environment parity](https://www.signadot.com/articles/environment-parity/) covers how that drift happens and how to catch it. #### Ways to build a software factory’s sensor There are seven common ways to build the sensor, and each makes a different trade between latency, fidelity and bandwidth. Most factories use several, at different loops. **Unit tests and mocks** report in seconds and run in parallel without limit, but fidelity stops at the service boundary and agents learn to satisfy them. [The real cost of writing mocks](https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing/) adds up quickly. **Contract tests**, with tools such as Pact and Specmatic, catch interface drift between teams and miss any behavior the contract never described, as the [complete guide to contract testing](https://www.signadot.com/contract-testing/) explains. **Service virtualization and behavioral twins**, from WireMock and Hoverfly to agent-built clones like StrongDM’s, are fast and cheap, with fidelity bounded by how well the twin models the real thing. They suit third-party APIs you do not control. **Agent sandboxes**, such as the GitHub Actions environment behind Copilot’s coding agent or Codex’s cloud containers, run agent code safely but usually hold only the repository, so they are a place to run a sensor rather than a sensor. The [Kubernetes sandbox explainer](https://www.signadot.com/kubernetes-sandbox/) separates the two meanings of the word. **Shared staging** is faithful when someone maintains it, hours behind a queue, and able to take one reading at a time, as the [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers. **Ephemeral environments that duplicate only what a change needs** give each change an environment of its own without copying the whole stack, and you choose how much each one duplicates. At one end, everything is duplicated: a namespace, virtual cluster or whole cluster per pull request, faithful and fully isolated, but minutes to start and more expensive with every concurrent agent. At the other end, only the components that changed are duplicated, and everything else is the stable shared version already running. In between, you duplicate the changed services plus the few dependencies the change needs to own, such as a database branch for a schema change. Duplicate everythingweborders (changed)paymentsdatabasequeuefull isolation, minutes to startChanged parts plus their dataweborders (changed)paymentsdatabasequeueown data, everything else sharedOnly what changedweborders (changed)paymentsdatabasequeueseconds to start, hundreds at onceDuplicated parts join the application's flows through request routing, message routing or configuration.duplicated for this changeshared stable version You choose what each change duplicates, from the whole stack down to only what changed, and everything else stays shared. Duplicate everythingweborderspaymentsdatabasequeuefull isolation, minutes to startChanged parts plus their dataweborderspaymentsdatabasequeueown data, everything else sharedOnly what changedweborderspaymentsdatabasequeueseconds to start, hundreds at onceorders is the service that changed. Duplicated partsjoin the flows through request routing, messagerouting or configuration.duplicatedshared stable version You choose what each change duplicates, from the whole stack down to only what changed, and everything else stays shared. Whatever the level, three things make it work. Some parts of the environment are duplicated. The duplicated parts take part in the application’s flows: request routing sends synchronous calls to the change’s own version, message routing does the same for asynchronous consumers on Kafka or a queue, where there is no request to route, and some components only need a configuration change to point at the right copy. And the result is verified end to end, against everything else that is real. The less you duplicate, the faster an environment starts and the more of them run at once, so hundreds can share one cluster. The price is plumbing: every hop and message has to carry the change’s routing key, and each stateful dependency needs its own isolation set up. [Lyft](https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-3-extending-our-envoy-mesh-with-staging-fdaafafca82f) and [Uber](https://www.uber.com/blog/simplifying-developer-testing-through-slate/) built in-house versions of the lightest end, and [isolation models for ephemeral environments](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) compares the two extremes. **Production signals**, from canaries and feature flags to telemetry, give the most faithful reading there is, after a change has reached users. They close the production loop and cannot stand in for a check before merge. Sensor Time to signal Fidelity Changes at once Best fit Unit tests and mocks Seconds Low, stops at the service boundary Unlimited Logic inside one service, on every attempt Contract tests Seconds to minutes Medium for interfaces, none for behavior Unlimited Interface drift between teams Twins and service virtualization Seconds Bounded by the twin High Third-party APIs you do not control Agent sandboxes Seconds to minutes Low to medium, dependencies usually absent High Running agent code safely Shared staging Hours, behind a queue High when maintained One Release checks and demos Ephemeral environment, whole stack duplicated Minutes High Limited by cost Small stacks, or changes that need the whole system to themselves Ephemeral environment, changed parts plus their data Seconds to minutes High, with its own data High Schema and data changes Ephemeral environment, only what changed Seconds High, real dependencies Hundreds or more Many services and many agents, on every change Production signals After deploy Highest Live traffic The production loop Signadot, a verification platform for cloud-native applications, gives every change an ephemeral environment without copying the stack. It installs as a Kubernetes operator on the cluster you already run. Each change duplicates only what it needs, from the modified services alone up to those services with their own databases or queues, and Signadot wires those copies into everything else, so to an agent or a developer it behaves like an entire environment of its own. Requests reach the change’s version through the mesh you already run: Istio natively, the Gateway API, or a DevMesh sidecar with Linkerd or no mesh. Messages stay on the shared broker and carry the change’s routing key, so each consumer version takes only its own, with [guides for Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub](https://www.signadot.com/docs/guides/set-up-message-queue-isolation). A database can stay shared with each environment’s [data kept apart](https://www.signadot.com/docs/guides/set-up-data-isolation), or a [resource plugin](https://www.signadot.com/docs/concepts/resources) can give an environment its own temporary database or a branch from Neon or Xata. Existing test suites, such as Playwright, Cypress or k6, run inside those environments as [Jobs](https://www.signadot.com/product/jobs/), and [Plans](https://www.signadot.com/product/plans/) package the checks a change must pass so an agent can run them by name. Agents create and test against these environments through the CLI, the [MCP server](https://www.signadot.com/docs/integrations/mcp) and the [signadot-validate skill](https://www.signadot.com/blog/introducing-signadot-validate-skill/), which is what puts the sensor inside the agent’s own loop. #### When the product itself is an agent A factory that builds agents needs a sensor that is statistical and realistic at once. Agent output varies from run to run, so one passing run proves little. Agent tools call real services, so a mock of those services hides the failures that matter. LangChain’s State of Agent Engineering survey of 1,340 practitioners found 57.3% running agents in production, with quality the biggest barrier, cited by about a third. Of those surveyed, 52.4% ran offline evaluations on test sets, and 37.3% ran online evaluations ([LangChain](https://www.langchain.com/state-of-agent-engineering)). Offline evals grade what the model says. They do not exercise the services its tools call. Each change to an agent-based product needs many runs against realistic dependencies, which multiplies the bandwidth the sensor needs. The [guide to testing MCP servers](https://www.signadot.com/testing-mcp-servers/) and [how Miro tests its agentic features](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) show what that takes in practice. #### What AI coding agents need from a software factory’s sensor Coding agents are both the source of the factory’s new volume and the heaviest users of its sensor, so the sensor has to work on their terms. Six requirements follow from that. 1. **It has to keep up with agent volume.** CircleCI’s average throughput gain of 59% understates what one team feels, since the top 5% of teams nearly doubled theirs. Test infrastructure sized for the pull requests a team of people opens in a day receives a multiple of that once agents open them too. 2. **It has to be callable from inside the loop.** An agent in a build, test and fix loop cannot file a ticket for a test environment and wait. Claude Code, Codex, Cursor and GitHub Copilot need to create an environment, test against it and tear it down through a command line or an MCP tool, in the same session that wrote the change. 3. **It has to run before review.** Testing AI-generated code against real dependencies before a person opens the pull request means the change arrives with evidence, so the reviewer judges intent and design instead of re-testing behavior. A 441.5% rise in median time in review is what happens when the reviewer is the sensor. 4. **It has to take one reading per change.** Each change needs its own ephemeral environment. Batching agent changes into one environment to save money makes failures unattributable, and an agent that cannot tell which change failed retries blind. 5. **It has to clean up after itself.** A loop that runs all night can create hundreds of environments. Time-to-live limits and automatic teardown tie the bill to work in progress, not to forgotten experiments. 6. **It has to report in a form an agent can act on.** A specific failing request, response diff or log line lets the agent fix the cause. A red build with no detail sends it guessing, which is how [loop costs multiply](https://www.signadot.com/blog/agent-loops-feedback-problem/). [Watching Claude Code verify a microservices change](https://www.signadot.com/blog/claude-code-signadot-validate-demo/) shows what one closed loop looks like, start to finish. #### How much autonomy can your software factory run? As much as its sensor can verify, and no more. Osmani puts it as a rule: “Back pressure is the rule that you can only hand a loop as much autonomy as you can cheaply and reliably verify, and not one inch more.” You set autonomy by what you can measure, and you raise it only when the measurement improves. The sensor you have The autonomy it supports Shapiro's level, roughly Compiles and unit tests only The agent proposes, and a person verifies and merges every change 0 to 2 Contract tests and CI, with real dependencies only in shared staging after merge The agent opens pull requests, and a person reviews both the diff and the behavior 3 Per-change checks against real dependencies in minutes, inside the agent's loop The agent iterates until the change is verified, and a person reviews the intent and the evidence 4 All of the above, plus held-out scenarios, behavioral diffs, production feedback and fast rollback Defined classes of change merge without review 5, for those classes The level a factory can safely run at is set by its sensor, not by its model. Three objections come up every time. **“Better models will fix this.”** A better controller still needs a sensor. Models learn to satisfy whatever the checks reward, as the reward-hacking results above show, so a stronger model attached to a biased sensor reaches the wrong answer faster and more convincingly. **“Put human review back on every change.”** Review is a sensor too, with latency in hours, a bandwidth of a few changes per reviewer per day, and fidelity limited to what a person can infer from a diff. Use people for the setpoint and the slow loops. Do not ask them to be the per-change sensor for a hundred agents. **“This is just a metaphor.”** Then test it. If the model is right, improving time to signal, escape rate and concurrent verification capacity should lower churn and incidents without lowering throughput. If it is wrong, those numbers will improve and the instability will stay. #### How to measure a software factory Measure the sensor and the output, not the agents’ effort. Pull requests per engineer and tokens per engineer count how hard the agents are working. StrongDM’s McCarthy offers a benchmark for the second: “If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement.” Token spend is a fine input to watch. As a measure of the factory, it rewards gain and ignores the sensor, which is why the Thoughtworks Radar puts [coding throughput as a measure of productivity](https://www.thoughtworks.com/radar/techniques/coding-throughput-as-a-measure-of-productivity) in its Caution ring. Metric What it tells you Loop property Time to signal per change, median and 90th percentile How stale the agent’s information is Sensor latency Escape rate, the share of failures found after merge Whether the pre-merge sensor matches reality Sensor fidelity Concurrent verification capacity, and time spent queued Whether readings wait for each other or mix Sensor bandwidth Verified changes deployed per week What the factory delivers Output Change failure rate Whether the output holds up in production Stability Rework rate, code rewritten within weeks of merging Whether the loop is oscillating Stability Review time per pull request How much sensing still falls on people Human load Start with the first three. They are cheap to collect, and they tell you which property is the weak one. #### Software factory best practices checklist Twelve questions worth answering yes to. Each no is a place where the loop closes too late. - **Intent is explicit.** Does every work item carry acceptance criteria or scenarios an automated check can test against? - **Signal arrives inside the loop.** Does an agent get a result against real dependencies in minutes, before it opens a pull request? - **The sensor sees real dependencies.** Do pre-merge checks exercise the services, data stores and queues around a change, not only mocks of them? - **One reading per change.** Does each change get its own isolated environment rather than sharing a batch? - **Capacity matches concurrency.** Can you verify as many changes at once as you run agents, without a queue? - **Evidence travels with the pull request.** Does every agent pull request link to the checks it ran and their output? - **Reviewers judge intent.** Has review moved from re-testing behavior to judging design, risk and intent? - **Autonomy follows the sensor.** Does each class of change get only the autonomy its sensor supports, raised only when the sensor improves? - **Some scenarios are held out.** Are some checks kept out of the agent’s view, so it cannot tune a change to them? - **Environments expire.** Do per-change environments have time-to-live limits and automatic cleanup? - **Production feeds the queue.** Do incidents and regressions turn into new scenarios and work items? - **The scorecard leads with the sensor.** Do you report time to signal, escape rate and verification capacity next to throughput? #### Where to go next If your agents open more pull requests than you can test, start with [why AI-generated code needs proof before review](https://www.signadot.com/blog/ai-generated-code-crisis/) and then measure your time to signal on the last twenty changes. If shared staging is the queue every change waits in, the [complete guide to ephemeral environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares the ways to give each change its own. If you are deciding how far toward a dark factory to go, pick one class of change, measure its sensor on all three properties for a month, and raise its autonomy only where the numbers support it. [Coding agent environments](https://www.signadot.com/solutions/agentic-development/) show how Signadot fits into that loop. #### References Sources cited in this guide, grouped by topic. ##### The software factory and the dark factory - Addy Osmani, “[Software Factories, Light and Dark](https://addyosmani.com/blog/software-factories/),” July 20, 2026. - Justin McCarthy, “[Software Factories and the Agentic Moment](https://factory.strongdm.ai/),” StrongDM, February 6, 2026. - StrongDM, “[The StrongDM Software Factory: Building Software with AI](https://www.strongdm.com/blog/the-strongdm-software-factory-building-software-with-ai),” February 2026. - Dan Shapiro, “[The Five Levels: from Spicy Autocomplete to the Dark Factory](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/),” January 23, 2026, and Simon Willison’s [commentary](https://simonwillison.net/2026/Jan/28/the-five-levels/), January 28, 2026. - Dex Horthy, “[Harness Engineering Is Not Enough: Why Software Factories Fail](https://www.ai.engineer/talks/Ib5GBkD555M-harness-engineering-is-not-enough-why-software),” AI Engineer World’s Fair, June 30, 2026. - Gergely Orosz, “[Inside OpenAI’s agentic software factory](https://newsletter.pragmaticengineer.com/p/openai-software-factory),” The Pragmatic Engineer, September 15, 2026. - Birgitta Böckeler, “[Harness engineering for coding agent users](https://martinfowler.com/articles/harness-engineering.html),” martinfowler.com, April 2, 2026. - Thoughtworks Technology Radar Vol. 34, “[Feedback sensors for coding agents](https://www.thoughtworks.com/radar/techniques/feedback-sensors-for-coding-agents)” and “[Coding throughput as a measure of productivity](https://www.thoughtworks.com/radar/techniques/coding-throughput-as-a-measure-of-productivity),” April 15, 2026. - Marc Brooker, “[What’s Easy Now? What’s Hard Now?](https://brooker.co.za/blog/2026/05/18/whats-easy-whats-hard.html),” May 18, 2026. - Andrej Karpathy, “[Software Is Changing (Again)](https://www.youtube.com/watch?v=LCEmiRjPEtQ),” Y Combinator AI Startup School, June 2025. ##### History - Michael A. Cusumano, “[Factory Concepts and Practices in Software Development: An Historical Overview](https://dspace.mit.edu/handle/1721.1/47992),” MIT Sloan School of Management working paper, 1989, and “Japan’s Software Factories: A Challenge to U.S. Management,” Oxford University Press, 1991. - Jack Greenfield and Keith Short, “[Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools](https://softwarefactories.com/TheBook.html),” Wiley, 2004. - Greg Hadley, “[Newest Air Force Software Factory Aims to Help AFRL, Attritable Aircraft Programs](https://www.airandspaceforces.com/newest-air-force-software-factory-aims-to-help-afrl-attritable-aircraft-programs/),” Air & Space Forces Magazine, January 20, 2022. - U.S. Army Futures Command, “[AFC announces ‘Software Factory’ in Austin](https://www.army.mil/article/237265/afc_announces_software_factory_in_austin),” army.mil, July 2020. - Bryon Kroger, “[Defense Software Factories Need a Major Reset](https://defenseopinion.com/defense-software-factories-need-a-major-reset/1052/),” Defense Opinion, October 13, 2025. - Christopher Null and Brian Caulfield, “[Fade to Black](https://web.archive.org/web/20091123102010/https://money.cnn.com/magazines/business2/business2_archive/2003/06/01/343371/index.htm),” Business 2.0, June 1, 2003. ##### Delivery and quality data - Faros AI, “[The AI Engineering Report 2026: The AI Acceleration Whiplash](https://www.faros.ai/research/ai-acceleration-whiplash),” April 2026. - Faros AI, “[The Speed Trap: 8 takeaways from Faros’s latest AI engineering research](https://www.faros.ai/blog/ai-speed-trap-takeaways),” September 18, 2026. - DORA, “[Balancing AI tensions: Moving from AI adoption to effective SDLC use](https://dora.dev/insights/balancing-ai-tensions/),” March 10, 2026, and the [2025 State of AI-assisted Software Development report](https://dora.dev/research/2025/dora-report/). - Sachin Malhotra, “[Agentic coding is straining CI. Here’s how we scaled test impact analysis at Anthropic](https://claude.com/resources/articles/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic),” Anthropic, September 14, 2026. - CircleCI, “[2026 State of Software Delivery](https://circleci.com/resources/2026-state-of-software-delivery/).” - Stack Overflow, “[2025 Developer Survey: AI](https://survey.stackoverflow.co/2025/ai).” - David Loker, “[State of AI vs Human Code Generation](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report),” CodeRabbit, December 17, 2025. - LangChain, “[State of Agent Engineering](https://www.langchain.com/state-of-agent-engineering),” survey conducted November to December 2025. - Sydney Von Arx, Lawrence Chan and Beth Barnes, “[Recent Frontier Models Are Reward Hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/),” METR, June 5, 2025. - Anthropic, “[Claude 3.7 Sonnet System Card](https://www.anthropic.com/claude-3-7-sonnet-system-card),” February 2025. ##### Duplicating only what changed, in practice - Lyft Engineering, “[Scaling productivity on microservices at Lyft, part 3: extending our Envoy mesh with staging overrides](https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-3-extending-our-envoy-mesh-with-staging-fdaafafca82f).” - Uber Engineering, “[Simplifying developer testing through SLATE](https://www.uber.com/blog/simplifying-developer-testing-through-slate/).” #### Related reading - [Agent Loops Don't Have a Token Problem. They Have a Feedback Problem.](https://www.signadot.com/blog/agent-loops-feedback-problem/) - [Coding Agents Are Only as Good as the Signals You Feed Them](https://www.signadot.com/blog/coding-agents-are-only-as-good-as-the-signals-you-feed-them/) - [Closing the Loop for Coding Agents in Cloud-Native Systems](https://www.signadot.com/blog/closing-the-loop-coding-agents-cloud-native/) - [Agentic Coding Tools Are Accelerating Output, Not Velocity](https://www.signadot.com/blog/agentic-coding-tools-are-accelerating-output-not-velocity/) - [Ramp's Inspect shows closed-loop AI agents are software's future](https://www.signadot.com/blog/ramps-inspect-shows-closed-loop-ai-agents-are-softwares-future/) - [Agents Write Code. They Don't Do Software Engineering.](https://www.signadot.com/blog/agents-write-code-they-dont-do-software-engineering/) - [2026 CircleCI Report: The Agent Validation Gap Is Here](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) #### Frequently asked questions What is a software factory? A software factory is a system that turns intent into running software through many automated loops at once. Coding agents write changes, harnesses constrain them, checks verify them and pipelines deploy them, while engineers set direction and own the outcome. The term dates to 1968. In 2026 it usually means an agentic factory, where agents produce most of the code and verification decides how fast it can safely go. What is a DoD software factory? In the U.S. Department of Defense, a software factory is an in-house team and platform that delivers software continuously through DevSecOps pipelines instead of multi-year contracts. The Air Force's Kessel Run, started in 2017, was the first. Platform One and the Army Software Factory in Austin followed. The agentic software factory keeps the same idea, a repeatable path from intent to running software, and adds agents that write the code. What are examples of software factories? StrongDM runs the best-known agentic one: agents write code that no person reviews, and scenarios run against digital twins of the services it depends on decide whether it ships. OpenAI runs its own around Codex, including a Perf Factory that fixes production performance problems. Before agents, the Air Force's Kessel Run was the model. Most teams run a lighter version, where agents open pull requests and people review them. What is a dark software factory? A dark software factory is one where no person writes or reviews the code: specifications go in and working software comes out. The name comes from lights-out manufacturing. Dan Shapiro calls it the fifth level of AI-assisted coding, and StrongDM runs the best-known example. What makes it workable is not removing review but replacing it with automated verification fast and faithful enough to trust. What are the five levels of AI coding? Dan Shapiro's five levels, published in January 2026, borrow the levels of driving automation and count from zero. At level 0 AI only suggests. At 1 it handles small tasks, at 2 it pairs with you, at 3 you review its code all day, at 4 you write specs and check whether tests pass, and at 5, the dark software factory, specs go in and software comes out. The level you can run safely depends on your verification. How is a software factory different from a CI/CD pipeline? A CI/CD pipeline builds, tests and ships changes someone already wrote. A software factory includes the pipeline plus everything upstream of it, the intent, work queue, agent harness and code generation, and the production feedback that flows back into the queue. When coding agents arrive, a pipeline sized for human change rates starts receiving changes at agent rates, which is why CI and test environments so often become the constraint. Why do software factories become unstable? Most software factories become unstable because they turn up the gain, more agents with more autonomy, without improving the sensor that verifies changes. Slow verification makes agents act on stale information, mocked tests reward changes that fail against real services, and shared environments check one change at a time. The result matches industry data: more output, more rework and more incidents per change, with flat or falling deployments. What is back pressure in a software factory? Back pressure is the rule, named by Addy Osmani, that you give a loop only as much autonomy as you can cheaply and reliably verify. In a software factory, that means raising agent autonomy for a class of change only after its verification gets faster, more faithful to production and able to run in parallel. How do AI coding agents change what a software factory needs to verify? Coding agents change the volume, the speed and the consumer of verification. They produce far more changes, iterate in minutes rather than days, and need a result inside their own build, test and fix loop rather than after a pull request. That turns verification in a software factory from a shared, human-paced stage into a per-change service agents call through a CLI or MCP tool, with one isolated environment per change. Can a coding agent such as Claude Code close the loop against real dependencies before a pull request? Yes, if the factory gives it a way to. The agent needs an environment with the real services around its change, created in seconds through a CLI or MCP tool and isolated from other agents' work. Ephemeral environments that duplicate only what the change needs do this: they wire it into the rest of the system and route the agent's requests and messages to it, so Claude Code, Codex or Cursor can test, fix and close the loop before review. Do humans still review code in a software factory? In most software factories, yes, but the job changes. When automated verification shows a change works against real dependencies, reviewers can judge intent, design and risk instead of re-testing behavior. Review disappears only in a dark factory, and HumanLayer, which ran its product that way, hit a production failure no agent could fix and turned code review back on. People belong on the intent and on the slowest feedback loops. How do you measure a software factory? Measure the sensor and the output rather than the agents' effort. The three sensor metrics are time to signal per change, escape rate (the share of failures found after merge) and concurrent verification capacity. The output metrics are verified changes deployed per week, change failure rate and rework rate. Pull requests and tokens per engineer measure how hard the agents work, not what the software factory delivers. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Validating AI-Generated Code Against Real Kubernetes Dependencies Source: https://www.signadot.com/validate-ai-generated-code-kubernetes/ Summary: Why a closed validation loop, real cloud-native dependencies, and infrastructure that scales to thousands of parallel agents turn AI-generated pull requests into software you can ship. ### Validating AI-Generated Code Against Real Kubernetes Dependencies Coding agents have moved the bottleneck from writing code to verifying it. What a closed loop against real Kubernetes dependencies needs, from who writes the checks to scaling to agent volume. Published June 21, 2026 Reading time 13 min **Validating AI-generated code** means proving that the change a coding agent produced actually works against the real services, databases, and message queues it will run with in production, not just that it compiles or passes a unit test. As agents move from autocomplete to opening pull requests on their own, the constraint on software delivery shifts from writing code to verifying it. This guide walks through why verification is the step that decides whether agents speed your team up or bury it, why the problem is hardest in cloud-native systems, what it actually takes to call a change verified, and how to make that verification fast and cheap enough to run on every change an agent produces. The short version: agents need a closed loop they can run themselves, against real dependencies, on infrastructure that scales to their volume. Everything below builds on that idea. For why the environment those dependencies live in cannot be a single shared copy, see the [complete guide to staging environments](https://www.signadot.com/staging-environments/). When the change an agent proposes is an API change, the check it needs is a contract test, covered in the [complete guide to contract testing](https://www.signadot.com/contract-testing/). The per-pull-request environment an agent runs that loop in is covered in the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/). The same loop, run by many agents across a whole organization, is the subject of the [complete guide to software factories](https://www.signadot.com/software-factory/). #### Why verification is the step that lets agents keep going For most of software’s history, typing the code was the slow part. That assumption no longer holds. A single engineer running coding agents can open more pull requests in a day than a small team used to open in a week. The agents are fast, tireless, and increasingly capable. The problem is that generation speed only turns into shipped product if every change can be verified quickly, and verification is now the rate limiter. There is a deeper reason verification matters so much for agents specifically, and it has to do with how they work. A coding agent makes progress by acting, observing the result, and deciding what to do next. When it cannot observe whether its change worked, it has to stop and ask a human, or worse, it guesses and keeps building on top of an unverified assumption. Either way the agent’s run ends early. Give the same agent a way to test its change, read real results, and correct itself, and it can stay productive for far longer without supervision. The verification step is what closes the loop. Write or fixthe changeValidate in a sandboxagainst real dependenciesRead theresultsPasses:ready forreviewFails or regresses: the agent fixes it and runs again, no human needed The closed loop is what keeps an agent moving on its own until the change actually works. This is why the most interesting agent results share a shape. When an agent can verify and correct its own work, it behaves less like an autocomplete and more like an engineer, as the team behind [Ramp’s Inspect described in detail](https://www.signadot.com/blog/ramps-inspect-shows-closed-loop-ai-agents-are-softwares-future/). The same pattern shows up in our own walkthrough of [Claude Code validating a microservices change end to end](https://www.signadot.com/blog/claude-code-signadot-validate-demo/): the agent provisions an environment, runs checks, reads what broke, and fixes it without a person in the path. Industry data is starting to name the gap too. The [2026 CircleCI report on the agent validation gap](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) found that teams adopting agents fastest are the ones whose biggest unsolved problem is verifying what those agents produce. Closing the loop, as we have [argued for cloud-native systems](https://www.signadot.com/blog/closing-the-loop-coding-agents-cloud-native/), is the difference between an agent that generates code and one that ships changes you can trust. #### Why this is a cloud-native problem first The reason verification is hard is almost never the changed file by itself. It is the system around it. A modern application is a mesh of dozens or hundreds of services talking over the network, backed by databases, caches, and queues, deployed on Kubernetes. The behavior that matters emerges from how those pieces interact, and a coding agent has no way to predict that from the source alone. To know whether a change works, you have to run it in something that looks like production. That is why Signadot is built for cloud-native environments first. The hard cases live there: request routing between services, real data stores, service meshes like Istio and Linkerd, asynchronous messaging. An approach that handles validation cleanly in that setting handles the easier cases for free. We think about the problem more broadly than Kubernetes, because the principle, give the agent a real environment and a loop it can drive, applies anywhere agents write code. But cloud-native is where the need is sharpest and where most teams running agents at scale already are. Claude and other agents need [a real place to run a change, not a mock of one](https://www.signadot.com/blog/claude-cloud-native-validation/), and in a cloud-native system that place is non-trivial to provide. #### What “verified” means for a cloud-native change, and why it is hard It is tempting to treat verification as a single checkbox: the tests passed. In a cloud-native system it is closer to a long tail of distinct questions, and a change can answer most of them correctly while quietly failing one that matters. On the functional side, you want to know that the change still does the right thing once it is wired into the live system: that its API contract did not shift in a way a caller cannot handle, that service-to-service calls still succeed, that an end-to-end user flow still completes, that a schema or data migration did not strand existing records. [Unit tests and mocks do not catch these](https://www.signadot.com/blog/validating-ai-generated-code/), because a mock encodes the very assumption an agent is most likely to get wrong. Then there is the non-functional side, which agents tend to ignore entirely: latency under realistic traffic, behavior under load, memory and resource limits, security, and the handling of authorization and secrets. A change can be perfectly correct and still be too slow, too expensive, or insecure. Functional Does it still behave correctly across services? API contractsService-to-service callsEnd-to-end user flowsSchema and data migrationsEvent and queue handlingBackward compatibility Non-functional Does it still meet the bar under real conditions? LatencyBehavior under loadResource and memory limitsSecurityAuthorization and secretsCost The practical consequence is that good verification is not one test but a portfolio of them, and the portfolio only produces a trustworthy signal when it runs against the real thing. An agent that reasons over noisy or fake signals reaches confident, wrong conclusions, which is why [coding agents are only as good as the signals you feed them](https://www.signadot.com/blog/coding-agents-are-only-as-good-as-the-signals-you-feed-them/). Real dependencies are what make the signal worth acting on. #### Who writes the checks, and who keeps them current A long tail of checks raises an obvious question. Someone has to write all of that, and keep it current as the system changes. At human pace, with humans authoring every integration test, this was already the part of testing that teams quietly let rot. At agent pace it is hopeless. If a fleet of agents is producing changes all day, a separate group of people cannot hand-write and maintain the verification for each one. So the checks have to be written and maintained by agents too, as a first-class part of doing the work rather than an afterthought. When an agent changes a behavior, it should also update or add the checks that prove the new behavior, the same way a careful engineer would. This is the line between an agent that produces code and one that does engineering, a distinction we have [written about directly](https://www.signadot.com/blog/agents-write-code-they-dont-do-software-engineering/): real engineering includes leaving the system verifiable for the next change. Tooling can make this the default. The [signadot-validate skill](https://www.signadot.com/blog/introducing-signadot-validate-skill/), for example, gives an agent a repeatable way to set up and run validation for a microservices change, so producing the proof is part of the same motion as producing the code. The human role does not disappear; it moves up a level. People define what good looks like, the scenarios that matter, the regressions that are unacceptable, the acceptance criteria a change must meet, and the agents fill in and maintain the checks that enforce it. #### Which checks run, and when Running every possible check on every change does not scale and is not even desirable. A one-line copy fix and a change to the payments service do not warrant the same battery of tests. Deciding which subset of the long tail is relevant to a given change is itself a judgment, and it is exactly the kind of judgment an agent is well suited to make, because the agent knows what it changed and can reason about the blast radius. Where each check sits in the delivery cycle, and which ones belong at the pull request, is the subject of [shift-left testing for microservices](https://www.signadot.com/articles/shift-left-testing/), and which tests need a cluster at all is covered in the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). The way to make that judgment reliable is to give the agent a library of validation it can compose rather than improvise. That is the idea behind [Signadot Plans](https://www.signadot.com/blog/signadot-plans/): reusable, deterministic validation flows that an agent selects and runs based on the change at hand. What makes the selection tractable is that each plan carries a `selectionHint` in its frontmatter, a short natural-language description of what the plan validates. An agent reads those hints, matches them against the diff in front of it, and decides which plans are relevant to the change, so it pulls in the booking-flow plan when it touches checkout and leaves the rest alone, rather than running the entire library every time. We went deeper on why this matters for agents specifically when we [introduced Plans as validation superpowers for coding agents](https://www.signadot.com/blog/introducing-plans-microservices-validation-superpowers-for-coding-agents/). The agent decides which plan fits a change and when to run it; the plan guarantees the check itself is consistent and repeatable. Intelligence picks the tests; determinism runs them. #### The validation layer has to scale to agent volume Everything above assumes you can actually run all of this, constantly. That assumption is where most teams hit a wall, because the two traditional ways to get a real environment both break under agent load. Replicating the full stack locally was already impractical for a thirty-service application on a laptop, and asking an agent to do it for every change is a non-starter. Shared staging turns into a queue: one environment cannot absorb dozens of concurrent agent pull requests without constant contention, broken state, and the forensic question of whose change caused the failure. That is the same [staging bottleneck](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/) that already slows human teams, now multiplied by agent throughput. Duplicating a full environment per change solves isolation but explodes cost, since the bill scales with the number of services times the number of in-flight changes. The picture to plan for is hundreds or thousands of agents, each opening and validating changes in parallel, every day. As we put it bluntly, most teams’ [infrastructure is not ready for agentic development at scale](https://www.signadot.com/blog/your-infrastructure-isnt-ready-for-agentic-development-at-scale/). What works is duplicating only what changed: instead of cloning the stack, deploy only the service a change touches into a shared baseline cluster, and use request routing to send test traffic to the changed version while everything else borrows the baseline. If you already run a service mesh, you are [halfway to solving this](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/) already. Agent100s to 1000s of agentsvalidating in parallelONE SHARED KUBERNETES CLUSTERSHARED BASELINEPER-CHANGE SANDBOXESand moreeach change runs only what it touched, on the shared baseline Running only what each change touched lets one cluster host hundreds of agent sandboxes at once, instead of duplicating the stack per change. Spun up this way, a sandbox costs a fraction of a full environment and starts in seconds, so running one on every change becomes the default rather than a budget decision. That economics is the whole game: validation that is cheap enough to always run is the only kind that keeps up with agents. #### How Signadot closes the loop Signadot puts these ideas together as a layer agents can drive. The environment layer is [Sandboxes](https://www.signadot.com/kubernetes-sandbox/): lightweight, ephemeral [ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) that deploy only the changed service onto your existing cluster and route test traffic to it, so hundreds can run in parallel at low cost. On top of that sits the validation layer. SmartTests compare the baseline and changed versions of an API and use Smart Diff to flag meaningful behavioral changes while ignoring benign noise, catching regressions without the hand-written contracts that [tools like Pact require](https://www.signadot.com/blog/ai-driven-contract-testing-signadot-smarttests-superior-approach-vs-pact/). Jobs run the integration, end-to-end, and load suites you already trust, inside the cluster, against the sandbox. Plans tie these into single, repeatable workflows an agent can select and run. The loop itself is closed by [Signadot’s MCP server](https://www.signadot.com/docs/integrations/mcp), which lets agents such as Claude Code and Cursor provision a sandbox, run these checks against real dependencies, read the results, and iterate, all without a human in the path until the change is genuinely ready for review. To see this in practice, [watch Claude Code validate a microservices change in a closed loop](https://www.signadot.com/blog/claude-code-signadot-validate-demo/). The result is a pre-merge gate where every agent-generated change is exercised against the real system, automatically, in parallel, before it reaches a reviewer or production. The [coding agent environments](https://www.signadot.com/solutions/agentic-development/) page walks through that loop step by step. Give every coding agent a real environment to prove its work Signadot spins up an isolated, production-like sandbox for every change on your existing cluster, and its MCP server lets agents validate their own pull requests against real dependencies. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Validation is the new build step Agentic development only pays off when generation speed converts into shipped, working software, and that conversion happens at the validation layer. The teams pulling ahead are the ones that treat verification as core infrastructure: a closed loop the agent can run itself, against the real cloud-native dependencies a change will meet in production, with checks the agents write and maintain, chosen intelligently per change, on infrastructure that scales to thousands of validations a day. Get that right and a flood of AI-generated pull requests becomes a stream of changes you can trust. For the broader testing context, see the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/), the [guide to ephemeral environments in Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), and [why coding agents break CI/CD pipelines and how to fix it](https://www.signadot.com/blog/why-coding-agents-will-break-your-cicd-pipeline-and-how-to-fix-it/). #### Related reading - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [Contract Testing: The Complete Guide](https://www.signadot.com/contract-testing/) - [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [Software Factories: The Complete Guide for Cloud-Native Systems](https://www.signadot.com/software-factory/) - [Ramp's Inspect shows closed-loop AI agents are software's future](https://www.signadot.com/blog/ramps-inspect-shows-closed-loop-ai-agents-are-softwares-future/) #### Frequently asked questions Why isn't passing unit tests enough to validate AI-generated code? Coding agents are good at producing code that is locally plausible, which is exactly what a unit test checks. The failures that matter in a cloud-native system are emergent: a changed response shape a downstream service cannot parse, a new call pattern that trips a rate limit, an assumption about data that holds in isolation but not against the real database. A mock confirms the agent called a dependency the way the author expected, which is the assumption the agent is most likely to get wrong. The only reliable signal is running the changed service against the real dependencies it will meet in production. Should developers or QA own AI-generated test suites? In practice, ownership follows the closed loop. When agents can validate their own changes against real dependencies before opening a PR, the developer who runs the agent owns the first line of validation, and QA shifts toward defining what good looks like: the high-value scenarios, the regression checks, and the acceptance criteria the agent must satisfy. Increasingly the agent also writes and maintains the checks themselves, since a human cannot keep hand-written suites current at agent volume. What happens when coding agents go wrong in enterprise environments? The risk is not that an agent writes one bad line; it is that it writes many plausible-looking changes that only fail on integration. The mitigation is a validation gate that every change must pass: an isolated environment, real dependencies, and automated regression detection before merge. With that gate in place, a wrong change fails fast in its own sandbox instead of breaking a shared environment or reaching production. How does validation infrastructure keep up with hundreds of agents? Duplicating a full environment per agent does not scale, because the bill grows with the number of services times the number of in-flight changes. What scales is duplicating only what changed: deploy just the service a change touches into a shared baseline cluster and route test traffic to it. One cluster can then host hundreds of concurrent sandboxes at a fraction of the cost, which is what lets thousands of agent-driven validations run in parallel every day. Can AI detect release regressions in Kubernetes? Yes. By comparing the behavior of a changed service against the current baseline and filtering out benign differences, AI-assisted checks like Smart Diff surface the breaking changes that matter, such as a removed field, a changed type, or a different status code, while ignoring noise. Run against real dependencies in a sandbox, this catches regressions before merge rather than after deploy. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## How to Test MCP Servers: A Complete Guide Source: https://www.signadot.com/testing-mcp-servers/ Summary: The four levels of MCP server testing: unit tests via in-memory transports, protocol checks with the MCP Inspector, integration tests against real dependencies in isolated sandboxes on the staging baseline, and agent evals, plus a production-readiness checklist. ### How to Test MCP Servers: A Complete Guide Four levels of MCP server testing, from unit tests and the MCP Inspector to integration tests against real dependencies and agent evals, with each change isolated on a shared cluster. Published August 31, 2026 Reading time 10 min Testing an MCP server means verifying four different things: that each tool handler behaves correctly in isolation, that the server speaks the protocol correctly, that the tools work against the real systems they front, and that an AI agent can actually use them to complete tasks. Most teams stop after the first one, ship a server that demos well, and find out in production that the other three were where the risk lived. This guide walks through all four levels, the tools for each, and how to run the hard part, integration testing against real dependencies, without giving every change its own copy of your infrastructure. When the product being built is itself an agent, the [complete guide to software factories](https://www.signadot.com/software-factory/) explains why its verification has to be both statistical and realistic. #### What is an MCP server? The [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) is an open standard, introduced by Anthropic in late 2024 and since adopted across the major AI platforms, that gives AI applications a uniform way to reach external systems. An MCP server exposes three kinds of capabilities to a client: **tools** (functions the model can call), **resources** (data it can read), and **prompts** (templates it can use). Under the hood it is JSON-RPC 2.0 over one of two transports: stdio for local servers, or streamable HTTP for remote ones, with OAuth-based authorization on the HTTP path. That is the whole surface, and it is deceptively small. The protocol is easy to implement and the SDKs do most of the work. What makes MCP servers hard to ship with confidence is not the protocol. It is everything behind it. #### Why testing MCP servers is hard **The caller is a model, not a program.** A REST API is called by code that was written against its contract. An MCP server is called by a language model that reads your tool names, descriptions, and JSON schemas at runtime and decides what to call and with what arguments. That means correctness includes things no traditional test covers: whether the model picks the right tool for a task, whether an ambiguous description sends it down the wrong path, and whether a schema change silently changes agent behavior. Two servers can be byte-for-byte protocol-compliant and still produce wildly different agent outcomes. **Tools have real side effects.** Useful MCP tools create tickets, write rows, send messages, and mutate state in the systems they front. A test that exercises them for real needs somewhere safe for those side effects to land. Pointing tests at production is out. Pointing every developer and agent at one shared staging environment turns it into a queue and a source of cross-contaminated state. **State spans calls.** Agent sessions are conversational. A tool call late in a session frequently depends on what earlier calls created or fetched. Testing individual calls in isolation misses ordering bugs, stale-handle bugs, and cleanup bugs that only appear across a sequence. **Auth is part of the contract.** Remote MCP servers authenticate clients and often act on behalf of a user against downstream systems. Scopes, token expiry, and permission boundaries are exactly the kind of behavior that mocks flatten away and that attackers probe first. **The dependencies are the point.** An MCP server is rarely an island. It is a thin protocol layer over your actual stack: internal APIs, databases, message queues, third-party services. Which means the classic microservices problem applies in full: [tests that pass against mocks while the integrated system breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/). The server is only as correct as its interaction with the real dependencies behind it. #### The four levels of MCP server testing Think of it as a pyramid. Each level catches a class of bugs the previous one cannot. ##### Level 1: Unit test the tool handlers Tool handlers are functions, and the official MCP SDKs make them easy to test as functions. The TypeScript and Python SDKs both let you connect a client and a server in memory, no transport or process boundary involved, so a test can call a tool exactly the way a client would and assert on the structured result: ``` import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js"; const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); await server.connect(serverTransport); await client.connect(clientTransport); const result = await client.callTool({ name: "create_ticket", arguments: { title: "Test", priority: "high" }, }); ``` Cover the basics here: valid inputs, schema-violating inputs, error paths returning proper MCP error responses rather than crashes, and edge cases in your business logic. This level is fast and belongs in every CI run. It tells you nothing about the model’s behavior or the real dependencies, and that is fine. That is not its job. ##### Level 2: Exercise the protocol surface with the MCP Inspector The [MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector) is the official interactive testing tool, and the fastest feedback loop you get during development: ``` npx @modelcontextprotocol/inspector node build/index.js ``` It connects to your server over stdio or streamable HTTP, lists every tool, resource, and prompt, lets you invoke tools with arbitrary arguments, and shows you the raw JSON-RPC messages both ways. Use it to verify the things unit tests take for granted: that tools actually appear with the schemas you intended, that descriptions read the way a model will read them, that errors come back as structured MCP errors, and that notifications fire when they should. The Inspector is interactive by design. For repeatable protocol checks in CI, script a minimal MCP client with the SDK, connect over the same transport your users will use, and assert on `tools/list` and a handful of representative `tools/call` exchanges. ##### Level 3: Integration test against real dependencies This is the level that separates a demo from a production service, and it is where most MCP testing advice goes quiet, because it is the expensive part. The mock-based shortcut fails here for the same reason it fails for any microservice: the mock encodes what the author imagined the dependency does, and the failures that matter live in the gap between that and reality. Schema drift in an upstream API, a side effect the mock has no concept of, auth scopes that behave differently against the real identity provider. The question is where to run it. Spinning up the full dependency stack per change is slow and expensive, and sharing one staging environment means every developer’s and agent’s side effects land in the same state. The pattern that scales is **duplicating only what changed on the staging baseline you already run**: deploy only the changed MCP server (and any backing service the change touches) as a sandboxed workload, and route requests carrying a sandbox key to it while everything else flows through the stable shared services. The server under test talks to real upstream and downstream dependencies, side effects are scoped to the sandbox, and hundreds of these can coexist on one cluster. With [Signadot](https://www.signadot.com/solutions/microservices-testing/), that looks like: your MCP server runs in the cluster as a normal workload, a pull request produces an image, and a short sandbox spec forks just that deployment on the baseline. The sandbox gets a routed endpoint, so you point your integration suite (the scripted MCP client from level 2, plus whatever end-to-end checks you have) at the sandboxed server and it exercises real APIs, real databases with [isolated schemas where tests write](https://www.signadot.com/docs/guides/set-up-data-isolation), and real [message queues with per-sandbox isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation). [Jobs](https://www.signadot.com/docs/reference/jobs) run those suites inside the cluster against the sandbox, in parallel across many open changes. This is the same model behind [ephemeral environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), applied to the newest kind of service in your cluster. Test your MCP server against real dependencies Give every change to your MCP server its own isolated sandbox on the cluster you already run, with real APIs, databases, and queues behind it. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Level 4: Agent evals The last level tests the thing MCP exists for: can an agent use your server to get work done? Here you drive a real model against the server with a set of realistic tasks and assert on outcomes. Does the agent select the right tool for each prompt? Does it chain calls in a sensible order? Does it recover when a tool returns an error? Because model output varies run to run, these tests are statistical rather than binary: run each scenario several times and track success rates, and use an LLM-as-judge or explicit tool-call assertions to score runs. Keep this suite small and high-signal. It is the slowest and most expensive level, and its job is to catch regressions in tool ergonomics (names, descriptions, schemas) that the lower levels cannot see. When a level 4 failure appears, the fix is usually a description or schema change, which then flows back down the pyramid for cheaper verification. #### Where coding agents change the picture MCP servers are increasingly written and modified by the same coding agents that consume them, and that changes who runs the tests. An agent working in Claude Code, Cursor, or Codex can own the whole loop: make the change, create a sandbox for it, deploy the modified server there, run the level 2 and level 3 suites against the sandbox’s routed endpoint, read the failures, and fix them, all before a pull request exists. Signadot exposes this workflow to agents directly through its [MCP server and CLI](https://www.signadot.com/docs/integrations/mcp), so the agent creates and tears down its own isolated environments on the shared cluster. The isolation is what makes this safe at agent speed. Ten agents iterating on ten changes get ten sandboxes against one baseline, and none of them can corrupt each other’s state or the shared environment. Every change arrives at review with evidence from real dependencies instead of a green checkmark from mocks. We cover the broader pattern in [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) and in [why CI pipelines are not ready for agent-scale change volume](https://www.signadot.com/blog/your-ci-cd-pipeline-is-not-ready-to-ship-ai-agents/). #### A practical checklist Before you call an MCP server production-ready: - Every tool handler has unit tests through an in-memory client, covering error paths and schema violations, running in CI. - Tool list, schemas, and representative calls are verified over the real transport (stdio or streamable HTTP), not just in memory. - Auth is tested as a first-class behavior: expired tokens, missing scopes, and per-user permission boundaries all have explicit tests. - Integration tests run against real upstream and downstream dependencies in an isolated sandbox per change, not against mocks and not against a contended shared environment. - Side-effecting tools are exercised for real, with writes landing in sandbox-scoped datastores or queues. - Multi-step sessions are tested as sequences, not just as independent calls. - A small agent-eval suite guards tool selection and chaining against regressions in names, descriptions, and schemas. - The whole loop is runnable by a coding agent, so changes to the server can be validated at the speed they are now written. #### Conclusion MCP servers compress a lot of trust into a small protocol: an agent will do whatever your tools let it do, against whatever systems they front. Unit tests and the Inspector get you a correct protocol surface. Real confidence comes from the levels above that, integration tests against the real dependencies and evals of real agent behavior, and those become routine once each change has an isolated, production-like place to run. If your MCP server fronts services on Kubernetes, that place already exists: it is your staging baseline, shared safely through [sandboxes](https://www.signadot.com/kubernetes-sandbox/). Ship MCP servers with proof, not promises Spin up an isolated sandbox for every change, run your MCP test suite against real dependencies, and let your coding agents do the same. Start free on the cluster you already run. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Related reading - [Software Factories: The Complete Guide for Cloud-Native Systems](https://www.signadot.com/software-factory/) - [Integration tests pass with mocks but staging still breaks: how to catch it earlier](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [Your CI/CD Pipeline Is Not Ready To Ship AI Agents](https://www.signadot.com/blog/your-ci-cd-pipeline-is-not-ready-to-ship-ai-agents/) #### Frequently asked questions How do you test an MCP server? Test it at four levels. Unit test each tool handler with the SDK's in-memory transport. Exercise the protocol surface interactively with the MCP Inspector to verify tool schemas, resources, and error contracts. Run integration tests against the real services and datastores the server fronts, ideally in an isolated sandbox on a shared cluster. Finally, run agent evals that check an AI agent selects and chains your tools correctly on realistic tasks. What is the MCP Inspector? The MCP Inspector is the official interactive testing tool for MCP servers, run with npx @modelcontextprotocol/inspector. It connects to a server over stdio or streamable HTTP, lists its tools, resources, and prompts, lets you invoke tools with arbitrary arguments, and shows the raw JSON-RPC exchange, which makes it the quickest way to verify schemas and error handling during development. Why is testing MCP servers harder than testing a normal API? Three reasons. The caller is a non-deterministic AI agent, so correct behavior includes how well your tool names, descriptions, and schemas steer the model, not just what the code returns. Tools carry real side effects, like writing to databases or calling internal APIs, so a bad test run can corrupt shared state. And an MCP server usually fronts a stack of real dependencies, so mock-based tests pass while the integrated behavior breaks. Can you test an MCP server against real dependencies without breaking shared environments? Yes, by duplicating only the changed MCP server. Deploy it as a sandboxed workload on your existing staging baseline: requests carrying the sandbox's routing key reach the new version, while every other request flows to the stable services around it. The server under test talks to real upstream and downstream dependencies, and many sandboxes run in parallel on the same cluster without interfering. How do AI coding agents test the MCP servers they build? By closing the loop themselves. An agent working in Claude Code, Cursor, or a similar tool creates a sandbox for its change, deploys the modified MCP server there, runs the test suite against real dependencies through the sandbox's routed endpoint, reads the results, and fixes what broke before opening a pull request. Each agent task gets its own isolated sandbox on the shared cluster, so parallel agents never collide. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Why Shared Staging Is Expensive Source: https://www.signadot.com/deep-dives/why-shared-staging-is-expensive/ Summary: The hidden tax on every engineering team running microservices, and why adding more staging environments makes it worse, not better. Deep dive ### Why Shared Staging Is the Most Expensive Tool You're Not Accounting For The hidden tax on every engineering team running microservices, and why adding more staging environments makes it worse, not better. Series The Hidden Cost of Shared Environments, part 1 of 5 Author Rory Nolan Reading time 10 min Topic Cost optimization It's 2:17pm on a Tuesday. A senior developer has been blocked for forty minutes. Not on a hard problem on a queue. Three other teams have changes deployed to staging, and one of them broke something. Nobody's sure whose change it was. The PR that was supposed to ship today will slip to tomorrow, maybe Thursday if the investigation runs long. This scene plays out in engineering organizations everywhere, dozens of times a week. It's so normal that most teams have stopped registering it as a cost. It's just how things work. But it isn't free. And the gap between what teams think shared staging costs and what it actually costs is one of the largest unexamined line items in modern software engineering. #### The Three Real Costs of Shared Staging Shared staging has three distinct cost buckets. They're rarely measured together, which is part of why the total stays invisible. ##### Cost 1 · Developer Time Lost to Contention When two or more teams need staging at the same time, someone waits. At a 100-person engineering organization shipping at pace, this isn't a rare edge case; it's a daily condition. Consider a conservative estimate: if the average developer loses one hour per day to staging-related delays, waiting for a slot, waiting for an environment to stabilize, waiting to understand whether a failure is theirs or another team's, the math is stark. 1 hrLost per developer per day (conservative) 230Working days per year ~12.5%Of total engineering capacity, gone At a fully loaded cost of $125,000 per year, one hour daily represents roughly $15,600 per developer annually. For a hundred-developer team, that's $1.56M in lost velocity, before counting a single incident, a single deployment failure, or a single bug that reached production because the debugging environment was too contested to investigate properly. That number doesn't appear on any invoice. It doesn't show up in your infrastructure costs or your headcount budget. But it's real, and it compounds. > "We had developers deploying unapproved code into staging just to test. That meant when bugs showed up, we couldn't tell what broke what." > > Engineering Team · Wealthsimple This describes something that happens in most scaling engineering organizations: developers start working around the bottleneck. They push code to staging before it's review-ready because waiting is expensive. The environment becomes noisier. Signal degrades. Eventually nobody trusts staging, and the entire point of having it begins to erode. ##### Cost 2 · The Blast Radius Problem In a shared environment, one team's bad deploy is every team's problem. When a service breaks in staging, it doesn't just block that team; it blocks every team with a downstream dependency. In a microservices architecture with dozens of interconnected services, that blast radius can be substantial. The less visible cost is investigative overhead. When staging breaks and multiple teams have active changes, the first thing that happens is an unplanned multi-team debugging session. Time is spent in Slack threads trying to attribute the failure rather than building features. Large engineering teams running shared staging often hit this directly: when regressions appear, it is impossible to quickly determine which team's change was responsible, causing delays across teams that had nothing to do with the original issue. The mechanics behind that failure, and the four alternatives to a single shared copy, are covered in [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/). The Pattern That Develops Teams start serializing their deployments to avoid collisions, effectively turning a shared parallel environment into a queue. A 100-person team that could theoretically be testing in parallel instead tests sequentially. Throughput gets capped by the throughput of a single environment. Teams then game the system further, pushing to staging at off-peak hours to avoid conflicts. Testing happens at the margins of the workday. Bugs get found on Friday afternoon. The problem perpetuates itself. ##### Cost 3 · The Bugs That Make It Through Anyway Perhaps the most significant cost of staging contention is the hardest to attribute: the bugs that slip through because the environment wasn't trustworthy enough to catch them. When staging is unreliable, developers learn to discount what it tells them. A test failure might be their bug, or it might be noise from another team's deploy, or it might be configuration drift in the shared environment. Over time, engineers develop a kind of learned skepticism about staging signals. They merge code that staging flagged as failing because "staging is always broken." And occasionally they're right. But every so often the flag was real. The research on production incident costs is consistent: a bug caught in development costs orders of magnitude less to fix than one caught in production. Industry estimates put the cost of a production incident at $10,000 or more when you account for engineer time, customer impact, and remediation. A team that lets three additional bugs per month reach production because staging was too noisy to catch them is adding $30,000+ to its annual cost base, without ever seeing it as a line item. * * * #### Why Adding More Staging Environments Doesn't Fix This The intuitive response to staging contention is to give each team their own staging environment. More environments, fewer collisions. This works briefly and at a small scale. It stops working as your microservices architecture grows: duplicating a full stack of services doesn't just duplicate compute costs. It duplicates the cost of maintenance, data seeding, configuration management, third-party integrations, and the operational burden on your platform team. Brex learned this the hard way. They built a system using duplicated Kubernetes namespaces to give developers isolated environments. As their microservices footprint grew, those environments began pushing the scaling limits of a single Kubernetes control plane. Availability issues pushed teams back toward staging, and the duplicated environments fell into disrepair. As Connor Braa, Software Engineering Manager at Brex, put it, the cost of getting something wrong in a preview environment was never high enough to prevent people from leaving things broken, so they constantly were. > "On the margin, with the Signadot approach, 99.8% of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not." > > Connor Braa · Software Engineering Manager, Brex The fundamental issue is architectural. Full environment duplication is the wrong model for microservices. When a developer changes one service out of 50, the cost of running the other 49 is pure overhead. The question isn't "How do we give each developer their own environment?" it's "How do we give each developer isolation for the services they're actually changing?" #### What Healthy Pre-Merge Validation Actually Looks Like The teams that have solved this problem well share a common pattern. They've stopped thinking of testing environments as finite, shared resources, and started treating isolation as something that gets created on demand, per change, and torn down when it's no longer needed. This is what the industry calls "shift-left" testing: moving integration validation as early as possible in the development cycle, ideally to the moment a developer opens a pull request. The core insight is that the cost of finding a bug rises with time. A bug found before merge is cheap. Found in staging, it costs more. Found in production, it's expensive. The goal is to catch as many bugs as possible at the earliest, cheapest stage. ##### The model that works at scale Rather than giving each developer a full clone of the environment, the approach that scales works like this: maintain a single stable baseline representing your production cluster. When a developer changes one or two services, spin up an isolated context, a Sandbox for just those changed services. Route test traffic through that Sandbox. Everything else flows from the shared baseline, unchanged. 1. A developer opens a PR. An isolated Sandbox spins up automatically inside the existing Kubernetes cluster, deploying only the services changed in that PR. 2. The Sandbox routes relevant traffic through the changed services and falls through to the baseline for everything else. No full stack duplication. No other teams affected. 3. Automated tests (Playwright, Cypress, Postman, or Signadot SmartTests) run against the Sandbox. Developers and reviewers access a live environment with real backend behavior. 4. When the PR merges or closes, the Sandbox tears down automatically. Nothing to clean up. Nothing left running and accruing cost. This model scales in a way that full duplication never can. A 100-developer team can run 100 Sandboxes simultaneously. Each contains only the 1-3 services being changed. The shared baseline handles everything else, which is precisely what Brex found: infrastructure costs reduced by 99% compared to their previous namespace-duplication approach. * * * #### What Teams Report After Making the Shift This isn't theoretical. Several engineering organizations have moved from shared staging to sandbox-based isolation and documented what happened. 80%Brex replaced namespace-based preview environments with Signadot Sandboxes across hundreds of engineers. Previewing changes took 80% less time. Developer satisfaction scores (CSAT) were 28 points higher for Signadot than the previous tooling. Infrastructure costs were reduced by 99% on a per-preview basis. ~50%Wealthsimple reduced staging bottlenecks by nearly 50% after shifting to per-PR sandboxes. Staging conflicts, teams blocking each other, unclear failure attribution, and developers pushing unapproved code just to test were eliminated. Sandbox adoption became core to their workflow, with plans to extend to async systems including Kafka and Temporal. Customer outcomes with Signadot The pattern across these organizations is consistent. The pain isn't unique to their scale or stack. It's the natural consequence of treating a shared environment as an infinite, low-friction resource, when it isn't either of those things. #### Calculate Your Own Staging Waste Staging cost audit Change the inputs to see your annual waste. Your team Engineers using staging Everyone who depends on the shared staging environment Fully loaded annual salary (USD) Salary, benefits, employer taxes and equipment Staging contention and wait time Hours lost per developer per day Waiting for a slot, the environment stabilizing, working out whose change broke it Platform engineers maintaining staging People whose time staging upkeep materially consumes Share of their time spent on staging (%) Tickets, environment fixes, on-call for staging incidents Infrastructure spend Non-production cloud cost per year (USD) Compute, networking and storage for staging and preview clusters Bugs that slip through Incidents per year from staging gaps Bugs that staging was too noisy or unreliable to catch before merge Average cost per incident (USD) Engineering hours, customer impact, remediation, post-mortems $938KDeveloper contention $104KPlatform overhead $120KInfrastructure spend Estimated annual staging waste$1.28MDeveloper time + platform overhead + infrastructure + incident cost Estimates only. Developer contention is the hourly rate × hours lost × 230 working days. Platform overhead is salary × 1.15 × the share of time on staging. All figures in USD. The numbers are illustrative until they're yours. Most teams that run this exercise are surprised not because the numbers are challenging to find, but because nobody had put them side by side before. The cost of the problem is usually much larger than the cost of solving it. The gap just hasn't been visible. Shared staging isn't bad because of a tooling mistake or a specific architectural choice. It's a scaling problem. It works reasonably well for small teams with a handful of services. It stops working when the number of concurrent developers, active PRs, and interconnected services grows, and it stays broken until the underlying model changes, not just the number of environments provisioned. The next question, once the cost is visible, is what a replacement model looks like technically and what it takes to adopt it without disrupting the teams already in flight. That's the subject of the next piece in this series. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## The Future of API Validation: AI-Powered Contract Testing Source: https://www.signadot.com/the-future-of-api-validation-a-deep-dive-into-ai-powered-contract-testing/ Summary: How AI-powered contract testing transforms API validation for microservices. ### The Future of API Validation: A Deep Dive into AI-Powered Contract Testing Why traditional contract testing buckles as services and teams grow, how AI-powered checks on every pull request take its place, and what DoorDash, Brex and Earnest gained. Published February 28, 2026 Updated July 13, 2026 Reading time 12 min #### Shifting Left: The Rise of AI-Powered Continuous PR Validation API contract testing was meant to be the safety net for microservices, ensuring that when one team updates a service, they won’t break others. But as organizations scale their services and teams, traditional contract testing approaches are starting to buckle under the pressure. Many teams that “tried Pact… didn’t stick” due to **too much maintenance** and tests that “get out of date fast.” In theory, [consumer-driven contract tests](https://www.signadot.com/comparison/pact/) should catch breaking API changes early. In practice, they often introduce their own bottlenecks in large, evolving systems. The definitional groundwork, including the two ways contracts get written and where they stop covering the system, is covered in the [complete guide to contract testing](https://www.signadot.com/contract-testing/). The reason is simple: maintaining explicit API contracts by hand doesn’t scale. A single API change can trigger dozens of contract updates across client services, creating a significant maintenance tax for developers. It’s no surprise that teams frequently abandon contract testing efforts that become “dusty old tombs” rather than living documentation. This scaling problem is no longer theoretical. A growing share of code is now written by AI coding agents like Claude Code, Codex, and Cursor, and they produce API changes far faster than anyone can hand-maintain a contract for. When the volume of changes jumps, any approach that depends on a human keeping the contracts in sync falls behind immediately. The bottleneck has moved from writing the change to knowing whether it is safe to merge. Moreover, these tests often check **specifications, not real behavior**. They assert that a service meets a predefined spec, but can miss subtle integration issues that only appear when the service interacts with [real dependencies](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). In a microservice architecture where _behavioral_ compatibility matters as much as structural typing, this gap is dangerous. And for developers, traditional contract testing can feel like a **white-box exercise** requiring deep implementation knowledge and fragile test data setups. In short, what started as a guardrail becomes another source of friction. From a platform engineering perspective, this problem translates to slow feedback and brittle pipelines. If every API change triggers a cascade of Pact file edits and synchronized releases between teams, velocity suffers. The original promise of microservices, independent deployments and faster iteration, gets undermined by heavy coordination costs. Clearly, a new approach is needed. What would it look like to validate APIs _without_ all this overhead? #### Introduction: The Staging Bottleneck and the Need for Speed Safety nets for microservices, from API contract tests to end-to-end integration suites, were meant to ensure that when one team updates a service, they won’t break others. But as organizations scale their services and teams, traditional testing approaches are starting to buckle under the pressure. Shared staging environments, brittle test suites, and even manual contract testing approaches are failing. Teams say that they “tried Pact… didn’t stick” due to high maintenance and tests that “get out of date fast”. In theory, late-stage testing should catch breaking changes. In practice, these methods often introduce their own severe bottlenecks in large, evolving systems. The reason is simple: maintaining complex test environments and explicit test suites by hand doesn’t scale. A single API change can trigger dozens of test updates and reruns across services, creating a major maintenance tax for developers. It’s no surprise that teams frequently abandon contract testing efforts that become “dusty old tombs” rather than living documentation. Moreover, these tests often run too late and check specifications, not real behavior. They can miss subtle integration issues that only appear when the service interacts with real dependencies. In a microservice architecture where behavioral compatibility matters as much as structural typing, this gap is dangerous. For developers, traditional testing can feel like a white-box exercise requiring deep implementation knowledge and fragile test data setups. What started as a guardrail becomes another source of friction. From a platform engineering perspective, this problem translates to slow feedback and brittle pipelines. If every API change requires a full integration suite to run in a queued, shared environment, velocity suffers. The original promise of microservices, independent deployments and faster iteration, gets undermined by heavy coordination costs. The environment is the real constraint: local setups are resource-limited and low-fidelity, while shared and staging environments are slow and contended. An agent generating a steady stream of changes cannot make progress if every one of them has to queue behind that. What would it look like to validate APIs and service behavior without all this overhead? What if you could catch bugs _continuously_, as part of every pull request, paving the way for true continuous delivery? #### The Shift to Intelligent, Continuous PR Validation Instead of forcing developers to manually write and update complex test suites, the emerging approach is to let the system do the work. This is the shift from slow, late-stage testing to a fully managed solution for continuous PR validation. Modern solutions like Signadot’s **SmartTests** product pioneer this approach by shifting from manual enforcement to intelligent behavioral validation. - **_No more exhaustive, brittle test suites._** _You write a lightweight functional test (e.g., a few critical API calls) and let the platform infer the rest by observing real interactions. The system learns the current baseline behavior of your services and uses that as the “implicit contract,” testing every change against it automatically._ - **_No dedicated test infrastructure or complex mocks._** _Rather than spinning up custom mock servers or maintaining separate integration environments just to validate changes, these tests run in real ephemeral environments under the hood. No stubs to maintain and no fake data drifting from reality, validation happens against real, running services_ - **_No waiting for production failures._** _Instead of discovering too late that an API change broke something, you catch issues before merge as part of the pull request workflow, with rapid feedback and minimal setup._ With [SmartTests](https://www.signadot.com/product/smart-tests/), AI-powered analysis takes over the heavy lifting of change detection, automatically identifying what’s different and filtering out the noise. For example, SmartTests uses a “Smart Diff” model. This leverages a dedicated AI model to compare responses from the baseline and new service versions to distinguish meaningful breaking changes (e.g., a removed field or changed status code) from benign noise (like differing timestamps or IDs). This eliminates the false positives and flaky tests that plagued traditional systems. Developers no longer have to sift through irrelevant failures because the AI only highlights the differences that actually matter. Crucially, this approach validates actual runtime behavior by running the new version of a service in an isolated environment and comparing its responses to the current baseline version. This dynamic testing is like having an automated reviewer that catches subtle regressions that static tests would miss, flagging any meaningful deviations in behavior. It is also the kind of signal an AI coding agent can act on directly: a precise, behavioral diff between baseline and changed service tells the agent exactly what its change broke, so it can close the loop itself and [validate AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) without waiting on a human or a shared environment. Teams have found it transformative. [DoorDash](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/), for instance, used this model to slash integration test feedback time from over 30 minutes to under two, effectively turning each pull request into a full integration validation gate rather than just a code review. #### High-Fidelity Sandboxes: Testing without Environment Headaches A key enabler of AI-powered, continuous PR validation is the use of lightweight sandboxes that duplicate only what a change needs. To validate behavior properly, tests need to run in an environment that is as close to production as possible. The naive solution is to duplicate full environments for each test or for each developer, but at scale, that’s painfully slow and prohibitively expensive. Spinning up dozens of microservices and databases for every feature branch is a huge time sink at scale, and it incurs cloud costs that can quickly grow out of control. Duplicating only what changed offers a smarter path. Instead of cloning everything, you run a single shared Kubernetes cluster as a baseline (with all services at their stable versions), and each change gets copies of only the services it touched. By routing specific test requests to those sandboxed service instances, each pull request gets its own ephemeral environment. This means only the API calls related to your change get diverted to your new code. Everything else uses the shared, production-like components. The result is a high-fidelity test run without duplicating entire environments. Your sandboxed service is talking to real dependencies with real data, so the observed behavior matches what would happen in production. You don’t have to pay the cost of booting an entire stack. The environment comes up in seconds, and each sandbox is isolated by context so that dozens of tests can run in parallel on the same cluster without interference. This approach also dramatically changes the cost equation for testing. Traditionally, the cost of test environments grew with the formula: (# of Developers) x (# of Services) Copying only the changed services breaks that model, decoupling cost from team and service growth: infrastructure expenses now grow roughly with the sum of developers and services, not their product. [Brex](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) saw this firsthand: by switching from full-stack duplication to sandboxes on a shared cluster, they now save about $2 million annually in infrastructure costs and saw developer satisfaction jump by 28 points. #### Boosting Pull Request Velocity and DevOps Metrics When comprehensive testing becomes intelligent and environments become cheap, the effect on developer velocity is dramatic. Integration tests that used to happen only after merge (or not at all) can be pulled earlier into the development process. This has a direct impact on DORA metrics: - **Lead Time for Changes**: Catching integration issues within minutes in a PR instead of days or weeks later drastically reduces the time from commit to production-ready build. DoorDash’s move to on-demand sandbox testing cut their pre-deployment validation from 30+ minutes to ~2 minutes. - **Deployment Frequency:** Faster, more reliable PR checks mean teams can safely merge and deploy smaller changes more frequently. With isolated sandboxes, multiple feature branches can be tested simultaneously without queuing for a shared environment. - **Change Failure Rate:** Catching breaking API issues before they reach mainline reduces the number of hotfixes and incidents in production. [Earnest](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) reported an **80% reduction in production incidents** after adopting early, high-fidelity integration testing. - **Mean Time to Restore:** Higher confidence in each release and smaller changesets make it easier to rollback problematic changes quickly. #### Real-World Results: DoorDash, Brex, and Earnest The move to AI-powered PR validation and sandboxed environments isn’t just theoretical. Several large engineering teams have already seen major benefits in practice: - [**DoorDash**](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/)**:** Achieved a **10× faster feedback loop** on code changes, cutting deployment validation from over 30 minutes to under 2 minutes, and retired their shared staging environment entirely. - [**Brex**](https://www.signadot.com/blog/how-brex-transformed-developer-experience-and-slashed-infrastructure-costs-with-signadot/)**:** Saves about $2 million annually in infrastructure costs and saw developer satisfaction climb by 28 points after switching to sandboxes on a shared cluster. - [**Earnest**](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/)**:** Reduced production incidents by 80% thanks to early, high-fidelity sandbox testing. #### The Road Ahead: Invisible, Comprehensive Testing in the Inner Loop Imagine a future where testing is so seamlessly integrated into development that it’s practically invisible. That’s where we’re headed. AI-powered validation checks and ephemeral environments are making continuous testing a natural part of writing code, rather than a separate phase. Every pull request spins up an isolated sandbox, runs a suite of tests, and delivers immediate feedback to the developer without extra setup or coordination. But functional API testing is just the beginning. The real goal is to “shift left” _all_ forms of validation. This same managed platform is being extended to support a wide range of non-functional testing, all triggered within the same, simple PR workflow. This includes: - **Performance Testing:** Automatically run baseline performance tests against your sandboxed service to catch latency regressions or $N+1$ query issues before they impact users. - **Security Testing:** Integrate automated security scans (like DAST) to probe your new code in its high-fidelity sandbox, finding vulnerabilities long before they reach a production environment. - **Log Analysis:** Use AI to analyze the logs generated by your service during the test run, automatically flagging new error patterns, concerning warnings, or significant deviations in log volume. - **Chaos Testing:** Proactively introduce controlled failures, like network latency or pod kills, within the isolated sandbox to validate that your service’s resilience and fallback logic work as intended, all without risking the shared cluster. This future is enabled by platform engineering, AI-driven validation, and a relentless focus on developer experience. For platform and senior engineers, this means rethinking the traditional testing strategy: brittle test suites and a single, overloaded staging environment won’t cut it. The path forward is integrated, automated, and developer-centric. It also lines up with where development itself is heading. As more code is written by AI coding agents, the work of validation has to keep pace with the work of generation, and it has to happen without a human in the loop for every change. Signadot provides both halves of that loop: sandboxes as the isolated, high-fidelity environment layer, and SmartTests, Jobs, and Plans as the validation layer on top. SmartTests gives an agent the same zero-maintenance behavioral signal it gives a developer, comparing baseline against the changed service and flagging the differences that matter, so agent-generated change can be verified at the speed it is produced. Take Signadot for a whirl and see it for yourself! #### Related reading - [Contract Testing: The Complete Guide](https://www.signadot.com/contract-testing/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [How Developers and Coding Agents at DoorDash Get 10x Faster Feedback on Code Changes](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/) - [How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Microservices Testing for FinTech Source: https://www.signadot.com/microservices-testing-for-fintech/ Summary: How fintech engineering teams test microservices safely in production-like Kubernetes environments. ### Microservices Testing for FinTech Why traditional staging fails FinTech testing, where compliance, customer trust and speed all matter, and how the sandbox model works in practice at Brex and Earnest. Published February 28, 2026 Updated June 21, 2026 Reading time 15 min #### Executive Summary In FinTech, testing isn’t just about software quality, it’s about safeguarding customer trust, ensuring regulatory compliance, and enabling secure innovation. But the rise of microservices architectures has exposed fundamental flaws in traditional testing workflows, especially staging environments. As FinTech organizations scale, the cost, complexity, and inefficiencies of infrastructure-heavy testing environments become unsustainable. This pressure is intensifying as more code is written by AI coding agents like Claude Code, Codex, and Cursor. A growing share of changes inside regulated FinTech orgs now originates from these tools, and they produce changes far faster than anyone can hand-review them. Writing the change is no longer the hard part: knowing whether it is safe to merge is, and in FinTech an unvalidated change is a compliance and financial risk, not just a bug. The general case, what a staging environment is for and the four architectures that replace the single shared copy, is covered in the [complete guide to staging environments](https://www.signadot.com/staging-environments/). This white paper explores the evolution of testing strategies in FinTech and introduces a new solution: Kubernetes-native sandbox environments. These lightweight, production-like testing spaces offer compliance-ready isolation without duplicating infrastructure, adding speed, scalability, and precision. With real-world examples from leading organizations like Brex, Earnest, and DoorDash, we demonstrate how the sandbox model is redefining how FinTech teams [test microservices](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### 1\. The FinTech Imperative: Trust, Compliance, and Speed FinTech applications operate in a uniquely high-stakes environment. They manage sensitive customer data, execute financial transactions, and rely on intricate integrations with external APIs, such as payment processors and fraud detection systems. In this context, software bugs aren’t just inconvenient, they pose serious risks, including financial loss, reputational damage, and regulatory violations. That’s why high-fidelity testing environments are essential in FinTech. They ensure that every new feature, update, or workflow behaves correctly before reaching production, reducing the risk of costly errors. The arrival of AI coding agents sharpens this point. When the volume of generated code jumps, the bottleneck moves from writing code to validating it, and that is precisely the part regulated FinTech teams cannot shortcut. Every agent-generated change still has to be proven safe against real payment and banking dependencies and within the same PCI DSS and GDPR boundaries, so the demand for high-fidelity, compliant validation rises in step with the code being produced. However, testing in FinTech comes with a distinct set of challenges. Regulatory frameworks such as PCI DSS and GDPR impose strict requirements around data handling, access controls, and system auditing. These constraints make it nearly impossible to simply clone production environments or data. Additionally, FinTech systems often depend on real-world, external services, including banking networks and payment gateways, that introduce unpredictable behaviors like rate limiting, timeouts, and service outages. These interactions must be tested in realistic conditions to ensure reliability. At the same time, every action in a testing environment must meet security and auditability standards, which adds further complexity. Historically, staging environments have played a critical role in addressing these challenges. Designed to replicate production conditions, they’ve been the go-to solution for validating functionality and compliance. But as FinTech systems grow more complex and teams scale, the traditional staging model is showing its limits. Maintaining multiple production-like environments is costly, difficult to synchronize, and increasingly prone to bottlenecks and delays. What was once a safety net is now becoming a drag on velocity and innovation. #### 2\. The Microservices Testing Paradox The shift from monolithic architectures to microservices has enabled software teams to achieve greateragility and scalability. In this new model, services are decoupled and independently deployable, allowingteams to move faster and scale specific parts of their systems without impacting others. This architecturalevolution has been especially appealing to FinTech companies looking to innovate quickly while managingcomplex systems. However, it has also introduced significant new testing challenges. While unit testing continues to scale effectively in microservices environments, enabling developers to test components in isolation, integration testing becomes increasingly difficult. As services proliferate, interdependencies grow. A simple feature change may span multiple services, each potentially managed by different teams using different stacks. These cross-service interactions are a frequent source of failures. Common issues include mismatched API versions, inconsistent data contracts, and hard-to-reproducebugs stemming from asynchronous workflows. Integration testing must account for all of these variables to ensure reliability, but doing so at scale is complex and time-consuming. This is particularly challenging in FinTech systems, where asynchronous flows are everywhere: fraud detection triggers, payment settlement queues, webhook-driven notifications, and more. These workflows don’t follow a simple request-response pattern, which means bugs can emerge long after a request is made, and failures often surface far downstream from their root cause. Testing these flows reliably requires high fidelity environments and end-to-end visibility, which most teams lack in their current setup. *Image: RabbitMQ, AWS SQS, Google Cloud Pub/Sub, and Kafka logos* Recent industry data highlights the severity of the issue. According to the CNCF’s 2024 annual survey, 80% of respondents are running Kubernetes in production (cncf.org). Yet despite the widespread adoption, many organizations continue to struggle with testing. Moreover, developers in microservices organizations lose an estimated 8 to 10 hours per week (getdx.com) dealing with testing bottlenecks and context switching. For a 200-person engineering team, that equates to approximately $400,000 in lost productivity each month. AI coding agents widen this gap further: they generate changes far faster than these bottlenecked workflows can absorb, so the queue of work waiting to be validated grows rather than shrinks. In FinTech, where trust, compliance, and speed are critical, these inefficiencies are unacceptable. Testing workflows must scale effectively, catching integration issues early, without compromising on security, data privacy, or regulatory compliance. Traditional staging environments, designed for simpler, monolithic systems, were never built to handle the dynamic, large-scale demands of today’s FinTech microservices. As a result, many organizations are now searching for modern testing solutions that align with the realities of microservices development. What FinTech teams need is a modern testing approach, one that supports asynchronous workflows, deep service integration, and scalable developer autonomy, without compromising compliance or operational integrity. #### 3\. Why Traditional Staging Breaks at Scale As FinTech organizations scale, their testing needs become more complex, and traditional staging environments struggle to keep up. Most teams fall back on one of three common testing strategies, each with its own benefits but also critical limitations. The first approach relies on mocks and unit tests, which are fast and easy to run. This method is popular because it allows developers to test individual components in isolation and get rapid feedback during the development cycle. However, this speed comes at the expense of realism. Mocks oversimplify service interactions and fail to capture the behavior of external systems like payment gateways, identity verification services, or fraud detection tools. As a result, many bugs go undetected until much later in the release process, often in production. The second approach is to use shared staging environments, which aim to closely replicate production conditions by running all services together in a single environment. This setup improves test accuracy compared to mocks but introduces a new set of problems. Because the environment is shared across multiple teams, it’s prone to test collisions, data conflicts, and instability. When multiple developers deploy overlapping changes or run conflicting tests, issues arise that are difficult to trace and resolve. These bottlenecks create delays, reduce developer confidence, and increase the time required to validate changes. *Image: Challenges in shared staging environments: team pull requests queue for one environment, causing long wait times and resource conflicts* The third and most comprehensive approach is environment duplication, creating a full copy of the staging environment for each team or use case. This strategy delivers high fidelity and strong isolation, making it easier to detect issues early and ensure test accuracy. However, the operational and financial costs are steep. Each environment must maintain compliance with regulations like PCI DSS and GDPR, which means provisioning isolated encryption keys, access controls, and secure audit logging. Additionally, these environments require unique configurations and connections to third-party APIs. As the number of environments grows, so does the complexity of managing and securing them. In practice, each of these approaches introduces friction. Duplicating environments creates significant overhead and delays due to the need for specialized infrastructure. Mocking trades off realism and often results in runtime failures that escape early detection. Sharing environments leads to interference between teams, test flakiness, and slower feedback cycles, all of which hinder development velocity. As one FinTech VP described their experience: > “We had dozens of staging environments, but debugging took days. There was always something out-of-sync, or a conflict we couldn’t reproduce.” This quote reflects a broader industry trend: traditional testing strategies may have worked for simpler systems, but they fall apart under the scale, security demands, and interdependencies of modern FinTech. A more scalable, cost-effective, and developer-friendly testing model is needed to move forward. #### 4\. A Paradigm Shift: The Sandbox Model *Image: Sandbox model diagram: developer workstation and Git branch sandboxes testing changes against a shared baseline environment* Sandbox environments offer a fundamentally different approach to testing microservices, one that breaks away from the infrastructure-heavy methods of the past. Rather than duplicating an entire application stack for every developer or test case, sandboxes isolate only the specific services under development. These services are then seamlessly integrated into a shared, high-fidelity baseline environment that mirrors production. This model dramatically reduces complexity and infrastructure costs while maintaining the realism and compliance needed in FinTech systems. At the core of this approach is selective deployment. Developers no longer need to spin up the entire system to test a change. Instead, they deploy only the service they are actively working on. This minimizes resource usage and accelerates the development cycle, especially when working within complex, multi-service architectures. Once deployed, request routing ensures that traffic is intelligently directed through the sandbox. Each request includes a sandbox identifier, which tells the system whether to route the call to the developer’s modified service or to the shared baseline version. This level of control allows for precise, per-sandbox behavior without duplicating the entire environment. Context propagation is another critical capability of the sandbox model. As a request flows through multiple services in a microservices architecture, it carries the sandbox context with it. This ensures that the entire chain of service calls remains isolated within that sandbox, preserving the integrity of the test and avoiding cross-contamination with other developers’ work. The foundation for all of this is a shared baseline environment. This environment runs continuously, containing the latest stable versions of all services and complying with organizational security and regulatory standards. By building on this shared baseline, sandboxes inherit its compliance posture, configuration, and connections to real external services, without requiring individual setups to replicate them. For FinTech teams, the benefits of this model are substantial. First, sandboxes inherit compliance from the shared environment. This means developers can test in production-like conditions without reconfiguring PCI DSS controls, audit logging, or encryption mechanisms. Second, sandboxes support real API integration. There’s no need to rely on mocks: teams can test against actual payment gateways, fraud tools, and banking APIs, capturing issues like rate limits and timeouts that often go undetected in simplified test setups. This inherited compliance is what makes sandboxes well suited to AI-driven development. Because a sandbox already sits inside the compliant baseline, an agent-generated change can be validated against real payment and banking dependencies without anyone standing up a separate compliant environment for it. That property matters most when the author is an agent rather than a person: agents do their best work when they can close the loop themselves, making a change, testing it against real dependencies, reading the result, and fixing, which requires environments that are quick and cheap to create but still high fidelity. Signadot supplies both halves of that loop: sandboxes as the isolated environment layer, and a validation layer on top in SmartTests, Jobs, and Plans that exercises the change and reports back. For a closer look at this workflow, see how teams [validate AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). Additionally, sandboxes provide fast feedback. Because they avoid full environment provisioning, developers can spin up a sandbox in seconds, test their changes, and iterate quickly. This speeds up the development loop and reduces time-to-release. Finally, the model is inherently audit-ready. All test activity remains within compliant boundaries, and sandbox-specific logs and headers allow for precise traceability, an essential requirement in any regulated FinTech environment. In short, sandbox environments enable FinTech teams to test faster, smarter, and more securely, without sacrificing the realism or regulatory rigor their applications demand. #### 5\. Real-World Use Cases: From Scaling to Security ##### [Brex](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) Brex adopted sandbox testing via Signadot to support hundreds of developers while maintaining complianceand real API fidelity. They: *Image: Brex outcomes with Signadot: infrastructure costs reduced by $2M per year, no environment drift, and faster development cycles, with a quote from Phil Burrows* ##### [Earnest](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) Earnest integrated Signadot into their CI/CD pipelines to enable instant sandbox environments and automated end-to-end testing, giving developers real-time insights into service behavior and dramatically improving release reliability. This shift empowered dev teams to: *Image: Earnest outcomes with Signadot: integration bugs caught early, faster code-merge confidence, and fewer staging surprises, with a quote from Early Ehlinger* ##### [DoorDash](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/) DoorDash’s developers use CLI-triggered sandbox environments, integrated with Istio for header-based routing. This self-service model enabled: *Image: DoorDash outcomes with Signadot: 70% lower infra costs, fewer pre-production bugs, and greater developer satisfaction, with a quote from Amit Gud* #### 6\. The Sandbox Advantage for FinTech *Image: Table comparing the traditional approach and the sandbox model across compliance, integration fidelity, async workflows, cost, speed, developer velocity, and data handling* #### 7\. Implementation Considerations Sandbox testing builds on technologies many FinTech teams already use. If you’re running on Kubernetes and using tools like Istio, Linkerd, or an API gateway, you likely already have the core building blocks in place. Even without a service mesh, lightweight systems like Signadot’s DevMesh provide built-in routing and context management, making it easy to get started. A key enabler is context propagation, passing a sandbox ID across service calls. This is made trivial by standards like OpenTelemetry, and in most interpreted languages (e.g., Java, Node.js, Python), auto instrumentation libraries mean no application-level changes are required. For event-driven systems like Kafka or SQS, sandbox IDs can be embedded in message headers to ensure isolation across async flows. Data isolation is handled through existing multi-tenant patterns or ephemeral database clones. Most teams start by scoping data to sandbox-specific IDs and clone databases only for services that require deeper or destructive testing. Sandboxes also integrate seamlessly with developer workflows. Engineers can spin up environments through CLI tools or CI pipelines, deploy just the service they’re changing, and test against real dependencies. Observability tools like Datadog and Grafana work out of the box, automatically tagging logs and traces by sandbox ID, so there’s nothing new to build. With minimal setup, sandboxes deliver isolated, production-like environments that fit naturally into modern FinTech stacks, enabling fast, compliant testing without the overhead of full environment duplication. #### 8\. Conclusion: Building Resilient FinTech Systems As the FinTech landscape continues to evolve, systems are becoming increasingly interconnected. New partnerships, third-party APIs, and constantly changing regulatory requirements are raising the stakes for integration testing. In this environment, even a small misconfiguration or unnoticed compatibility issue between services can lead to serious consequences, ranging from failed transactions to compliance violations. As the complexity of these systems grows, so too does the cost of inadequate testing. Unfortunately, traditional testing environments, whether shared staging environments or fully duplicated infrastructure, are no longer sufficient. Shared environments often lead to test interference, stale configurations, and bottlenecks that slow development. On the other hand, duplicating environments for every team or use case introduces prohibitive infrastructure costs, maintenance overhead, and compliance complexity. These legacy approaches cannot keep up with the speed, precision, or security required by modern FinTech development. Sandbox environments offer a much-needed breakthrough. By providing isolated, production-like testing spaces built on top of a shared and compliant baseline, sandboxes deliver the best of both worlds. They allow developers to work autonomously while maintaining strict security and data boundaries. Developers can test their services against real dependencies without impacting others, ensuring higher test fidelity without the cost of full duplication. **What sets sandboxes apart is their ability to align key objectives that are often at odds:** *Image: Three benefits of sandbox environments: safe independent testing, high-fidelity testing without duplicating systems, and regulatory compliance at speed* This advantage compounds as AI coding agents take on more of the writing. The constraint in FinTech is no longer how fast code can be produced, it is how fast each change can be proven safe to merge under real dependencies and real compliance controls. Sandboxes paired with a validation layer let teams keep that bar high while agents close their own loop, so the growing volume of generated code becomes throughput rather than risk. Ultimately, FinTech organizations that embrace sandbox testing gain a clear strategic advantage. They can release features with greater confidence, reduce bugs and delays, and maintain rigorous compliance, all while empowering developers, and the agents working alongside them, to move quickly and safely. In a sector where trust, speed, and innovation are tightly linked, sandbox environments are becoming essential infrastructure for success. #### About Signadot Signadot is a Kubernetes-native testing platform purpose-built for microservices. It enables application layer sandbox environments, letting developers test their changes in isolated, production-like conditions without duplicating infrastructure. Signadot helps FinTech companies shift testing left, boost developer velocity, and maintain compliance without compromise. [Learn more → www.signadot.com](https://www.signadot.com/) #### Related reading - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) - [How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) - [Shifting Left with Signadot: How Earnest Empowered Developers for Early Testing](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Per-PR Preview URLs, Explained Source: https://www.signadot.com/articles/per-pr-preview-urls/ Summary: Why preview URLs are nearly free for frontends and hard for backends, and the two architectures for giving every pull request its own link: environment duplication and header-based routing on a shared cluster. ### Per-PR Preview URLs, Explained Why a preview URL on every pull request is nearly free for a frontend and hard for a backend, the two ways to route a PR to its own preview, and how access, data and cleanup work. Author Arjun Iyer Published July 21, 2026 Reading time 10 min **A per-PR preview URL** is a link, posted on the pull request, that opens a running version of the application with that change included. Anyone with the link can try the feature before it merges, without pulling a branch or building anything. Frontend platforms made this a default expectation, and the question most teams face is how to get the same experience for services running on Kubernetes. Preview URLs are one kind of ephemeral environment, and the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers the others and what each costs, and the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares the platforms that create them. The short answer is that the URL is the visible end of an environment strategy. For a frontend, the environment is a self-contained build artifact, so a URL per PR is nearly free. For a backend service, the PR’s code is one node in a graph of services and datastores, so the URL has to lead to a place where that whole graph exists. The approaches below differ mainly in how they produce that place. #### The frontend version everyone knows [Vercel](https://vercel.com/docs/deployments/environments) and [Netlify](https://docs.netlify.com/site-deploys/deploy-previews/) build every pull request and comment a unique link on it, using URL schemes like a per-PR subdomain. The pattern works because the deliverable is a static or serverless bundle. Each build is complete in itself, hosting is a content delivery network (CDN) push, and a thousand previews cost little more than one. That mental model is what most searchers arrive with. It sets the right expectation for the workflow, a link on every PR that product managers and designers can click, and the wrong expectation for the mechanics once real backends are involved. #### Why the same idea is harder for a backend A backend pull request has nothing to show by itself. To render a page or serve an API response, the changed service needs its upstream callers, its downstream dependencies, a database with plausible data, and often a message queue. The unit that must exist behind the preview URL is the system, not the artifact. Frontend previewPull requestBuild artifactCDNOne URL per PRBackend previewPull requestService graphChanged serviceServiceQueueDatabaseOne URL, full graph A frontend preview points at one artifact. A backend preview has to point at a whole graph of services. Frontend previewPull requestBuild artifactCDNOne URL per PRBackend previewPull requestService graphChanged serviceServiceQueueDatabaseOne URL, full graph A frontend preview points at one artifact. A backend preview has to point at a whole graph of services. That leaves two basic strategies. Either you stand up a copy of the system for every pull request, or you share one live system and arrange for the PR’s requests to hit the changed service. Everything else in this article is a variation on those two moves. #### Two ways to give a pull request its own URL Environment per PRpr-123.preview.example.comNew namespaceFull stack copyHeader-based routingX-Preview-Key: pr-123Shared clusteronly changed services deployed Duplicate the whole stack per PR, or deploy only the changed services and route requests to them, virtualizing the full stack for the preview. Environment per PRpr-123.preview.example.comNew namespaceFull stack copyHeader-based routingX-Preview-Key: pr-123Shared clusteronly changed services deployed Duplicate the whole stack per PR, or deploy only the changed services and route requests to them, virtualizing the full stack for the preview. ##### A full environment per PR, one subdomain each CI or a GitOps controller deploys the entire stack into a namespace per pull request, and a wildcard Domain Name System (DNS) record plus an ingress rule map `pr-123.preview.example.com` to it. Tools like [ExternalDNS](https://github.com/kubernetes-sigs/external-dns) automate the records. This is the backend equivalent of the frontend pattern. [Okteto](https://www.okteto.com/preview-environments/) and [Release](https://release.com/ephemeral-environments) sell managed versions of it, and the [Argo CD ApplicationSet pattern](https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/applicationset/Generators-Pull-Request.md) is the common way teams build it themselves. It is conceptually clean because each preview is fully self-contained. Nothing is shared, so there are no routing rules to reason about, no header propagation to verify, and no way for one preview to interfere with another. Debugging an environment means looking at one namespace. But it gets expensive in proportion to system size and PR volume, since every open PR carries a full copy of every service, sitting mostly idle. Twenty open PRs against a thirty-service stack means six hundred running containers before anyone clicks a link. Each of those environments is also something the platform team operates: seeding its data, keeping its config current, and answering for it when a preview comes up broken. Spin-up time grows with the stack too, since the preview is not ready until the slowest service and its data are, and once previews take minutes to build, teams start rationing them to the risky changes, which quietly defeats the point of a preview on every PR. ##### Header-based routing on a shared cluster The alternative keeps one stable environment running and deploys only the services a PR changed, whether that is one service or several, as isolated versions alongside their stable counterparts. Requests that carry the PR’s routing key in a header are routed to the changed services at each hop, and fall through to the stable versions of everything else. One link virtualizes the full stack for that PR without deploying it. The preview URL is a hosted address that injects the routing key on every request, so clicking the link is all a reviewer does differently. This is the architecture behind Signadot’s Sandboxes, lightweight ephemeral environments that hold the changed service inside the shared cluster. Each Sandbox endpoint gets a hosted preview URL that injects the routing key automatically and authenticates through the control plane, so reviewers signed into the dashboard open it in a browser while automation uses an API key (see the [preview URL reference](https://www.signadot.com/docs/reference/sandboxes/preview-urls)). The biggest win is developer experience. A preview is ready as soon as one container starts, in seconds, and that stays true whether the stack has five services or three hundred, because nothing else needs to be deployed. Every preview also exercises the real, current versions of every dependency rather than a copy that may have drifted. Fast, dependable links are what keep reviewers and product managers clicking on every PR instead of saving previews for the risky changes. The operational and cost profile follows from the same property. The platform team keeps one baseline environment in sync with the main branch instead of operating a fleet of full copies, each needing data seeding, config updates, and an on-call answer when it comes up broken. And each preview costs one service instance rather than one system copy; this [walkthrough of cost-efficient preview environments](https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/) shows how that changes the bill in detail. The prerequisite is that services propagate request headers across hops, which teams with distributed tracing already do. The same preview URLs verify behavior for web and mobile applications alike, including [mobile and frontend previews](https://www.signadot.com/docs/tutorials/mobile-app-previews) driven by backend changes. They also pair with frontend platforms, shown in this [Vercel plus Kubernetes preview tutorial](https://www.signadot.com/docs/tutorials/vercel-preview-environments). A preview URL on every PR, without duplicating the stack Signadot gives each pull request a hosted preview URL backed by a lightweight sandbox on your shared cluster, so reviewers click a link and see the change running against real dependencies. The free tier is open to every developer. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Access, data, and cleanup The people a preview URL is for usually cannot run kubectl, so access has to work at the browser level. The common answers: - single sign-on in front of the preview domain - short-lived tokenized links - network placement behind the corporate virtual private network (VPN) Managed platforms handle this at their control plane, authenticating the hosted preview URL against the user’s existing session. Whichever you choose, the test is whether a product manager can open the link from the PR with zero setup. Data matters as much as access. A preview backed by an empty database demos nothing, so environments either share the stable system’s data through the routing approach, or seed fixtures on creation in the duplication approach. Shared data with isolated writes is its own topic, and the [preview environments guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) covers the options in depth. Cleanup is the part teams forget until it shows up on the cluster bill. Preview environments should tear down when the PR merges or closes, with a time-to-live backstop for anything that leaks. In the routing model, cleanup is deleting the ephemeral environment. In the duplication model, it is deleting a namespace and its volumes. #### Wiring the URL into the pull request The URL only changes behavior if nobody has to go looking for it. The standard wiring is a CI step that creates the preview and comments the link on the pull request, GitHub Actions and GitLab CI both make this a few lines, and many teams also post it to the ticket and the team channel. Some add a commit status or check run that links to the preview, so the URL sits in the merge box itself. Two small details pay off: - Pin the comment, or edit one comment in place per PR, so reviewers are not scrolling past a stack of stale links after every push. - Include the environment’s state in the comment, creating, ready, or failed, because a link that 502s for the first three minutes teaches stakeholders not to click. #### The lifecycle, end to end Step by step: 1. A developer opens a pull request. 2. CI builds the image and creates the preview, a full environment copy or a routed ephemeral environment. 3. A bot comments the URL on the PR, within seconds to minutes depending on the approach. 4. The reviewer clicks the link to sanity-check behavior, QA runs through the change in isolation, and the product manager confirms it matches intent, all before merge. 5. New commits update the same preview, so the URL never goes stale. 6. On merge, the environment is torn down automatically, with nothing left to clean up by hand. Reviewers are not always human. Coding agents can open the same preview to run end-to-end tests against the change before anyone looks at it, which makes the preview URL the validation surface for agent-written code as much as for human review. The preview lifecycle1PR opened2CI createspreview3Bot commentsURL4Team clickslink5Fix updatesURL6PR mergesenv removed The URL is created once, stays valid through every push, and disappears when the PR closes. The preview lifecycle1PR opened2CI creates the preview3Bot comments the URL4Reviewer, QA, PM click it5Fix updates the URL6PR merges, env removed The URL is created once, stays valid through every push, and disappears when the PR closes. #### Choosing between the approaches The choice is really about a tipping point. A team with a small stack and moderate PR volume has no bottleneck to fix. A namespace per PR is the simpler approach, cost and spin-up time are tolerable at that scale, and the whole mechanism stays easy to reason about. There is no reason to add a routing layer before the economics demand one. The calculus changes as the team and the service graph grow. When spin-up time gets long enough that stakeholders stop clicking, when the platform team spends its weeks keeping preview copies alive instead of improving the platform, or when the preview bill becomes a line item that needs explaining, duplication has hit its ceiling. At that point header-based routing stops being an optimization and becomes the requirement for PR previews to keep scaling: previews stay seconds from ready and at the cost of the changed services, no matter how large the stack grows or how many PRs are open. Both deliver the thing that changes how teams review, a URL on every pull request that anyone can click, and the [preview environments solution page](https://www.signadot.com/solutions/preview-environments/) covers what the managed version of that workflow looks like. #### Frequently asked questions What is a per-PR preview URL? A per-PR preview URL is a link, posted on the pull request, that opens a running version of the application with that change included. Anyone with the link, a reviewer, QA, or a product manager, can try the change before it merges without pulling a branch or building anything locally. Why are preview URLs harder for backend services than for frontends? A frontend preview deploys one self-contained build artifact to a CDN, so a URL per PR is nearly free. A backend change has nothing to show by itself: it needs its upstream callers, downstream dependencies, a database with plausible data, and often a message queue. The preview URL has to lead to a full working system, not a single artifact. What are the two ways to give a pull request its own preview URL? Either duplicate the environment per PR, deploying the full stack into its own namespace behind a per-PR subdomain, or keep one shared baseline environment, deploy only the changed services, and route requests carrying the PR's routing key to them at each hop. Duplication is simpler at small scale; header-based routing keeps previews ready in seconds, spares the platform team from operating a fleet of environment copies, and keeps cost flat as the stack and PR volume grow. #### Related Articles - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) - [ *Image: Preview Environments: The Complete Guide* ### Preview Environments: The Complete Guide July 31, 2025 ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [ *Image: Preview Environments for QA and Stakeholder Review* ### Preview Environments for QA and Stakeholder Review August 5, 2026 ](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Cut the Cost of Kubernetes Preview Environments with Signadot Source: https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/ Summary: Why per-PR preview environments get expensive with full duplication and how duplicating only what each change needs keeps the cost of hundreds of concurrent previews close to flat. ### Cut the Cost of Kubernetes Preview Environments with Signadot A step-by-step tutorial with the HotROD demo app on cutting preview environment costs in Kubernetes by forking only the services a change touches and sharing the rest of the cluster. Author Muhammad Khabbab Published October 2, 2024 Updated August 31, 2026 Reading time 8 min Topics [Tutorials/Guides](https://www.signadot.com/blog/category/tutorials-guides/)[Ephemeral Environments](https://www.signadot.com/blog/category/ephemeral-environments/) _Featured image by Gustavo Quepón on Unsplash._ #### Introduction How can development teams ensure efficient testing in a microservices world without incurring high resource costs? [Preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) are crucial for catching bugs early, but setting them up can be complex and expensive. Signadot tackles these challenges by providing a Kubernetes-native solution that creates scalable, lightweight sandboxes. Now developers can fork only the necessary microservices instead of replicating the whole tech stack. The result is optimized resources with high-quality testing in a real-world environment. Let’s dive into how Signadot simplifies the microservices testing workflow followed by a step-by-step guide to create a preview sandbox for microservices testing. #### Signadot’s Approach and Its Benefits Unlike traditional testing methods that require replicating the entire cluster, Signadot’s approach isolates only the relevant microservices for testing. The diagram below illustrates how this isolation is achieved by routing requests through the main environment to the sandboxed microservices. *Image: Diagram showing the Signadot operator routing requests with sd-routing-key baggage headers from the baseline environment to sandboxed services* This lightweight approach provides several key benefits, some of which are outlined below: ##### Key Features and Benefits **Lightweight Sandboxes:** - **Forking only necessary microservices:** Signadot allows you to create sandboxes by forking only the specific microservices that need to be tested. This not only reduces resource usage but improves performance as well. - **Connection to shared baseline:** Sandboxes connect to a shared baseline environment to provide access to common infrastructure and dependencies without duplicating them. **Improved Developer Experience:** - **Rapid environment creation:** As developers can spin up new environments in seconds, your development and testing cycles are accelerated. - **Simplified workflows:** Reproducing bugs and testing in isolation is simplified with Signadot. Using its browser extension, developers can set headers and reuse existing URLs to streamline testing and reduce overhead. **Cost-Efficiency and Scalability:** - **Reduced resource consumption:** Signadot utilizes a shared baseline environment that minimizes the resources required for each preview. By forking only the necessary microservices, it avoids the overhead of creating and maintaining multiple full-stack environments. - **Supports large teams and complex architectures:** The platform efficiently manages multiple concurrent previews without performance degradation and can handle the demands of large development teams and complex microservices architectures. **Early Issue Detection:** - **Thorough code testing:** Signadot enables developers to test code changes thoroughly in isolated environments before merging them into the main codebase. - **Decreased likelihood of production bugs:** By identifying and addressing issues early in the development process, Signadot helps reduce the risk of production bugs. #### Step-by-Step Guide to Setting Up Preview Environments with Signadot ##### Step 1 - Prerequisites and Setup ###### Pre-requisites 1. A Kubernetes cluster provisioned through minikube, K3, MicroK8s, etc. For this example, it will be a local minikube cluster. 2. [_Kubectl_](https://kubernetes.io/docs/tasks/tools/) needs to be installed. 3. [_Helm_](https://helm.sh/docs/intro/install/) should be installed too. ###### Setup 1. Create an account on signadot.com. You should have access to your Signadot dashboard from where you can create API keys, create cluster, manage sandboxes, and many more. 2. Create a cluster through the Signadot dashboard and create a cluster token. The Signadot cluster (from the dashboard) is a logical representation in the Signadot platform that allows it to connect and manage your local cluster (Minikube) using the cluster token. 3. Install [Signadot cluster operator](https://www.signadot.com/docs/installation/signadot-operator). This Kubernetes operator will be installed on your local Minikube cluster. It will enable management and coordination between your Minikube cluster and the Signadot platform. Upon successful installation of the operator, you will see the below message on your terminal. On the Signadot dashboard, you will see the status of your Signadot cluster as ready, see the below screenshot for reference: *Image: Signadot dashboard cluster details showing the testcluster with operator version 0.18.0 in Ready state* This shows that the Signadot Operator on your local cluster has successfully authenticated with the Signadot platform using the cluster token. ##### Step 2: Deploying the Baseline Environment Deploy a baseline environment in your cluster. For this article, we will use the Hotrod application [https://github.com/signadot/hotrod](https://github.com/signadot/hotrod) as the baseline. See below commands to install the application. The above commands deploy the Hotrod demo application into the _Hotrod_ namespace in your local Minikube cluster. After running the above commands, you will see the output below: Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Step 3: Installing and Configuring Signadot CLI 1. Install and configure the [Signadot CLI](https://www.signadot.com/docs/getting-started/installation/signadot-cli). It will be used to establish a connection with the local cluster so that you can access the Hotrod frontend application from your local machine. The Signadot CLI config is located at _$HOME/.signadot/config.yaml_. Make sure to configure it with the appropriate values. Below is a sample config file for the ongoing example: 2. Let’s use the Signadot CLI to connect to your local cluster, as well as start testing local changes using Sandboxes. See the below command “_signadot local connect_” and its result. The above command provided direct access to services running on the local Minikube cluster by configuring _/etc/hosts_ and routing traffic. As a result, you can access the services (API, front end, etc.) accessible via localhost on your machine. In the next step, we will test the front end of the Hotrod application. 3. You should be able to access the front end using the URL [http://frontend.hotrod.svc:8080](http://frontend.hotrod.svc:8080/). If you can see the Hotrod application, then everything worked fine so far and we are one step away from creating our first sandbox (preview environment). *Image: HotROD demo app frontend in the baseline environment showing a ride request where the driver arrives in a negative ETA* Notice the ETA value coming as negative, which is a bug intentionally introduced in this baseline environment. We will discuss this in next section where we will create a sandbox to test its fix. ##### Step 4: Creating and Testing a Signadot Sandbox The Hotrod application consists of 4 services: _frontend_, _location_, _driver_, and _route_. We already noticed a bug in the “route” service that we want to fix and test through sandbox. We will deploy the docker image of that fix in a docker registry (in this case, Dockerhub) and reference this image in the sandbox configuration file. We can either create the sandbox through the Signadot dashboard or CLI. In both cases, the sandbox configuration file will be the key. Here are the details of the configuration file. ###### Example Configuration for Forking a Deployment We will continue with the example of Hotrod application and create a sandbox _negative-eta-fix_ as an example. In the baseline environment, when you book a ride, the ETA shows negative, which is a bug. We have tagged its fix as _signadot/hotrod:quickstart-v3-fix._ Below is the sandbox configuration for _negative-eta-fix_ sandbox\_.\_ You can just place this configuration in the Signadot dashboard when creating a sandbox and apply it. The sandbox containing the fix will be deployed immediately. See the below screenshot for reference. *Image: Signadot dashboard confirming the negative-eta-fix sandbox was created and applied, with the sandbox YAML spec forking the route deployment* You can achieve the same through CLI. Just run the below command providing the path to the sandbox configuration file and specifying the Signadot cluster name. The terminal will display the status of sandbox creation as below: You will see the status of the sandbox as “Ready” in the Signadot dashboard as well. At this stage, we have two versions of the “route” service. The buggy one in the baseline environment, and the fixed one in the sandbox. So how to divert traffic to the sandbox one? The answer lies in the last step which is about routing the request. ##### Step 5: Routing to the Sandbox The Signadot operator is responsible for routing the request to the sandbox instead of the baseline version of “route” service but it needs the routing key for this purpose. To implement this, pass the header _baggage: sd-routing-key=_ in your requests, and the Signadot Operator will redirect traffic to the sandboxed service instead of the baseline. For this example, we will be using [Chrome extension](https://chromewebstore.google.com/detail/signadot/aigejiccjejdeiikegdjlofgcjhhnkim) for Signadot although we can use any extension that allows setting a header. See the below screenshot of the Signadot Chrome extension setting the header. You will see a list of all the sandboxes that are created. We will select the _negative-eta-fix_ sandbox here. *Image: Signadot Chrome extension over the HotROD app listing the negative-eta-fix sandbox to route requests to* Now that the routing is set, we will reload the Hotrod application on same URL and we can see the ETA is now positive. *Image: HotROD app with requests routed to the negative-eta-fix sandbox, showing the route service handling the request and the driver arriving in a positive ETA* After completion of sandbox testing, you can either delete it from Signadot dashboard or through the below command: #### Conclusion With Signadot, creating efficient and scalable [preview environments in Kubernetes](https://www.signadot.com/solutions/preview-environments/) has never been easier. Its ability to isolate specific microservices in lightweight sandboxes reduces resource usage, increases productivity, and improves code quality. By incorporating Signadot into your development workflow, you can accelerate testing and release cycles while minimizing costs at the same time. Why not try [Signadot](https://www.signadot.com/) and transform the way your team builds and tests microservices? Explore how companies like [Brex](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) and [DoorDash](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/) have scaled their testing and improved productivity with Signadot. #### Related Posts - [ *Image: How to Migrate from Telepresence to Signadot: A Technical Guide* Ephemeral EnvironmentsTutorials/Guides ### How to Migrate from Telepresence to Signadot: A Technical Guide August 27, 2025 ](https://www.signadot.com/blog/how-to-migrate-from-telepresence-to-signadot-a-technical-guide/) - [ *Image: Using Istio or Linkerd To Unlock Ephemeral Environments* Ephemeral EnvironmentsTutorials/Guides ### Using Istio or Linkerd To Unlock Ephemeral Environments March 7, 2025 ](https://www.signadot.com/blog/using-istio-or-linkerd-to-unlock-ephemeral-environments/) - [ *Image: Reimagining Environments for Cloud Native Architectures* Ephemeral Environments ### Reimagining Environments for Cloud Native Architectures June 2, 2025 ](https://www.signadot.com/blog/reimagining-environments-for-cloud-native-architectures/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control Source: https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/ Summary: What drives the cost of an ephemeral environment per pull request, a full copy against duplicating only what changed at 1 to 300 concurrent environments, the costs that never reach the invoice, the controls that work, pricing models, and figures from Brex, Arkestro and Yellow.ai. ### What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control What drives the cost of ephemeral environments, a full copy per pull request versus only what changed, the hidden costs, the controls that work, and what Brex, Arkestro and Yellow.ai report. Author Arjun Iyer Published July 17, 2025 Updated October 11, 2026 Reading time 10 min Ephemeral environments cost whatever they run, for as long as they run. Copy the whole application for every pull request and the bill grows with the number of services times the number of open pull requests. Duplicate only what each change touched and plug it into a shared environment, and the bill grows with the changed services alone, which at any real concurrency is an order of magnitude less. This article is the cost part of the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), which covers what they are, how they work on Kubernetes and the tools that build them. The question here is narrower: what drives the bill, where the costs hide, and what teams do to keep an environment per pull request affordable as coding agents multiply the number of changes. For the per-pull-request case and the platforms that offer it, see the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/). #### What drives the cost of ephemeral environments Five things set the bill, and only the first three show up on a cloud invoice: 1. **What each environment runs.** Every service, database and queue in an environment reserves CPU and memory, and the node count follows those requests. 2. **How many exist at once, and for how long.** An environment for a pull request that waits two days for review runs for two days, whether or not anything tests it. 3. **Data.** A database restored or seeded per environment costs storage and minutes of wall-clock time on every pull request. 4. **Waiting.** Minutes of spin-up per environment, multiplied by every developer, CI job and agent that needs one. 5. **Operations.** Engineering time to keep the environment automation, the data seeding and the shared services working. The model you choose sets the first item. Lifetimes and cleanup set the second. Everything else follows from those two. #### A full copy per change vs duplicating only what changed Take a system of 40 services where a typical change touches two. A full copy per change, in its own cluster or namespace, runs 40 service instances per environment. Duplicating only what changed runs the 40 stable versions once and adds two instances per change. The table counts running instances, which is what drives node count and the cloud bill. Concurrent environments Full copy per change Only what changed 1 40 instances 42 instances 10 400 instances 60 instances 30 1,200 instances 100 instances 100 4,000 instances 240 instances 300 12,000 instances 640 instances Service instancesfull copy per changechanged services only4042400601,2001004,00024011030100concurrent environments, 40-service system A full copy per change scales with the whole stack. Duplicating only what changed scales with the changed services. full copy per changechanged services only4042400601,2001004,00024011030100concurrent environments, 40-service system A full copy per change scales with the whole stack. Duplicating only what changed scales with the changed services. The same arithmetic holds at any size. A team with 20 services and 10 feature branches open at once runs 200 service replicas with full copies, and about 40 when each branch duplicates the two services it changed. In dollars, a 50-service system that fits on a single 32-vCPU, 128 GiB node such as an m6a.8xlarge costs about $1.38 an hour at [AWS on-demand prices](https://aws.amazon.com/ec2/pricing/on-demand/), roughly $12,000 a year per always-on copy, so 50 copies cost more than half a million dollars a year before storage and data transfer. Three things follow from the shape: - **Duplicating only what changed has a fixed cost**, the shared stable versions, which is why it is not cheaper at one environment. The gap widens with every environment after the first. - **A full copy bills for concurrency, not for the size of the change.** A one-line fix and a schema migration cost the same 40 instances, and 100 open pull requests pay for 100 copies of the system. - **Most of a full copy is waste.** Brex ran more than 800 services so a developer could work on one or two of them, and [on the margin 99.8 percent of a duplicated environment’s infrastructure cost went to services nobody was testing](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/). The models that build ephemeral environments on Kubernetes sit along the same line: Model What each environment runs Spin-up Cost as environments multiply Cluster per environment Every service, every datastore and a control plane Tens of minutes, under a minute for a virtual cluster Highest. Grows with the whole stack plus cluster overhead Namespace per environment Every service and datastore, on shared nodes Minutes, growing with the application High. Grows with the whole stack Only what changed, on a shared cluster The changed services, plus a database branch or topic when the change needs one Seconds Low. Grows with the changed services, on top of one shared set [Isolation models for ephemeral environments](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) compares the models on isolation and fidelity as well as cost. #### The costs that never reach the invoice Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The compute bill is the visible part. Four other costs are usually larger. - **Idle time.** A full copy runs from the moment a pull request opens until it merges, and most of that time nothing is testing it. Reviews take hours or days, and the environment bills for all of them. - **Waiting.** Brex’s namespace-based previews took 30 to 60 minutes to deploy, and moving to duplicating only what changed brought that [under five minutes](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/). Multiply the difference by every pull request and every rerun. A shared staging environment has the same cost in another form, a queue of changes waiting for their turn. - **Overprovisioning.** Test environments often copy production’s resource requests, sized for production traffic, into every copy. Environments that serve a handful of test requests need a fraction of that, and copied requests are paid for whether or not they are used. - **Building and running the system.** Teams that build environments per pull request in-house spend years of platform time on it. Bitso [spent a few years on its own full-copy preview environments](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) and found them expensive in both time and cost, and Arkestro put the [implementation cost it avoided by not building in-house at $120,000](https://www.signadot.com/case-studies/how-arkestro-supercharged-developer-efficiency-and-slashed-costs-with-signadot/). #### How teams keep ephemeral environment costs under control Three controls matter more than the rest: duplicate only what each change needs, put a time to live (TTL) on every environment, and size test environments for test load rather than production load. The rest are refinements. 1. **Duplicate only what changed.** The largest single lever, because it changes what the bill scales with. Each change duplicates the services it touched, and a database or queue only when it writes destructively, while request routing and message routing connect those copies to the shared environment. 2. **Put a TTL on every environment, and delete on merge or close.** Most of an environment’s life is spent waiting for review. Yellow.ai’s platform team called Signadot [cost effective because of the TTL on sandboxes](https://www.signadot.com/case-studies/how-yellow-ai-enabled-parallel-feature-development-using-signadot/) and because only the application under test is deployed. 3. **Sleep what you cannot delete.** Full-copy environments that must stay up for a long review can scale to zero after a period of inactivity and wake on the next request. 4. **Right-size requests.** Set CPU and memory requests for test traffic, run one replica of each service, and keep autoscalers at their minimum in environments that nobody loads. 5. **Use cheaper capacity.** Run environment workloads on a node pool of spot or preemptible instances, since an environment that loses a node can be recreated. 6. **Branch or share data instead of copying it.** A database branch from Neon or Xata, or the shared database with each environment’s test data kept apart, avoids a full restore per pull request. 7. **Cap and measure.** Label every environment with its pull request and team, attribute cost with a tool such as [OpenCost](https://www.opencost.io/), set a concurrency limit per team, and review environment count, cost and spin-up time monthly. A short checklist covers the same ground: - Each change deploys only the components it changed. - The stable services and infrastructure are shared, not copied. - Environment creation and cleanup are automated, from CI and from the CLI. - Existing CI pipelines and test frameworks run against the environment unchanged. - Cost per environment is a number someone reviews. #### Cost at scale: hundreds of environments and coding agents Coding agents change the concurrency number, and concurrency is what full copies bill for. A 50-engineer team that opened 100 to 150 pull requests a day can face [closer to 1,000 once agents write much of the code](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/), and an agent wants an environment for every attempt in its build, test and fix loop, not one per feature. At Bitso, [pull requests per developer rose 60 percent in a year](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) as agents took on more of the work. As an illustration, 1,000 environments a day that each live two hours average about 80 at once, and peaks run several times higher. At 250 concurrent environments of the 40-service system above, a full copy per change runs 10,000 service instances and duplicating only what changed runs 540. Teams that operate in several regions or clusters multiply the full-copy figure again for each one, while duplicating only what changed adds one shared set per cluster. At that scale the model is the cost decision, and TTLs and right-sizing tune the result. #### What pricing models ephemeral environment platforms use Platforms for ephemeral environments charge in one of three ways: per seat, by usage such as environments created or environment hours, or a platform fee while the environments run in your own cloud account and the compute lands on your cloud bill. Whichever model a vendor uses, the compute the environments consume is paid for somewhere, so the number of instances each environment runs matters more than the line item it appears on. Signadot’s Business plan is usage-based, metered on sandboxes created per month, test invocations and concurrent local connections, with the environments themselves running on your own cluster, as the [pricing page](https://www.signadot.com/pricing/) details. #### What teams report - **Brex** replaced namespace-based previews across more than 800 microservices and [cut infrastructure cost by 99 percent, about $2 million a year](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/), while previews went from 30 to 60 minutes to under five. - **Arkestro** reported [more than $1 million a year in developer bandwidth](https://www.signadot.com/case-studies/how-arkestro-supercharged-developer-efficiency-and-slashed-costs-with-signadot/), from a 12.5 percent increase across a 40-person engineering team, five hours saved per developer per week, and $120,000 in implementation cost avoided. - **Yellow.ai** tried a virtual cluster per developer, found it [cost quite a lot and was hard to keep in sync](https://www.signadot.com/case-studies/how-yellow-ai-enabled-parallel-feature-development-using-signadot/), and moved to one shared cluster where each change deploys only the application under test and every environment carries a TTL. #### Getting started The path to lower-cost ephemeral environments on a cluster you already run has four steps: 1. Install the Signadot operator in the cluster you already run, such as staging, next to the stable versions of your services. 2. Create an ephemeral environment per pull request from CI, as the [PR sandboxes guide](https://www.signadot.com/docs/guides/set-up-pr-sandboxes) shows, or per task from a coding agent through the CLI or the [MCP server](https://www.signadot.com/docs/integrations/mcp). Each one duplicates only what the change needs and carries a TTL. 3. Run your existing tests against it: Playwright, Cypress or Selenium for the UI, Postman or RestAssured for APIs, or in-cluster with [Jobs](https://www.signadot.com/product/jobs/). [Smart Tests](https://www.signadot.com/product/smart-tests/) compare the changed version of a service against the stable one to flag regressions. 4. Post the result back to the pull request and let the TTL remove the environment. #### Related reading - [Ephemeral environments: the complete Kubernetes guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Isolation models for ephemeral environments: full copy vs shared cluster](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) - [Smart ephemeral environments: share more, copy less](https://www.signadot.com/blog/smart-ephemeral-environments-share-more-copy-less/) - [How Brex scales developer testing across hundreds of engineers](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) - [Cut the cost of Kubernetes preview environments](https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/) - [The complete guide to staging environments](https://www.signadot.com/staging-environments/) #### Related Articles - [ *Image: Best Microservices Testing Tools for Kubernetes in 2026* ### Best Microservices Testing Tools for Kubernetes in 2026 July 17, 2025 ](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) - [ *Image: AI Powered Contract Testing for Microservices Excellence* Contract Testing ### AI Powered Contract Testing for Microservices Excellence July 3, 2025 ](https://www.signadot.com/articles/ai-powered-contract-testing-for-microservices-excellence/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## ArgoCD Preview Environments: The ApplicationSet Pattern Source: https://www.signadot.com/articles/argocd-preview-environments/ Summary: How Argo CD's ApplicationSet pull request generator stamps out a preview per PR, where duplication strains as stacks and PR volume grow, and when request routing on a shared cluster fits better. ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains How Argo CD's ApplicationSet pull request generator builds a preview per PR, where that duplication strains as stacks and PR volume grow, and when shared-cluster request routing fits better. Author Arjun Iyer Published July 24, 2026 Reading time 9 min **ArgoCD preview environments** give every open pull request its own running copy of the application. The standard way to build them is an ApplicationSet with the pull request generator: it watches your repository and stamps out a full [Argo CD](https://argoproj.github.io/cd/) Application for every open PR, deployed into its own namespace and exposed at its own URL. The pattern is GitOps-native, widely used, and worth understanding in detail, including the points where it starts to strain. This article walks through the mechanics first, then the failure modes, then the alternative architecture, duplicating only what changed on a shared cluster, for teams whose service count or PR volume has outgrown per-PR duplication. The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares the ApplicationSet model with the three other ways to build them on Kubernetes, and the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares Argo CD with the preview platforms built for this job. #### The pattern: one Application per open pull request The [pull request generator](https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/applicationset/Generators-Pull-Request.md) polls your source control provider and produces one set of template parameters per open PR: the PR number, the branch, and the head commit SHA. The ApplicationSet controller renders an Application from each, and Argo CD deploys it like any other. GitHub and GitLab are the common providers, with GitLab configuration keyed by project rather than owner and repository, and label filters let you gate previews to PRs that opt in with a preview label. Configure webhooks early. The generator’s default is polling on an interval, which means a preview can lag a pushed commit by minutes. Source control webhooks trigger the ApplicationSet on push, so previews create and update within seconds of the commit. ReviewerPull requestApplicationSetcontrollerApplicationpreview-123Namespacepr-123full stackpreview URLAuto-pruned on PR close Each open PR gets its own Application, namespace, and preview URL, all removed when the PR closes. Pull requestApplicationSetcontrollerApplicationpreview-123Namespacepr-123full stackpreview URLReviewerAuto-prunedon PR close Each open PR gets its own Application, namespace, and preview URL, all removed when the PR closes. A trimmed example shows the moving parts: ``` apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: pr-previews spec: generators: - pullRequest: github: owner: your-org repo: your-service # tokenRef is required for private repos. Anonymous access # only sees public repos and has a lower rate limit. requeueAfterSeconds: 120 template: metadata: name: preview-{{number}} spec: project: default source: repoURL: https://github.com/your-org/your-service targetRevision: "{{head_sha}}" path: deploy/chart helm: parameters: - name: image.tag value: "{{head_sha}}" - name: ingress.host value: pr-{{number}}.preview.example.com destination: namespace: pr-{{number}} server: https://kubernetes.default.svc syncPolicy: automated: prune: true syncOptions: - CreateNamespace=true ``` ##### Per-PR namespaces and Helm overrides Each Application targets a namespace derived from the PR number, so environments cannot collide. [Helm](https://helm.sh/) value overrides do the per-PR specialization: the image tag pins to the head SHA, the ingress hostname embeds the PR number, and resource requests and replica counts shrink to preview scale. Teams on Kustomize do the same with per-PR overlays. ##### DNS, ingress, and the preview URL A wildcard DNS record for `*.preview.example.com` points at the cluster’s ingress, and each preview’s ingress rule claims its own hostname. [ExternalDNS](https://github.com/kubernetes-sigs/external-dns) can manage the records automatically, and a wildcard TLS certificate keeps the URLs presentable to stakeholders. The result is the per-PR link that gets posted back to the pull request by CI. ##### Cleanup When a PR closes or merges, the generator stops emitting its parameters, the controller deletes the Application, and automated pruning removes the namespace. Two backstops are worth adding: a time-to-live sweep for environments that leak past the prune, and explicit handling for persistent volume claims, which otherwise outlive the namespace’s intent and keep billing. ##### Practices that keep the fleet healthy A few conventions keep a preview fleet manageable: - Apply a resource quota to every preview namespace, so one runaway PR cannot starve the rest. - Label everything with the PR number and team, which is what makes the preview line item attributable when the cluster bill needs explaining. - In monorepos, filter the generator by changed path so a docs-only PR does not deploy the stack. - Keep preview Helm values in the same file lineage as production values, overriding the minimum, because every hand-copied value is future drift. - Treat secrets deliberately, since stamping production credentials into short-lived namespaces per PR widens the blast radius of every leak. #### Where the pattern strains The mechanics above are sound, and for a single service or a small stack they hold up indefinitely. The strain shows up with system size and PR volume, on several fronts at once. Spin-up time grows with the stack. A preview that needs twenty services, migrations, and seed data takes minutes to become clickable, and slow previews stop being consulted. Data seeding is its own tax, since every namespace starts empty and realistic fixtures age badly. Cost scales with the number of services in the stack multiplied by the number of open PRs. Every open pull request holds a complete copy of every service, mostly idle, and preview fleets routinely become one of the larger line items in a cluster bill. Reduced replica counts soften this but do not change the shape of the curve. Operational overhead scales the same way, because every preview namespace is an environment the platform team is on the hook for: its seed data, its config currency, and the question of why it came up broken. Cluster capacity becomes a scheduling problem of its own, since preview load is spiky, quiet at night and heavy before a release, so teams either overprovision nodes for the peak or bolt on autoscaling and accept slower previews at the spike. On managed clusters like Amazon Elastic Kubernetes Service (EKS) this turns into node group and autoscaler tuning that exists only to serve previews. Two less visible problems compound over time. Preview Helm values drift from production values, so previews slowly stop predicting production behavior. And third-party dependencies, payment sandboxes, external APIs, message brokers, either get mocked per namespace or shared with no isolation at all. Coding agents amplify all of this. A team whose PR volume multiplies finds the duplication model multiplying its spin-up queues, its bill, and its fleet of environments to keep alive. #### The alternative: route requests instead of duplicating the stack The other architecture keeps one stable environment running, continuously deployed by Argo CD exactly as today, and deploys only the changed service per PR, as an isolated version beside the stable one. Requests carrying that PR’s routing key in a header are steered to the changed version at the sidecar or mesh layer, and every other request flows through stable services. The approach is GitOps-compatible rather than GitOps-replacing. Argo CD keeps owning the stable environment, and the preview layer operates on top of it, which also means previews exercise the real, current versions of every dependency instead of a per-namespace copy that seeded an hour ago. Signadot implements this model with Sandboxes, lightweight ephemeral environments created per pull request by CI, each holding the changed service inside the shared cluster and exposed at a hosted preview URL that injects the routing key. Route groups combine several changed services under one key when a change spans repositories. The first difference a team feels is developer experience. A preview is ready as soon as one container starts, in seconds rather than the minutes a full-stack deploy and seed takes, and it stays that fast whether the stack has five services or three hundred, because nothing else needs to be deployed. Fast previews are the ones reviewers and product managers actually click on every PR, instead of saving them for the changes that seem risky. Operational overhead drops with it. The platform team runs one baseline environment, kept in sync by Argo CD, instead of a fleet of per-PR copies that each need seed data, current config, and an answer when they come up broken. Capacity planning stops being a preview-specific problem, since the marginal preview is one or two pods rather than a namespace full of them. Cost follows the same shape. Each preview costs one or two service instances instead of a full system copy, so the bill scales with the size of the change, not the size of the stack. That economics is also what lets the model absorb agent-amplified PR volume: when coding agents multiply the number of open changes, previews that start in seconds and cost a single service multiply with them, without multiplying the cluster. Shared clustersyncscreatesArgoCDpreview URLpull requestservice-1service-1bSANDBOXservice-3dbtagged requestshared dependencies ArgoCD keeps syncing the shared cluster of stable services. The Sandbox holds only the change, and the preview URL's tagged requests route through it. syncsArgoCDpreview URLpull requestservice-1service-1bSANDBOXservice-3dbtagged requestshared dependencies ArgoCD keeps syncing the shared cluster of stable services. The Sandbox holds only the change, and the preview URL's tagged requests route through it. Three things have to be in place for this to work. Services must propagate trace context headers across hops, stateful resources like databases and queues need explicit per-resource isolation, and a platform component runs in the cluster. The [preview environments solution page](https://www.signadot.com/solutions/preview-environments/) covers the managed version of the workflow. Keep Argo CD, add previews that start in seconds Signadot gives each pull request a hosted preview URL backed by a lightweight sandbox on the shared cluster Argo CD already syncs, so a preview costs one service instead of a stack copy. The free tier is open to every developer. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### When each approach fits ApplicationSet: duplicate per PRRequest routing: only what changedpr-101full stackpr-102full stackpr-103full stackpr-1011 servicepr-1022 servicespr-1031 serviceCost: services x PRsCost: one or two services per PR ApplicationSet cost grows with every service and every open PR. Request routing costs one or two services per PR. ApplicationSet: duplicate per PRpr-101full stackpr-102full stackpr-103full stackCost: services x PRsRequest routing: only what changedpr-1011 servicepr-1022 servicespr-1031 serviceCost: one or two services per PR ApplicationSet cost grows with every service and every open PR. Request routing costs one or two services per PR. The ApplicationSet pattern fits: - stacks small enough to duplicate comfortably - previews that must exercise cluster-level changes, such as operators, ingress, or the manifests themselves - teams that want the whole mechanism in open source tools they already run It is the right default for a handful of services and moderate PR volume. Duplicating only what changed on a shared cluster fits: - large service graphs, where a full copy per PR is slow to start and expensive to hold - teams that want previews ready in seconds, so reviewers and stakeholders keep clicking them - platform teams that want one environment to operate instead of a fleet - high or agent-amplified PR volume The two are not mutually exclusive, and mature platform teams often run both. Application code changes, the overwhelming majority of PRs, get routed previews on the shared cluster, while the occasional PR that changes manifests, operators, or the ingress layer gets a full ApplicationSet preview, since cluster-level changes are exactly what request routing cannot isolate. Gating the expensive path behind a preview label keeps the default cheap. The [preview environments guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares these architectures alongside vCluster and Okteto, and [Per-PR Preview URLs, Explained](https://www.signadot.com/articles/per-pr-preview-urls/) covers the URL pattern itself, including access, data, and cleanup. For the routed model end to end, see this [walkthrough of cost-efficient preview environments](https://www.signadot.com/blog/create-cost-efficient-preview-environments-in-kubernetes-with-signadot/). #### Frequently asked questions How do ArgoCD preview environments work? An ApplicationSet with the pull request generator polls your source control provider and emits one set of template parameters per open PR. The controller renders an Argo CD Application from each, deploying that branch's code into a per-PR namespace behind a per-PR URL, and prunes the Application and namespace when the PR closes or merges. Where does the ApplicationSet preview pattern strain? Spin-up takes minutes because the full stack must deploy, migrate, and seed before the preview is clickable, cost scales with services multiplied by open PRs, and the platform team ends up operating a fleet of environment copies. All three compound as the service graph and PR volume grow, especially once coding agents multiply the number of open PRs. What is the alternative to duplicating an environment per pull request? Duplicating only what changed, on a shared cluster: one stable environment stays continuously deployed by Argo CD, each PR deploys only its changed services, and requests carrying the PR's routing key are steered to them. Previews are ready in seconds, the platform team operates a single environment instead of a fleet, each preview costs one or two service instances, and the model absorbs agent-amplified PR volume. Can I use request routing without giving up Argo CD? Yes. Argo CD keeps syncing the stable environment exactly as before, and the routing layer operates on top of it. Mature platform teams often run both: routed previews for application code changes, which are the overwhelming majority of PRs, and a full ApplicationSet preview for the occasional PR that changes manifests, operators, or ingress. #### Related Articles - [ *Image: Preview Environments: The Complete Guide* ### Preview Environments: The Complete Guide July 31, 2025 ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [ *Image: Per-PR Preview URLs, Explained* ### Per-PR Preview URLs, Explained July 21, 2026 ](https://www.signadot.com/articles/per-pr-preview-urls/) - [ *Image: Preview Environments for QA and Stakeholder Review* ### Preview Environments for QA and Stakeholder Review August 5, 2026 ](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Preview Environments for QA and Stakeholder Review Source: https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/ Summary: How per-PR preview environments move QA, product, and design feedback to before the merge, what changes for the QA role, and what it takes to run the workflow credibly. ### Preview Environments for QA and Stakeholder Review How a preview environment per pull request moves QA, product and design review ahead of the merge, what changes for the QA role, and the access, data and teardown the workflow needs. Author Arjun Iyer Published August 5, 2026 Reading time 7 min **Preview environments for QA and stakeholder review** give every pull request a running version of the application that anyone can open from a link, before the merge happens. That inverts the usual order. On most teams, the first person to see a feature running is not the person who asked for it: QA meets the change after it merges, on a shared staging environment carrying everyone else’s changes too, and the product manager often meets it in production. Preview environments are one kind of ephemeral environment, and the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares the four ways to build them on Kubernetes. A preview environment is an on-demand, isolated, ephemeral deployment created automatically for a specific branch or pull request. This article is about what that unlocks for the people around the code: QA leads, product managers, designers, and the engineering managers who own the release process. The infrastructure that makes it affordable is covered in the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/). #### Where review happens today The standard sequence puts every reviewer downstream of the merge. Code review looks at the diff, not the behavior. QA tests on a staging environment where a dozen changes landed since yesterday, so a failure could belong to any of them and attributing it means digging through everything that merged. Stakeholders see the feature when it ships, which is the most expensive possible moment to learn it missed the intent. Code reviewon the PRMergeQA finds bugs(shared staging)PM sees it(in production) Today every reviewer sits downstream of the merge, so feedback travels backward across it. Code reviewon the PRMergeQA finds bugs(shared staging)PM sees it(in production) Today every reviewer sits downstream of the merge, so feedback travels backward across it. The cost is more than bug latency. Feedback that arrives weeks after the code was written lands on a developer who has moved on, so every round trip carries a context-switch tax. And a contended staging environment adds its own failure mode, teams overwriting each other’s deployments and losing days to bugs that were never in anyone’s code. #### The per-PR review workflow With previews in place, the pull request becomes the review surface for everyone, not only engineers. When a PR opens, CI creates an environment for it and posts the URL on the PR and in the ticket. How that URL gets produced and wired into the PR is its own topic, covered in [Per-PR Preview URLs, Explained](https://www.signadot.com/articles/per-pr-preview-urls/). From there, each reviewer uses the same link differently. Developerpushes fixesCIbuilds the previewOpen PRpreview URL postedMergeQA · Product · Designreview the preview linkpushfeedbackcomments One loop: push, preview, feedback, fix. The merge comes after the loop settles. Developerpushes fixespushCIbuilds the previewOpen PRpreview URL postedcommentsQA · Product · Designreview the preview linkMergefeedback One loop: push, preview, feedback, fix. The merge comes after the loop settles. QA tests the change in isolation, against real, current versions of the system’s other services rather than a snapshot or a mock. One change per environment means a found bug has exactly one suspect, no bisecting a staging pileup. The fix arrives as a push to the same PR, and the same URL updates in place. Product managers and designers click the link. There is no branch to pull, no build to run, no request to an engineer for a demo. Design reviews rendering and interaction on the running feature, product confirms the behavior matches the intent, and both leave comments while the author still has the whole context in their head. Acceptance happens before merge, so the merge itself becomes a non-event. #### What changes for the QA role The gatekeeper model puts QA at the end of the line, batch-testing a release candidate under deadline pressure. Previews remove the batch. QA reviews changes one at a time, as they appear, and the release stops accumulating untested surface area between cycles. That shifts where QA time goes. Less of it is spent reproducing environment-specific failures and arguing about whose change broke staging, and more goes to exploratory testing per change and to promoting the checks worth keeping into automated suites that run against the same per-PR environments. The role moves upstream, from inspecting the finished batch to reviewing each change while it is still cheap to reject. #### How to tell it is working For the engineering manager who owns the rollout, a few measures separate a preview program that changed the workflow from one that added a link nobody clicks: - Escaped defects, the bugs QA or customers find after merge, should fall within a quarter, because the same bugs are being caught on the PR instead. - Time from PR opened to first non-author feedback should compress from days to hours. - The fraction of PRs whose preview is visited by someone other than the author measures whether stakeholders actually participate. - Release batch size should shrink toward single changes, since the point of previews is that merges stop waiting for a batch QA pass. #### What stakeholders do with a preview link The audience for a preview URL is wider than QA, and each group uses it differently: - Product managers do acceptance on the open PR instead of filing post-release correction tickets. - Designers check real rendering with real data instead of static mocks. - Documentation writers capture screenshots of features that have not shipped. - Support and sales teams see what is coming without waiting for a release note. The same links verify behavior for web and mobile applications alike, and reviewers are not always human: coding agents can run end-to-end tests against the same per-PR preview before a person ever opens it. None of these people will ever run kubectl, which is the point. The preview link has to work at the browser level, behind the company’s single sign-on, or it will not get clicked. #### What it takes to run this credibly The workflow above has one load-bearing prerequisite: per-change environments must be cheap and fast enough that creating one for every pull request is unremarkable. If a preview takes twenty minutes to build or carries the cost of a full environment copy, teams start rationing previews to the changes that seem risky, and the workflow quietly collapses back to batch review. That economics problem is why the architecture matters. [Duplicating the full stack per PR](https://www.signadot.com/articles/argocd-preview-environments/) works for small systems. Larger service graphs get there by sharing one live cluster and routing each PR’s requests to the services it changed. The first payoff is the reviewer’s experience: only the changed service deploys, so the preview is ready in seconds, and a link that is always ready is a link stakeholders keep clicking. The platform team runs one baseline environment instead of a fleet of copies to seed, patch, and answer for. And each preview costs one or two service instances rather than a stack copy, which is what keeps the practice affordable as coding agents multiply the number of open PRs waiting for review. This is what Signadot provides as a managed workflow: per-PR ephemeral environments on the cluster you already run. The [preview environments solution page](https://www.signadot.com/solutions/preview-environments/) covers what that looks like. Beyond the environments themselves, three practical pieces make or break adoption: - Access has to be one click for non-engineers, meaning single sign-on rather than credentials. - Data has to be realistic enough to review against, whether shared from the stable system or seeded. - Teardown has to be automatic on merge, so nobody manages an environment fleet by hand. A review link on every PR, before the merge Signadot gives each pull request a hosted preview URL backed by a lightweight sandbox on your shared cluster, so QA, product, and design review the change against real dependencies while the author still has context. The free tier is open to every developer. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### The change that matters Preview environments are usually sold as infrastructure, but the durable value is organizational: feedback moves to the moment it is cheapest to act on, and the merge becomes the end of review rather than the beginning. For QA leads and product owners evaluating the pattern, the test is concrete. Count how far feedback currently travels back from staging and production, and what each of those round trips costs. #### Frequently asked questions What is a preview environment for QA and stakeholder review? A preview environment is an on-demand, isolated, ephemeral deployment created automatically for a specific branch or pull request. A link posted on the PR opens the running change, so QA, product managers, and designers can review it before the merge instead of after. How do preview environments change the QA workflow? QA stops batch-testing a release candidate on shared staging and reviews changes one at a time, as they appear. Each change runs in isolation against real, current versions of the system's other services, so a found bug has exactly one suspect, and the fix arrives as a push to the same PR with the same URL updating in place. What do product managers and designers do with a preview link? They click it. There is no branch to pull or build to run: product confirms the behavior matches the intent, design reviews rendering and interaction on the running feature with real data, and both leave comments on the PR while the author still has the context in their head. What does it take to give every pull request a preview environment? Previews must be cheap and fast enough that creating one per PR is unremarkable. Larger service graphs get there by sharing one live cluster and routing each PR's requests to its changed services, so a preview is ready in seconds and costs one or two service instances. Beyond that, access needs single sign-on, data has to be realistic, and teardown has to be automatic on merge. #### Related Articles - [ *Image: Preview Environments: The Complete Guide* ### Preview Environments: The Complete Guide July 31, 2025 ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [ *Image: Per-PR Preview URLs, Explained* ### Per-PR Preview URLs, Explained July 21, 2026 ](https://www.signadot.com/articles/per-pr-preview-urls/) - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Signadot vs Garden.io: Which Kubernetes Environment Model Fits Your Team? Source: https://www.signadot.com/articles/signadot-vs-garden-io/ Summary: How Garden's full-stack environment automation compares to Signadot's shared-cluster Sandboxes: the duplication versus routing tradeoff, a feature-by-feature table, and a decision framework by stack size and bottleneck. ### Signadot vs Garden.io: Which Kubernetes Environment Model Fits Your Team? How Garden's full copy of the stack per environment and Signadot's Sandboxes on a shared cluster each work, where each strains, and which fits your stack size and number of environments. Author Arjun Iyer Published August 6, 2026 Reading time 10 min Comparing Signadot and Garden.io comes down to two different answers to the same question: how should a team running microservices on Kubernetes get environments for development and testing? Garden builds and deploys a complete copy of your stack for every environment, with a dependency graph and caching to make repeat runs fast. Signadot deploys only the services you changed into Sandboxes, lightweight ephemeral environments on a shared cluster, and routes requests to a single shared set of stable dependencies for everything else. Neither model wins everywhere. Garden is stronger when you need reproducible environments built from zero, isolation of stateful services without extra configuration, and build and test caching across development and CI. Signadot is stronger when the stack is large, the number of concurrent environments is high, and provisioning time and infrastructure cost are the constraints. Both are ways to build ephemeral environments, and the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares them with the other two. This article explains how each tool works, compares them feature by feature, and closes with a decision framework by team size and use case. #### The problems that push teams to either tool The trigger is usually one of four problems, and which one it is matters for the choice. - A shared staging environment has become a queue. Teams wait for a turn to test, and one broken deploy blocks everyone. - CI takes too long to produce a testable environment, so end-to-end tests run after merge instead of before. - Preview environments exist but cost too much. Every open pull request duplicates the full stack, and the bill scales with PR volume. - Local development stopped working because the stack outgrew laptops, and mocks have drifted from the real services. Garden and Signadot both address these problems. They differ in the mechanism, and the mechanism determines the cost curve and suitability for your stack. #### How Garden works [Garden](https://garden.io/) is a DevOps automation tool that describes your entire system as a graph and can build, deploy, and test it from a single command. You define actions of four kinds, Build, Deploy, Test, and Run, in garden.yml files, and Garden assembles them into a dependency graph it calls the [Stack Graph](https://docs.garden.io/overview/what-is-garden). Because the graph knows what depends on what, Garden caches aggressively: the same image is never built twice, and the same test never reruns against unchanged code. Environments are full deployments of that graph. Garden templates a unique [Kubernetes namespace per developer, CI run, or pull request](https://docs.garden.io/guides/namespaces), and each namespace receives its own copy of every service in the project, datastores included. Sync mode live-reloads code changes into a running service without a rebuild or redeploy. Plugins cover Helm charts, Kustomize, plain manifests, and container builds, and Terraform and Pulumi actions can provision infrastructure as part of the same graph. The core is open source under the Mozilla Public License 2.0. Garden’s [hosted offering](https://garden.io/plans) adds team-wide result caches shared across clusters and a remote container build service. Repogarden.ymlStack GraphBuildDeployTestRundev-aliceapiwebjobsdbpr-101apiwebjobsdbpr-102apiwebjobsdb One graph, but every namespace receives a full copy of every service and datastore. Repogarden.ymlStack GraphBuildDeployTestRundev-aliceapiwebjobsdbpr-101apiwebjobsdbpr-102apiwebjobsdb One graph, but every namespace receives a full copy of every service and datastore. Where Garden fits: - One configuration drives development, CI, and preview environments, so the environment a developer tests in matches what CI runs. - Dependency-aware caching cuts repeat build and test time, which compounds in CI. - It reuses the Helm charts, Dockerfiles, and Terraform you already have. - Every environment is fully isolated, including its own datastores, with no extra per-resource setup. - Environments build from zero, so you do not need an existing healthy cluster to point at. Where it strains: - Every environment carries the full stack. Twenty services means twenty deployments, plus datastores, per developer and per open PR. - A cold environment still has to deploy and start every service, which takes minutes even when caching skips the builds. - Each service needs garden.yml actions written and maintained, and the configuration has a documented learning curve. - Infrastructure cost scales with the number of concurrent environments, which is roughly developers plus open PRs. That last point is the same ceiling any duplication-based approach runs into, whether the boundary is a namespace, a virtual cluster, or a whole cluster. [Namespace-based environments](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) hit it first because the cost is multiplicative rather than additive. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### How Signadot works Signadot is a Kubernetes-native validation platform built around [Sandboxes](https://www.signadot.com/docs/concepts/sandbox), lightweight ephemeral environments that share one cluster. A Sandbox deploys only the services a change touches. Requests carry a routing key in a header, and the routing layer sends matching traffic through the sandboxed service versions while every other hop is served by the shared stable versions already running on the cluster. Because nothing else is duplicated, a Sandbox is ready in seconds and hundreds can run concurrently at marginal cost. requestrouting key: alice-1Kubernetes clusterSandboxorders v2frontendorderspaymentsdatastore The routing key sends one hop through a Sandbox. Every other hop uses the shared stable version, so the greyed service is never duplicated. requestrouting key: alice-1Kubernetes clusterfrontendordersSandboxorders v2paymentsdatastore The routing key sends one hop through a Sandbox. Every other hop uses the shared stable version, so the greyed service is never duplicated. Testing runs where the environments are. Jobs execute existing Playwright, Cypress, or k6 suites inside the cluster against a Sandbox. Smart Tests run the same API calls against the PR version and the stable version of a service and flag behavioral differences. Plans package validation workflows so CI pipelines and coding agents can run them by name, and an MCP server lets agents create Sandboxes and validate their own changes. Where Signadot fits: - Environments arrive in seconds at marginal cost, which makes one per pull request, per developer, and per agent task affordable. - Changes are tested against real running dependencies rather than mocks, because the shared cluster runs the real services. - It works with your existing manifests. There is no per-service configuration format to adopt. - Concurrency scales past what duplication can afford. Brex replaced namespace-based preview environments with Sandboxes and [cut preview infrastructure costs by 99 percent](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/), about 2 million dollars a year. What it requires: - A shared cluster, usually staging, that runs stable versions of every service, and an owner who keeps it healthy. Signadot installs as an operator on that cluster. - Services must propagate request headers, typically trace context. Teams already running distributed tracing have this. Teams that do not must add propagation. - Isolating stateful resources is explicit. Resource plugins create per-Sandbox databases, queues, or other dependencies where a test needs them, instead of every environment getting copies by default. - Signadot is not a build tool. Images come from your existing CI, and Signadot does not build or cache them. - The platform is a commercial product rather than an open source core. The deeper version of this tradeoff is a question about where the isolation boundary sits, which is covered in [full copy vs shared cluster isolation](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/). #### Signadot vs Garden.io: feature by feature Capability Garden Signadot Environment model Full copy of the stack per namespace Only changed services deploy, the rest is shared Time to a ready environment Minutes for a cold environment, caching speeds repeats Seconds Cost per concurrent environment Full-stack resource cost Marginal, only the changed services Build and test caching Core strength, dependency-aware None, uses your CI builds Local development Sync mode reloads code into a deployed environment Local process joins the cluster and receives routed requests Datastore isolation Every environment gets its own copies by default Explicit per-resource setup through resource plugins Configuration garden.yml actions per service Works with existing manifests Infrastructure provisioning Terraform and Pulumi actions in the same graph Out of scope Coding agents No per-agent isolation model One Sandbox per agent task, MCP server built in Licensing Open source core, hosted paid tiers Commercial The caching rows favor Garden and the environment rows favor Signadot, and that split summarizes the comparison. Garden optimizes the work of producing an environment. Signadot reduces how much environment there is to produce. #### Signadot vs Garden.io by team size and use case Choose Garden when: - Your stack is small enough that a full copy per environment fits comfortably in your cluster budget. At that scale duplication is simple and the isolation is complete. - You have no stable shared cluster to route against, and environments must be built from zero, infrastructure included. - Reproducibility is the goal: one command that takes a fresh cluster to a running, tested system. - CI build and test time is your bottleneck, and dependency-aware caching would remove most of it. - Policy requires an open source, self-hostable core. Choose Signadot when: - The stack is too large to duplicate per change. The cost is multiplicative: services times concurrent environments, and it grows with the team. - You want an environment per pull request, per developer, and per coding agent at the same time, without the bill scaling the same way. - You already run a staging cluster and want every change validated against real dependencies before merge. - Time to environment is your bottleneck: reviews and tests wait on provisioning, not on builds. What is thebottleneck?Start withGardenShared clusteralready running?Start withSignadotbuild and test timeenvironment availabilityyesno Two questions decide it: what is slow, and whether a stable shared cluster already exists. What is the bottleneck?build and test timeenvironmentavailabilityStart withGardenShared clusteralready running?noyesStart withSignadot Two questions decide it: what is slow, and whether a stable shared cluster already exists. Team size is a workable proxy for stack size and concurrency. A team of five with eight services can run either model, and Garden’s single-config simplicity and caching may win outright. A team of fifty with forty services and coding agents opening PRs cannot afford a full stack per change, and the shared-cluster model becomes the only affordable way to give every change its own environment. The duplication versus routing tradeoff, alongside two other approaches, is covered in more depth in the [guide to ephemeral environments in Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/). The two tools also do not solve identical problems. Garden’s center of gravity is the build and deploy pipeline. Signadot’s is validation against real dependencies. A team whose pain is slow CI builds should evaluate Garden first, and a team whose pain is environment scarcity should evaluate Signadot first. #### How to choose Count your services, count your concurrent changes, and check whether a healthy shared cluster exists. A small stack, no shared cluster, or a build-time bottleneck points to Garden. A large stack, high concurrency, agents in the loop, and an existing staging cluster point to Signadot. For a product-level breakdown of the same comparison, see the [Signadot vs Garden comparison page](https://www.signadot.com/comparison/garden/). Further reading: - [Ephemeral Environments in Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Preview Environments: The Complete Guide](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [Local Development on Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-local-development-kubernetes/) #### Frequently asked questions What is the difference between Signadot and Garden.io? Garden builds and deploys a complete copy of your stack into a dedicated Kubernetes namespace for every environment, using a dependency graph and aggressive caching to make repeat runs fast. Signadot deploys only the services a change touches into a Sandbox on a shared cluster and routes requests to the stable shared versions for every other hop, so nothing else is duplicated. Which is cheaper, Garden or Signadot? Cost depends on stack size and concurrency. Garden's cost per environment is the full resource cost of the stack, so the bill scales with services multiplied by concurrent environments. Signadot's cost per environment is marginal because only the changed services are deployed. Below roughly ten services with low concurrency the difference is small. At forty services with dozens of open pull requests the gap is large, and Brex cut preview infrastructure cost by 99 percent moving from namespace duplication to Sandboxes. Can you use Garden and Signadot together? Yes. Signadot is not a build tool and consumes images produced by your existing CI, which Garden can be. Teams commonly keep Garden or their current pipeline for building and caching, and replace full environment duplication with Sandboxes for per-pull-request testing. Does Signadot require a shared staging cluster? Yes. Signadot needs a cluster running stable versions of every service, usually staging, with an owner who keeps it healthy. Garden has no such requirement because each environment is built from zero, which is why Garden fits teams with no stable shared cluster to route against. Does Signadot require changes to my services? Services must propagate request headers, typically trace context, so the routing layer can follow a request through the call graph. Teams already running distributed tracing usually have this. Teams that do not must add propagation before Sandboxes will route correctly. #### Related Articles - [ *Image: AI Powered Contract Testing for Microservices Excellence* Contract Testing ### AI Powered Contract Testing for Microservices Excellence July 3, 2025 ](https://www.signadot.com/articles/ai-powered-contract-testing-for-microservices-excellence/) - [ *Image: Shift-Left Testing for Microservices on Kubernetes* ### Shift-Left Testing for Microservices on Kubernetes September 30, 2026 ](https://www.signadot.com/articles/shift-left-testing/) - [ *Image: What Is Environment Parity? How to Make Dev Environments Match Production* Ephemeral Environments ### What Is Environment Parity? How to Make Dev Environments Match Production September 10, 2026 ](https://www.signadot.com/articles/environment-parity/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Integration Tests Pass With Mocks but Staging Still Breaks Source: https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/ Summary: Why mocked integration tests miss staging failures, the escalation ladder of contract tests, real-dependency tests, and scoped end-to-end tests, and the two architectures that let them run pre-merge. ### Integration tests pass with mocks but staging still breaks: how to catch it earlier Mocks check your assumptions about a dependency, not the dependency. The failures they miss, three additions that catch them before merge, and how stack size picks the environment. Author Arjun Iyer Published August 12, 2026 Reading time 9 min When integration tests pass with mocks but staging still breaks, the gap is dependency fidelity, not test coverage. A mocked test verifies your assumptions about a dependency. Staging is the first place those assumptions meet the dependency itself, and the failures that surface there are exactly the ones a mock cannot represent. Adding more mocked tests cannot close that gap. A mock encodes the dependency’s behavior on the day the mock was written, and nothing updates it when the real service changes. More mocked tests raise the coverage of your assumptions, not their accuracy. This gap is one part of the wider practice covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). Contracts have the same snapshot problem as mocks, which the [complete guide to contract testing](https://www.signadot.com/contract-testing/) covers in detail. This article covers why mocked integration tests miss what they miss, the escalation ladder of what to add, and how early each rung can realistically run. For the wider picture of what the stage is for and why the shared copy becomes a bottleneck, see the [complete guide to staging environments](https://www.signadot.com/staging-environments/). #### Why do mocked integration tests miss staging failures? A mock can only fail on what its author anticipated, and the failures that reach staging are the unanticipated ones. They cluster into four classes that live outside a mock’s reach: - **Mock drift.** The mock matches the dependency as it was when the test was written. The dependency’s team ships changes on their own schedule, and no build step fails when the mock falls behind. Comparing the changed service’s responses against the stable version’s on the same requests is one way to surface it, an approach covered in [shadow testing for microservice integration](https://www.signadot.com/blog/shadow-testing-superpowers-four-ways-to-bulletproof-apis/). Mock drift is the dependency layer of [environment parity](https://www.signadot.com/articles/environment-parity/), and the layer a local copy loses the moment it is created. - **Contract versus behavior.** A mock can match the schema exactly and still return values the real service never produces: an empty list where production always has entries, a status the service stopped emitting a year ago. - **The network is absent.** Timeouts, retries, serialization errors, and header propagation never execute in a mocked test, and each of those is a production failure class of its own. - **Deployed-only configuration.** Connection strings, feature flags, resource limits, and auth config exist only in the deployed environment, so no in-process test can exercise them. Integration failures are the failures between processes: data inconsistencies, latency behavior, fault tolerance. Unit tests cannot verify inter-service communication no matter how many of them exist, which is why a green pipeline says little about what happens when services meet. The second-order cost shows up in how teams respond. When the automated suite stops predicting staging behavior, verification moves to manual checks against the staging environment, and the suite degrades into background noise. A suite with a stable count of failing tests stops being a signal at all: teams have run for months watching that the number of failures stayed constant rather than investigating them. Shared staging then makes the feedback worse. Multiple teams deploy versions into the same environment at once, so test conditions change under your feet, and a failure could belong to any change that landed that day. 1Mock writtenmatches Service B v12Service B ships v23Tests still pass4Staging breaks The mock encodes the dependency as it was. Nothing fails when the dependency moves on. 1Mock writtenmatches Service B v12Service B ships v23Tests still pass4Staging breaks The mock encodes the dependency as it was. Nothing fails when the dependency moves on. #### What should I add to catch integration bugs before staging? The additions form a ladder. Each rung catches a failure class the one below it cannot, and each needs a higher-fidelity environment and more moving parts to run, so the question is how much fidelity your failure pattern actually demands. Simplest first: 1. **Contract tests.** A consumer and provider agree on the shape of their interaction, and each side verifies against the recorded contract without deploying the other. [Pact](https://pact.io/) is the most widely used implementation. This catches signature and schema drift in under a minute of pipeline time. It does not catch behavior, and it carries a standing cost: both sides have to write and maintain the contracts continuously, and sustaining that discipline across many teams is where the approach usually breaks down. The [Pact comparison](https://www.signadot.com/comparison/pact/) covers where the contract boundary ends. 2. **Integration tests against real dependencies.** Give the changed service an environment where its dependencies are real running services rather than mocks, then point the tests you already have at it. This rung catches behavior, routing, configuration, and serialization failures, the classes mocks miss. Something has to supply that environment, whether as a full copy of the stack or as an isolated slice of a shared cluster, and the next section compares those options. 3. **Pre-merge end-to-end tests scoped to the changed service.** Run the two or three user journeys the change touches, not the full suite. This catches cross-journey regressions and demands the most from both the environment and the suite, so scope is what keeps it viable. fidelityScoped E2Ecross-journeyregressionsIntegration againstreal dependenciesbehavior, config,routingContract testsschema drift Each rung catches what the one below it cannot, and costs more environment to run. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### How early can this realistically move? All three rungs can run pre-merge. Contract tests already do so trivially, because they verify recorded contracts without deploying anything. The open question is rungs two and three, which need a per-change environment with real dependencies, available quickly enough to sit inside a PR pipeline. They are also the rungs where pre-merge matters most, because post-merge feedback lands in the same place staging failures already land. Two architectures supply that. A full environment per PR deploys a complete copy of the stack for every open pull request. Routing on a shared cluster deploys only the changed service next to a single shared set of stable dependencies and steers tagged requests to it. Full environment per PR Routing on a shared cluster Cost per PR One full copy of the stack Only the changed service Isolation Complete, including data Per-request, shared stable dependencies Spin-up time Minutes Seconds Prerequisite Deployable manifests for every service Header propagation across every hop [Release](https://release.com/) and [Bunnyshell](https://www.bunnyshell.com/) provide the full-environment pattern as a managed workflow. It is simpler to reason about, since each PR gets a fully independent copy, and below roughly ten services it is the right call. Teams that build the same pattern in-house tend to run ten or fifteen standing test environments and absorb a second cost the table does not show: keeping every copy in parity with production configuration, which is ongoing operational work rather than a one-time setup. The crossover comes from arithmetic. Cost scales with the number of services multiplied by the number of open PRs, so a 40-service stack with 30 open PRs is provisioning 1,200 service instances to test 30 changes. Routing on a shared cluster flips the multiplier, because each PR deploys one service and shares the rest. The shared-cluster pattern is what Signadot provides. A Sandbox is a lightweight ephemeral environment holding the changed service, deployed into the cluster where the stable versions of its dependencies already run. Requests carrying the Sandbox’s routing key reach the changed service, every other call resolves to the shared stable versions, and provisioning takes seconds because nothing else is duplicated. The prerequisite is header propagation: every service forwards the routing header on each hop, and teams already propagating trace context have most of that in place. Shared staging gated by feature flags comes up in this comparison often, and it does not belong in it. Flags gate exposure after the code has merged, so nothing on the ladder moves pre-merge, and the flags add a cleanup obligation of their own. Whatever their value for release control, they are not a testing architecture. Full environment per PRenvironment copypull requestcreatesservice-1service-2service-3dbevery service deployed for this one PRRouting on a shared clusterShared clustercreatestestspull requestservice-1service-1bSANDBOXservice-2dbtagged requestshared dependencies A full copy runs every service per PR. Routing deploys the changed service into the shared cluster beside its stable version and steers tagged requests through it and back to the shared services. Full environment per PRpull requestcreatesenvironment copyservice-1service-2service-3dbevery service deployed for this one PRRouting on a shared clustertestspull requestShared clustercreatesservice-1service-1bSANDBOXservice-2dbtagged requestshared dependencies A full copy runs every service per PR. Routing deploys the changed service into the shared cluster beside its stable version and steers tagged requests through it and back to the shared services. #### How to choose Two decisions come out of this article, and they are worth separating. The first is which rungs to add, and the failure classes you actually see decide it. Shape mismatches between services point to contract tests, the cheapest rung and the one with no infrastructure requirement. Failures that only appear when services actually talk, wrong values, broken routing, misread configuration, point to the real-dependency rung, because no amount of contract or unit coverage reaches them. Recurring breakage in flows that span several services justifies the scoped end-to-end rung on the journeys that keep breaking. The second is which architecture supplies the environment for those rungs. Stack size is the main input: below roughly ten services, a full environment per PR is simpler to reason about and the duplication cost stays tolerable. Above that, the services-times-PRs arithmetic takes over, and routing on a shared cluster is the option that keeps cost and provisioning time flat as the stack grows. Header propagation is the gate. If your services already forward trace context, the shared-cluster pattern is close to free to adopt, and if they do not, that plumbing is the price of entry. The wider testing lifecycle around these stages is covered in [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). The placement question, [which tests shift left to the pull request](https://www.signadot.com/articles/shift-left-testing/) and what they need there, has a guide of its own. #### Frequently asked questions My integration tests pass with mocks but staging still breaks. What am I missing? The mocks encode assumptions about your dependencies, and staging is where the assumptions meet reality. The failure classes that get through are mock drift, behavior a schema-correct mock never produces, network effects, and deployed-only configuration. Adding a stage that tests the changed service against real dependencies is what closes the gap. Should I delete my mocked tests? No. Mocked tests are the right tool for logic inside a service, and they stay fast and deterministic. Stop asking them fidelity questions they cannot answer, and add a real-dependency stage for those. Do I need a full copy of my stack to test against real dependencies? No. A full copy per PR is one architecture, and below roughly ten services it is a reasonable one. Past that size the arithmetic turns: cost grows with the number of services multiplied by open PRs, and spin-up time tracks the slowest service in the stack. Routing on a shared cluster deploys only the changed service and resolves everything else to a single shared set of stable dependencies, which keeps cost and provisioning time flat as the stack grows. That is why it is the architecture that holds up for larger systems. How do I know when mock drift is happening? Indirectly: staging or production failures in interactions your suite says are covered. Directly: contract tests, which fail when a provider's actual interface no longer matches what consumers recorded. Comparing the changed service's responses against the stable version's on the same requests also surfaces drift. Mock drift itself produces no signal, which is the problem. #### Related Articles - [ *Image: What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control* ### What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control July 17, 2025 ](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) - [ *Image: The Ultimate Guide to Lightweight Kubernetes Environments* Tutorials/Guides ### The Ultimate Guide to Lightweight Kubernetes Environments May 1, 2025 ](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) - [ *Image: Signadot SmartTests vs Pact Contract Testing* Contract Testing ### Signadot SmartTests vs Pact Contract Testing June 13, 2025 ](https://www.signadot.com/articles/signadot-smarttests-vs-pact-contract-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## What Is Environment Parity? How to Make Dev Environments Match Production Source: https://www.signadot.com/articles/environment-parity/ Summary: Defines environment parity, the four kinds of drift (code, configuration, data and schema, dependency and topology) and how each is detected, and how to get a production-like environment by sharing real dependencies and isolating the change by request routing. ### What Is Environment Parity? How to Make Dev Environments Match Production What environment parity means, the four kinds of drift behind bugs that show up in staging but not locally, and how to get a production-like test environment without cloning production. Author Arjun Iyer Published September 10, 2026 Reading time 14 min Environment parity is the degree to which development, staging, and production run the same code, configuration, data shapes, and dependencies, so that a change behaves the same way in each. The problem that sends people looking for the term is narrower. QA finds a bug in staging, the developer cannot reproduce it on their machine, and the ticket bounces between them. That bug lives on a difference between the two environments, and there are only four kinds of difference it can live on. Parity is not a cloning exercise in which every environment becomes a copy of production. It is the question of which differences matter for the change under test, and how to remove those while accepting the rest. Environment parity is one piece of the wider practice covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers the four ways to build the per-change environments this article assumes. Most of that gap is between the developer’s machine and staging, which is what this article covers. When the staging environment itself is the problem, because nobody owns it or it is always several merges behind, the [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers how to run and share one. Parity is also what makes an automated check trustworthy when coding agents run it, which the [complete guide to software factories](https://www.signadot.com/software-factory/) calls sensor fidelity. #### What is environment parity? Environment parity means every environment a change passes through behaves like production for the properties that affect that change. The term comes from the [twelve-factor app framework](https://12factor.net/dev-prod-parity), whose tenth factor, dev/prod parity, reduces to one line: keep development, staging, and production as similar as possible. The twelve-factor text names three gaps to close: the time gap between writing code and deploying it, the personnel gap between the people who do each, and the tools gap between the backing services each environment runs. It was written for one deployable app with a few backing services. In a system of fifty services, the tools gap widens because the backing services are now other teams’ services, each with its own release cadence, and a local environment cannot hold them at production’s versions for long. Parity splits into four layers, which drift independently and are fixed by different means: - **Code parity:** the same image, built from the same commit, as what is deployed. - **Configuration parity:** the same environment variables, ConfigMaps, Secrets, feature flags, resource limits, and network policies. - **Data parity:** the same schema, and data with the same shape and distribution. - **Dependency and topology parity:** the same versions of every service the change calls, reached over the same network path, through the same mesh and ingress. #### Why “works on my machine” bugs reproduce in staging but not locally A bug reproduces only in an environment that matches production on the property the bug depends on. Staging matches production on more properties than a laptop does, so a class of bugs appears there for the first time. The developer’s machine is not wrong. It is missing the property that triggers the failure. Four mechanisms account for most of these bugs: - **Real dependency versions.** Locally, the service under test talks to mocks or to images pulled weeks ago. In staging, it talks to whatever the payments team deployed this morning, including a changed error response or a new required field. - **Real data shapes.** Seed data has no unexpected nulls, no unicode in name fields, and ten rows where production has ten million. Query planners, pagination, and serialization behave differently against each. - **Real network behavior.** A call to a process on the same host does not time out, retry, get load balanced, or negotiate mutual TLS. In staging, every call does all four. - **Real concurrency.** Two replicas race on a shared row, or a consumer processes a message before the producer’s transaction commits. One replica on a laptop cannot produce either. Making the local environment bigger helps with data and network for a while. It does nothing for dependency versions, because a local copy of a dependency stops tracking production the moment it is created, and nothing for concurrency. #### The four kinds of drift and how to detect them Drift is the process by which two environments that started identical stop being identical. Each parity layer drifts for a different reason and is detected by a different check. Treating drift as one problem is why most parity efforts stall. mainCodedriftImage digestdiffConfigurationdriftRenderedmanifest diffData and schemadriftMigrationversion diffDependency andtopology driftDependencyversion inventory Each layer drifts for its own reason and is caught by its own check. mainCodedriftImage digestdiffConfigurationdriftRenderedmanifest diffData and schemadriftMigrationversion diffDependency andtopology driftDependencyversion inventory Each layer drifts for its own reason and is caught by its own check. ##### Code drift Code drift is an environment running a different build of a service than the one on main. It happens when deploys to lower environments are manual, when a branch is pushed to a shared environment and forgotten, or when a hotfix goes to production and never lands on main. Detection is a digest comparison between the image running in each environment and the image built from the target commit. GitOps controllers such as [Argo CD](https://argo-cd.readthedocs.io/en/stable/) and [Flux](https://fluxcd.io/flux/) report this continuously as sync status. ##### Configuration drift Configuration drift is the same service running with different environment variables, ConfigMaps, Secrets, feature flags, resource limits, or network policies in different environments. It accumulates one hand edit at a time: a flag turned on in staging to unblock a test, a memory limit raised in production during an incident, a secret pointed at a vendor sandbox account with different rate limits. Detection is a diff of rendered manifests. Render the [Helm](https://helm.sh/) or [Kustomize](https://kustomize.io/) output for each environment and diff the results. Every line that differs should trace to a deliberate, reviewed per-environment override. GitOps controllers catch the rest, because a kubectl edit that is not in Git shows up as a difference from the desired state. ##### Data and schema drift Schema drift is the database schema in one environment differing from another, usually because a migration ran in one place and not the other, or in a different order. Data drift is the content differing in ways that change behavior: distribution, volume, nulls, and edge cases that fixtures never contain. Schema drift is detected by comparing the applied migration version in each environment, which every migration tool records in a table. Data drift is managed rather than detected, and the [staging environment guide](https://www.signadot.com/blog/kubernetes-staging-environment/) describes the three-tier strategy: synthetic data for most flows, anonymized production snapshots where distribution matters, and ephemeral per-change databases for destructive migrations. API schema drift between services is a contract-testing problem and out of scope here. ##### Dependency and topology drift Dependency drift is the change under test calling a different version of another service than production runs. Topology drift is reaching it over a different path: no mesh locally, a different ingress, different network policies. Both are invisible from inside the service under test, which is why they produce the bugs no one can reproduce. Detection is an inventory: for each dependency the change calls, which version does each environment run, and through what path. In a shared cluster that tracks main, every dependency is at the version production will run next, over the real mesh. On a laptop, the answer is a mock or a pinned image. [Testcontainers](https://testcontainers.com/) closes the gap for databases and brokers, because the container runs production’s version, but it does not cover the other forty-nine services. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Does your dev environment need to mirror production? No. A development environment needs to match production on the properties the change depends on, and it should differ on everything else. Chasing a full mirror is how teams end up with a laptop that cannot run the system, or a per-developer cloud environment that costs more than the developer time it saves. Where parity stops paying: - **Capacity.** Replica counts, node sizes, and autoscaling limits can be scaled down, as long as the count is at least two wherever concurrency matters. - **Data volume.** Production-scale data is needed for performance and query-plan bugs, which belong in a dedicated performance environment. - **Third-party services.** Payment, email, and identity providers offer sandbox tiers. Use them, and record the behavioral differences. - **Secrets.** These should never be identical across environments. Parity here means the same shape and injection path, not the same values. Where parity cannot be traded away is the behavior of the code path under test: the versions of the services it calls, the network hops between them, the configuration they read, and the schema they write to. Scale down capacity, never fidelity. The [integration tests pass but staging breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) article covers mocks, which are the most common way teams trade away dependency parity without noticing. #### How to get environment parity without cloning production The approach that holds up is to stop replicating dependencies and start sharing the real ones, with only the change under test isolated. Every other approach reaches parity by copying, and every copy starts drifting the moment it exists. The most commonly implemented approaches are: - **Copy production per environment.** A namespace, a virtual cluster from [vCluster](https://www.vcluster.com/), or a managed environment from [Release](https://release.com/) or [Bunnyshell](https://www.bunnyshell.com/) provisions a full stack per developer or pull request. Parity is high at provisioning time. The costs are spin-up time, cloud spend that scales with the number of copies, and dependency drift from the first minute, because each copy holds its own versions of every service. - **Connect a local process to a shared cluster.** [Telepresence](https://www.telepresence.io/) and [mirrord](https://mirrord.dev/) run the service under test on the laptop and route cluster traffic to it, so every dependency is real and current. The limit is contention: intercepting a service’s traffic affects everyone else using that service. - **Share the cluster, isolate by routing.** One environment tracks main and holds the shared stable versions of every service. Each change runs as a fork of only the services it touches, and requests carrying a routing key in a header reach the fork while all other traffic reaches the stable versions. Parity is inherited from the shared environment, cost is per change rather than per environment, and the prerequisite is that services propagate request headers across hops. shared clusterrouting keyGatewayOrdersPaymentsInventoryOrdersforkdatabase Only the changed service is forked. Everything else is the shared stable version, reached over the real network path. shared clusterrouting keyGatewayOrdersOrdersforkPaymentsInventorydatabase Only the changed service is forked. Everything else is the shared stable version, reached over the real network path. The third model is not new. [Lyft and Razorpay](https://www.signadot.com/blog/how-lyft-and-razorpay-share-development-environments-with-hundreds-of-devs/) each built internal platforms on it so that hundreds of developers could share one environment without deploying over each other. This is what Signadot provides as a product. [Sandboxes](https://www.signadot.com/docs/concepts/sandbox) are lightweight ephemeral environments on a shared Kubernetes cluster. A Sandbox forks the workloads a change modifies from the running deployment, so the fork inherits the deployed image, configuration, and resource limits rather than a hand-maintained copy. Requests carrying the Sandbox’s routing key are routed to the forked workloads, through the service mesh or through a sidecar Signadot provides, and everything else flows to the shared stable versions. Scored against the four layers: code and configuration parity come from forking the deployed workload, dependency and topology parity come from the shared cluster tracking main, and data parity is handled per Sandbox with ephemeral databases or queues where a change needs isolated state. Two prerequisites are real. Services must propagate context headers, typically [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) or B3, so the routing key survives a multi-hop call chain, and consumers on message brokers must read the routing key from message metadata and skip messages that belong to another change. The [local development guide](https://www.signadot.com/guide-to-local-development-kubernetes/) compares the first two options in more depth, including local clusters and cloud development environments. #### What AI coding agents change about environment parity An agent cannot walk to a colleague’s desk and ask how staging is configured. For a developer, low parity is a productivity problem: a bug that will not reproduce costs an afternoon. For an agent, low parity is a correctness problem. An agent given a low-parity environment produces a change that passes its own tests and fails on merge, and it has no way to know which of its changes will. Four things follow: - **Agents cannot triage a parity failure the way a person can.** A developer who sees a bug in staging that did not appear locally starts a diagnosis from tacit knowledge about how the two environments differ. An agent sees a green test run and opens the pull request, so the environment has to be right the first time. - **Volume thins the human backstop.** When agents write a larger share of the changes, pull request counts rise several-fold, as described in [the staging trap](https://www.signadot.com/blog/scaling-coding-agents-enterprise-kubernetes/), and a reviewer who once noticed that staging differed from a developer’s machine cannot do that for every change. The test environment has to catch what the reviewer used to. - **Reproducing a reported production bug is now often an agent task.** It only works if the agent has an environment shaped like the one the bug happened in: the same dependency versions, configuration, and network path. - **Per-run isolation on real dependencies is the only version of parity that scales to agent volume.** Copying production per run multiplies cost by the number of runs. Sharing a cluster and isolating each run by routing keeps the cost per run at the forked services alone. Agents such as [Claude Code](https://claude.com/claude-code), [Cursor](https://cursor.com/), [Codex](https://openai.com/codex/), and [GitHub Copilot](https://github.com/features/copilot) can drive this loop when the environment is exposed to them through a CLI or an MCP server, which is how Signadot exposes Sandboxes to agents. What the loop looks like against a real Kubernetes cluster is covered in [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). Agent writes changeFork on shared clusterTests againstreal dependenciesResult to agentfixopen PR The agent's loop runs against real dependencies on every iteration, so the first green result is the one that holds on merge. #### Environment parity checklist Answer yes or no for the environment your changes are tested in before merge. The first three questions ask whether drift would be detected. The last three ask whether the environment is real or a copy, and a no on any of them fails the check. 1. Is every deploy to it automated from main, with a check that flags when the running image differs from the target commit? 2. Is every per-environment configuration difference an explicit, reviewed override rather than a hand edit? 3. Does the applied database migration version match production, and is the check automated? 4. Do the services your change calls run the versions currently on main, rather than mocks or pinned images? 5. Do requests traverse the same network path as production, including the mesh and its retry and timeout policies? 6. Can two changes be under test at the same time without either one deploying over the other? #### Where to go next If the problem is the staging environment itself, who owns it, how it is shared, and what its alternatives are, start with the [complete guide to staging environments](https://www.signadot.com/staging-environments/). If the problem is the developer’s machine, the [local development guide](https://www.signadot.com/guide-to-local-development-kubernetes/) compares local clusters, sync tools, remote development, and shared-cluster approaches. If mocked integration tests keep passing while staging fails, [integration tests pass but staging breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) covers that case. Related reading: - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [Local Development on Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-local-development-kubernetes/) - [Integration tests pass with mocks but staging still breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [The Complete Guide to Microservices Testing: From Unit Tests to Production](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [Kubernetes Staging Environments: How to Build, Run, and Share One](https://www.signadot.com/blog/kubernetes-staging-environment/) #### Frequently asked questions What is environment parity? Environment parity is the degree to which development, staging, and production run the same code, configuration, data shapes, and dependency versions, so that a change behaves the same way in each. Full parity is neither achievable nor necessary. The useful target is parity on the properties a given change depends on, with capacity, data volume, and third-party services deliberately scaled down or substituted. What is dev/prod parity in the twelve-factor app model? Dev/prod parity is the tenth of the twelve factors. It says to keep development, staging, and production as similar as possible, and it names three gaps to close: the time gap between writing and deploying code, the personnel gap between the people who write and the people who deploy, and the tools gap between the backing services used in each environment. It calls for the same type and version of each backing service everywhere. Why do bugs only show up in staging or production? A bug appears only in an environment that has the property it depends on. Staging and production have real dependency versions, real data shapes, real network hops with timeouts and retries, and real concurrency across replicas. A laptop has mocks, seed data, same-host calls, and one replica. A bug that depends on any of the first four cannot appear in the second set, however carefully the local environment is maintained. Does a development environment need to mirror production? No. It needs to match production on the properties the change under test depends on, usually dependency versions, network path, configuration, and schema. It should differ on capacity, data volume, secrets, and third-party services, which can be scaled down or substituted without changing behavior. Chasing a full mirror produces environments that are too expensive to give to everyone and too slow to give to anyone. How do you keep development, staging, and production in sync? Deploy every merge to main into staging automatically, so staging is never more than one merge behind. Generate configuration from one source with explicit per-environment overrides and diff the rendered output. Run migrations through the same pipeline in every environment and compare applied versions. For development, stop copying dependencies and test the changed service against the shared environment that tracks main, so the developer's view is in sync by construction. What causes configuration drift between environments? Configuration drift comes from changes made to one environment and not the others: a flag flipped in staging to unblock a test, a resource limit raised in production during an incident, a secret pointed at a vendor sandbox, a kubectl edit that never made it into Git. Each is reasonable on its own. Together they make the environments behave differently for the same code. GitOps controllers surface these as differences from the desired state. Should you copy production data into lower environments? Rarely, and never unmodified. A three-tier strategy works better: synthetic data for most flows, anonymized production snapshots for the stores where data distribution changes behavior, and ephemeral per-change databases for testing destructive migrations. Unmodified production copies carry personal data into environments with weaker access controls, which is a data protection problem before it is a technical decision. How do AI coding agents change environment parity requirements? An agent cannot notice that its environment differs from staging the way a developer does, so a low-parity environment produces changes that pass the agent's tests and fail on merge, with no signal to the agent about which ones. At agent volume, a person cannot catch those failures one at a time. Parity moves from a developer-comfort question to a correctness input, and the environment must be right on the first run. Can a coding agent reproduce a production bug without a copy of production? Yes, if it has an environment that matches production on the properties the bug depends on: the dependency versions, the configuration, the schema, and the network path. A fork of the affected service running against the shared stable versions of everything else, isolated by request routing, gives the agent that environment per run without provisioning a copy. Bugs that depend on production data volume or production request rates remain the exception and need tracing rather than reproduction. #### Related Articles - [ *Image: Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster* Ephemeral Environments ### Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster June 30, 2025 ](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) - [ *Image: Why Signadot Outperforms Other K8s Testing Solutions* ### Why Signadot Outperforms Other K8s Testing Solutions April 30, 2025 ](https://www.signadot.com/articles/why-signadot-outperforms-other-k8s-testing-solutions/) - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Shift-Left Testing for Microservices on Kubernetes Source: https://www.signadot.com/articles/shift-left-testing/ Summary: How to run integration tests against real dependencies at the pull request instead of after a deploy to shared staging: why more unit tests and mocks do not move cross-service failures, the four properties a per-change environment needs, duplicating the stack per pull request versus a lightweight ephemeral environment that deploys only the changed service, the order to add checks to a unit-test-only pipeline, shifting further left into the developer's local loop and a coding agent's iteration loop, what stays at release, adoption problems, DORA-based measurement, and a checklist. ### Shift-Left Testing for Microservices on Kubernetes How to move integration tests from shared staging to the pull request: what the environment behind that gate needs, two ways to build one on Kubernetes, and the order to add checks. Author Arjun Iyer Published September 30, 2026 Updated October 11, 2026 Reading time 22 min Shift-left testing means running a test at the earliest point in the delivery cycle where it can give a trustworthy answer. [Larry Smith introduced the term in 2001](https://web.archive.org/web/20150106145542/http://www.drdobbs.com/shift-left-testing/184404768), arguing that quality assurance should work alongside development instead of testing each build after it is handed over. The underlying economics have held: a defect costs more the later it is found, because more work has been built on top of it and more people are involved in tracing it back. In a microservices system the test that is most out of place is the integration test. Unit tests already run in the editor and on every commit. Contract tests, where they exist, run at the pull request. But the test that checks whether the changed service still works with the services it calls, and with the services that call it, waits for a merge and a deploy to shared staging. It is the one test that finds the bugs peculiar to a distributed system, and it runs after the point where the author has moved on. This guide is about moving that test to the pull request. It covers what the environment behind such a gate has to provide, the two ways to build one on Kubernetes, the order in which to add checks to a pipeline that runs only unit tests today, and what breaks when teams try it. The rest of the testing lifecycle around it is covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### Where microservices tests run today, and why Most teams running microservices on Kubernetes have a pipeline shaped like this: - **On every commit:** linters, static analysis, unit tests, an image build. - **At the pull request:** the same, plus a review. Sometimes contract tests. - **After merge:** a deploy to a shared staging environment, where integration and end-to-end suites run against whatever else is deployed there. - **Before release:** a longer end-to-end pass, sometimes a load test. - **In production:** canaries, feature flags, dashboards. Nothing about that shape is careless. It is the shape that environment availability dictates. Unit tests need no environment, so they run everywhere. Integration tests need the real neighbors, and the only place the real neighbors exist is staging, so that is where they run. The cost is that the feedback loop for the most valuable test is measured in hours or days. By the time the integration suite fails, the pull request has merged, the author has started something else, and the failure may belong to any of the changes that landed in staging since the last green run. In developer terms, integration testing has moved out of the inner loop, where a developer writes code and gets feedback in minutes, and into the outer loop, where feedback arrives after merge. That move has a measurable price. A VP of platform engineering at a company with more than 200 developers told us their engineers each lost about 20 hours a month to post-merge integration failures: switching back to code they had moved on from, working out which merged change broke staging, and running the pull request cycle again. At $100 an hour, that is about $400,000 a month. #### Why shift-left testing is not more unit tests The instinct, when integration bugs keep reaching staging, is to add coverage earlier: more unit tests, better mocks, a Docker Compose file with a few neighbors in it. For a monolith this works, because the interesting behavior is inside the process. For microservices it does not, for three reasons. - **The failures live between services.** A change that renames a field a consumer still reads, alters the shape of an event a downstream service parses, or adds a call that trips a timeout under the mesh’s retry policy is correct in isolation. It is only wrong in the presence of the real neighbor. No amount of testing inside the service can see it. - **Mocks reproduce the author’s assumptions.** A mock of a downstream service behaves the way the person who wrote it believed the service behaves. When that belief is wrong, the code and the mock are wrong together, and the test passes. The [four failure classes mocks cannot catch](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/#why-do-mocked-integration-tests-miss-staging-failures) are the same four that make up most staging incidents. - **A local copy of the stack stops being possible.** Docker Compose and local clusters work until the service count reaches the low tens, at which point the laptop runs out of memory and the compose file falls out of date. The [local development guide](https://www.signadot.com/guide-to-local-development-kubernetes/#why-localhost-stops-working) covers where that line falls and why. So the classic recipe shifts left everything that was already left, and leaves the one test that matters where it was. #### Which tests shift left, and what each one needs Going through the test layers one at a time makes the constraint visible. - **Unit tests** stay where they are. Keep them fast. - **Contract tests** move to the pull request if they are not there already. They need no environment, run in seconds, and catch the most common cross-service break, a changed request or response shape. They also stop at shape: the [contract testing guide](https://www.signadot.com/contract-testing/#where-contract-testing-still-lets-bugs-through) is specific about which failures get through. - **Integration tests against real dependencies** move from staging to the pull request. This is the move that finds the distributed-system bugs, and it needs an environment per change that holds the real current versions of the services the change touches. - **Scoped end-to-end tests** move to the pull request too, for the one or two user paths the change affects. They need the same per-change environment and a way to point a browser or API client at it. - **Full end-to-end, soak, and load tests** stay at release. Whole-system behavior under sustained load is a release question, and trying to answer it per pull request produces a gate nobody waits for. Three of those five need the same thing: an environment for this pull request, with real dependencies in it. That environment is the whole problem. Once it exists, moving the tests is a CI change. Environment per PRpr-123.preview.example.comNew namespaceFull stack copyLightweight ephemeral environmentX-Preview-Key: pr-123Shared clusteronly changed services deployed The two ways to give a pull request an environment with real dependencies: copy the whole stack into a namespace, or deploy only the changed services onto a shared cluster and route test requests to them. Environment per PRpr-123.preview.example.comNew namespaceFull stack copyLightweight ephemeral environmentX-Preview-Key: pr-123Shared clusteronly changed services deployed The two ways to give a pull request an environment with real dependencies: copy the whole stack into a namespace, or deploy only the changed services onto a shared cluster and route test requests to them. #### What the pull request environment has to provide Four properties. A gate missing any one of them gets worked around within a quarter. - **Real dependencies at their current versions.** The services the change calls and the ones that call it, the database at the current schema, the broker with its real topics, the mesh with its real timeout and retry policy, all at the versions on main today. This is what makes the test worth running. The [four kinds of drift](https://www.signadot.com/articles/environment-parity/#the-four-kinds-of-drift-and-how-to-detect-them) describe what goes wrong when any of them is stale. - **Isolation between changes.** Two pull requests under test at once must not see each other’s work. Otherwise the gate inherits the [shared staging problem](https://www.signadot.com/staging-environments/#everyone-deploys-to-it-at-once): a failing run could come from your change, from another team’s, or from two changes that each pass alone but not together. - **Spin-up in seconds.** The whole gate, environment plus tests, has a budget of about ten minutes before developers start batching changes to avoid it. An environment that takes most of that budget to appear leaves no room for the tests. - **Automatic teardown.** Environments that outlive their pull request accumulate, cost money, and drift. The lifetime should be tied to the pull request or to a time-to-live of a few hours. #### Two ways to build a pull request environment on Kubernetes Every product and in-house system in this space takes one of two shapes, and both are ephemeral environments: a full copy of the stack per change, or a lightweight ephemeral environment that deploys only the changed services onto a shared cluster. The [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) compares four ways to build them on Kubernetes in general. For a pull request gate the question is narrower: which one keeps the four properties above at your service count and pull request volume? ##### Duplicate the stack per pull request Each pull request gets its own copy of every service, in a dedicated namespace or virtual cluster, created when the pull request opens and deleted when it merges. - **How it is built.** [Release](https://release.com/) and [Bunnyshell](https://www.bunnyshell.com/) provide this as a product. [Argo CD ApplicationSets](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Pull-Request/) with a pull request generator [build it from GitOps primitives](https://www.signadot.com/articles/argocd-preview-environments/). - **What it gets right.** Fidelity is high because every dependency is present, and isolation is total. There is nothing to propagate and no shared state to reason about. - **Where it strains.** Cost and spin-up time grow with the size of the stack. Thirty services, a database, and a broker take minutes to become healthy and consume a full stack of compute for as long as the pull request is open. Each copy also drifts from main independently. The [ApplicationSet article](https://www.signadot.com/articles/argocd-preview-environments/#where-the-pattern-strains) works through where the model breaks, and below roughly a dozen services it usually does not. ##### Lightweight ephemeral environments on a shared cluster One shared environment tracks main and holds the stable version of every service. A pull request’s lightweight ephemeral environment is only the service it changed, deployed alongside those stable versions. Each test request carries the pull request’s routing key as a header, and at each hop a keyed request goes to the changed version if one exists and to the stable version if not. shared clusterrouting keyGatewayOrdersPaymentsInventoryOrdersforkdatabase A pull request's integration test run: the changed Orders service is deployed beside the stable one, and requests carrying the pull request's routing key reach it while everything else stays shared. shared clusterrouting keyGatewayOrdersOrdersforkPaymentsInventorydatabase A pull request's integration test run: the changed Orders service is deployed beside the stable one, and requests carrying the pull request's routing key reach it while everything else stays shared. - **How it works.** The routing decision is made at each hop by a sidecar or the mesh, based on a header the request carries. The header has to survive every hop, which is what [OpenTelemetry context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) or a service mesh already does for trace ids, and the same mechanism carries the key. Uber built this in-house as [SLATE](https://www.uber.com/blog/simplifying-developer-testing-through-slate/), and DoorDash built a similar system for its developers. [How Uber and DoorDash let developers test in production](https://www.signadot.com/blog/how-uber-and-doordash-enable-developers-to-test-in-production/) compares the two. - **What it gets right.** The environment for a pull request is one or two workloads. It appears in seconds, costs those workloads alone, and is current with main by construction because everything else is the shared stable version. One cluster can hold hundreds of pull requests under test at the same time. - **Where the work is.** Stateful dependencies. A change that migrates a schema cannot share the database with everyone else, and a changed consumer must process only the messages meant for it. The model handles this by isolating only what the change needs. A schema change gets an ephemeral database, and messages carry the routing key so each consumer on the shared topic processes only the messages meant for it. The rest stays shared. The header also has to propagate through every service, so a service that drops unknown headers breaks the chain. Signadot builds the second model as Sandboxes: lightweight ephemeral environments on the cluster you already run. A pull request’s Sandbox deploys the changed service beside its stable version, and the routing key travels through the mesh or through [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/). Writes to a database are kept apart with [data isolation](https://www.signadot.com/docs/guides/set-up-data-isolation) configured per Sandbox. Messages on shared topics reach the right consumer through [message queue isolation](https://www.signadot.com/docs/guides/set-up-message-queue-isolation). In CI, the [GitHub Action](https://github.com/signadot/actions) creates the Sandbox when the pull request opens and removes it when the pull request closes. Tests in that workflow send the Sandbox’s routing key with each request, which is how their traffic reaches the changed service. ##### Which one fits Duplicate the stack Lightweight ephemeral environment What a PR gets A complete copy of every service The changed service, plus the shared stable version of everything else Spin-up Minutes, growing with service count Seconds, independent of service count Cost per open PR A full stack One or two workloads Drift from main Each copy drifts independently None. The shared versions are main Concurrent PRs per cluster Bounded by capacity Hundreds Prerequisite Namespace or vCluster automation, per-PR data seeding Header propagation through every service, per-change isolation for state the change writes Fits Under about a dozen services, or no shared state Large service counts, high pull request volume, coding agents Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### How to implement shift-left testing in a pipeline that only runs unit tests Order matters more than the list. Add each check, make it required, and let it settle before adding the next. 1. **Contract tests.** Consumer-driven contracts for every API the service publishes and consumes, verified on both sides in CI. No environment needed. This step alone removes most shape-change breaks, and the [consumer-driven versus provider-driven](https://www.signadot.com/contract-testing/#consumer-driven-vs-provider-driven-contracts) choice is the only design decision in it. 2. **An environment per pull request.** Wire the environment model you chose into the pull request workflow so that opening a pull request creates the environment and closing it removes it. Measure the time from open to ready. If it is over a minute, fix that before going further, because everything after this step runs inside that budget. 3. **The service’s integration tests against real dependencies.** The tests the team already has, or the ones that used to wait for staging, run in CI against the pull request’s environment. In a lightweight ephemeral environment, every request the tests make carries the pull request’s routing key, so the traffic reaches the changed service and real neighbors. Make the check required. 4. **A scoped end-to-end check.** One Playwright or Cypress flow, or one API sequence, that exercises the change through the system’s real entry point. Not the full suite. Teams that run the entire end-to-end suite at the pull request end up with a thirty-minute gate that everyone learns to bypass. 5. **A performance smoke test, where latency matters.** A short k6 or Locust run against the pull request’s environment with a p95 threshold, so a change that doubles a hot path’s latency fails the check rather than showing up on a dashboard later. On tooling, keep two jobs separate. A test runner executes tests in or against the cluster: [Testkube](https://testkube.io/) does this as a Kubernetes-native runner, and a CI job with kubectl and the framework’s own runner does it too. An environment provider supplies the real dependencies those tests hit. The runner does not give you the environment, and most environments do not run your tests. The [comparison of microservices testing tools for Kubernetes](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) sorts the field into these two layers. Choose the environment model first, because it sets the cost and the spin-up time that decide whether the gate survives. #### Shifting further left: local development and the coding agent’s loop The pull request is not the leftmost point a real-dependency test can run. Once a per-change environment exists, the same mechanism lets a change be exercised against real neighbors before there is a pull request at all, from the developer’s editor or from inside a coding agent’s iteration loop. This is where the feedback loop drops from minutes to seconds. ##### Local development against a shared cluster The pattern is to run only the service being changed as a local process, under a debugger and with hot reload, and connect it into the shared cluster so every dependency it calls is real and current. - The [local development guide](https://www.signadot.com/guide-to-local-development-kubernetes/#option-3-develop-locally-against-a-shared-kubernetes-cluster) compares the tools that do this, Telepresence, mirrord, Gefyra and Signadot, on how they wire the connection and what happens when several developers connect at once. - With Signadot, the [CLI opens an authenticated connection from the laptop into the cluster](https://www.signadot.com/docs/tutorials/quickstart/local-development). In-cluster service names then work locally, and requests carrying the developer’s routing key reach the local process instead of the in-cluster workload. [How Signadot implements the pattern](https://www.signadot.com/guide-to-local-development-kubernetes/#how-signadot-implements-the-pattern) covers the architecture. The [local development](https://www.signadot.com/solutions/local-development/) page covers the developer experience. - Because each developer has a routing key, many people can be connected to the same cluster with their own version of the same service without answering each other’s requests. That collision problem is what makes or breaks shared-cluster development at team scale. - The integration tests that will later run at the pull request gate can run here first, from the developer’s terminal, against the same real dependencies. The gate then confirms what the developer already saw rather than discovering it. ##### Inside the coding agent’s loop, before the pull request A coding agent iterates on a change many times before it opens a pull request, and each iteration is a chance to run against real dependencies instead of the agent’s own mocks. - The [signadot-validate skill](https://www.signadot.com/blog/introducing-signadot-validate-skill/) drives exactly this loop: the agent runs its modified service in its own environment while Postgres, Kafka, Redis and the downstream services come from the cluster, reads the logs and test results, fixes, and reruns against the same environment until the change passes, then leaves the environment up for review. - The same operations are exposed to the agent through the CLI and an MCP server, so Claude Code, Cursor, or Codex can create the environment, run the checks, and iterate without a person in the path. [How the loop is closed](https://www.signadot.com/validate-ai-generated-code-kubernetes/#how-signadot-closes-the-loop) covers what the agent sees at each step. The [coding agent environments](https://www.signadot.com/solutions/agentic-development/) page covers running it for a team of agents. - Two Claude Code sessions given the same task on the same service show the difference. [The one with a real environment](https://www.signadot.com/blog/claude-code-signadot-validate-demo/) caught a cross-service break before declaring the work done. The other shipped a change that looked correct. The pull request gate still matters in both cases. It is the check that does not depend on the developer or the agent having remembered to run the tests. But when the same environment model serves the editor, the agent’s loop, and the gate, most changes arrive at the gate already green, and the gate becomes confirmation rather than discovery. #### Shift left vs shift right: what does not move Not everything moves, and pretending otherwise produces gates nobody waits for. - **Full end-to-end and soak tests** stay at release. Whole-system behavior under sustained load is a release-time question. - **Production validation** stays in production. Canary releases, traffic mirroring, and feature flags catch what only real traffic reveals, and [shadow testing](https://www.signadot.com/blog/shadow-testing-superpowers-four-ways-to-bulletproof-apis/) is how a service is checked against real requests without exposing users to it. - **Exploratory testing** stays with people. Shifting left frees QA from being the first to find integration breaks, which is what makes time for it. Shift left and shift right are not competing philosophies. Shift right handles what cannot be known before release, and shift left keeps that set small enough to handle. #### Why coding agents make shift-left testing mandatory A team that verifies agent-written changes by having a reviewer read them carefully is using the reviewer as the integration test, and that stops working at the volume agents produce. - An agent cannot notice that its change broke a downstream service unless something runs the change against that service. Its unit tests and its mocks share its assumptions. - An agent iterating on a task runs its build-test-fix loop many times before opening a pull request, and each iteration needs the environment. A model where an environment takes fifteen minutes to appear cannot serve that loop. - Dozens of agents testing different changes to the same services on one cluster is the normal case, so isolation per run and attribution of each failure to one change are requirements, not refinements. The result is that the pull request gate stops being a productivity improvement and becomes the step that lets agent output merge at all. Why that is a cloud-native problem first, and what verified has to mean for a distributed change, is the subject of [validating AI-generated code against real Kubernetes dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/). The section above on the agent’s loop covers how the same environment model moves the check earlier still, into the agent’s own iterations. #### Shift-left testing challenges and how to fix them When shift left slows a team down, one of four things is usually true. - **The gate is too slow.** Past about ten minutes, developers batch changes to avoid it, pull requests get larger, and review gets harder. Treat gate duration as a metric and a regression in it as a bug. - **The environment is flaky.** A test that fails for reasons unrelated to the change teaches people to ignore red. Most environment flakiness comes from sharing: another change deployed over yours, data left behind by a previous run, a dependency mid-deploy. Isolation per change removes the first, per-change data isolation removes the second, and a shared environment that deploys automatically from main removes the third. [Changing the test environment](https://www.signadot.com/blog/how-to-fix-flaky-tests/) is usually the fix, not changing the test. - **Nobody owns the shared environment.** With lightweight ephemeral environments, one shared environment tracks main. It needs an owner, a deploy-on-merge pipeline, a health signal, and a data strategy. The staging guide’s section on [running the environment like a product](https://www.signadot.com/staging-environments/#if-you-keep-a-staging-environment-run-it-like-a-product) applies unchanged. - **Cost.** Duplicating the stack per pull request gets expensive as service count and pull request volume grow, and [cost-savvy testing for microservices](https://www.signadot.com/blog/shift-left-on-a-budget-cost-savvy-testing-for-microservices/) works through the arithmetic. A lightweight ephemeral environment costs only the changed workloads, which is what keeps it flat. Done well, shifting left removes more time than it adds. The hours a change spent waiting for a staging slot disappear. The bug found before merge is fixed by its author in minutes, with the context fresh, instead of being triaged by whoever noticed it in staging. And QA moves from finding integration breaks first to [the strategy and tooling work](https://www.signadot.com/blog/why-shift-testing-left-part-2-qa-does-more-after-devs-run-tests/) that no pipeline replaces. #### Measuring shift-left testing with DORA metrics Use the four [DORA metrics](https://dora.dev/guides/dora-metrics-four-keys/) as outcomes and three leading indicators to see movement within a quarter. Outcomes: - **Lead time for changes** should fall, because changes stop waiting for staging. - **Change failure rate** should fall, because seam failures that reached production now fail the pull request. - **Deployment frequency** should rise as a consequence, and **time to restore** should hold or improve because smaller, verified changes are easier to roll back. Leading indicators: - The share of pull requests whose integration tests ran against real dependencies before merge. It should climb toward all of them. - The ratio of defects found before merge to defects found in staging or production. It should invert. - The median time a change waits for an environment. It should approach zero. If change failure rate holds steady, the pull request tests are still hitting stand-ins rather than the dependencies that break in production. If lead time holds steady, changes are still queuing for an environment. Instrumenting the four without gaming them is its own problem, and [how to do DORA metrics right](https://www.signadot.com/blog/how-to-do-dora-metrics-right/) works through it. #### Shift-left testing checklist For the pull request pipeline of the service you ship most often: 1. Do contract tests run for every API the service publishes and consumes? 2. Do the service’s integration tests run before merge against the real current versions of its dependencies? 3. Is the environment for those tests created per pull request, in under a minute? 4. Can two pull requests be under test at once without either seeing the other’s changes? 5. Does the whole gate, environment plus tests, finish in under ten minutes? 6. Is the gate a required check? 7. Does a scoped end-to-end check exercise the paths the change touches, rather than the whole suite? 8. Are environments removed automatically when the pull request closes or after a fixed lifetime? 9. Does the shared environment deploy automatically from main, with an owner? 10. Do you track the share of pull requests that ran integration tests against real dependencies before merge? Fewer than seven yeses means integration testing has not moved, and the bugs that matter are still waiting for staging. #### Related reading - [Staging Environments: The Complete Guide](https://www.signadot.com/staging-environments/) - [The Complete Guide to Microservices Testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) - [What Are Ephemeral Environments? The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [Local Development on Kubernetes: The Complete Guide](https://www.signadot.com/guide-to-local-development-kubernetes/) - [Contract Testing: The Complete Guide](https://www.signadot.com/contract-testing/) - [Integration Tests Pass With Mocks but Staging Still Breaks](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [What Is Environment Parity?](https://www.signadot.com/articles/environment-parity/) - [ArgoCD Preview Environments: The ApplicationSet Pattern](https://www.signadot.com/articles/argocd-preview-environments/) - [Guide to Shift-Left Test Approach in Kubernetes](https://www.signadot.com/blog/shift-left-testing-in-kubernetes/) - [Why Testing Must Shift Left for Microservices](https://www.signadot.com/blog/why-testing-must-shift-left-for-microservices/) - [Shifting Testing Left: Isolating Changes on a Shared Cluster](https://www.signadot.com/blog/shifting-testing-left-the-request-isolation-solution/) - [Introducing the signadot-validate Skill](https://www.signadot.com/blog/introducing-signadot-validate-skill/) - [Validating AI-Generated Code Against Real Kubernetes Dependencies](https://www.signadot.com/validate-ai-generated-code-kubernetes/) #### Frequently asked questions What is shift-left testing? Shift-left testing is the practice of running each test at the earliest stage of the delivery cycle where it can give a reliable result, so the developer who made a change finds out it is wrong before anyone else does. The name refers to the left-to-right timeline that software lifecycles are drawn on. For microservices it mostly means running integration tests at the pull request instead of after a deploy to a shared staging environment. Where does the term shift left come from? Larry Smith introduced shift-left testing in a 2001 Dr. Dobb's Journal article that argued quality assurance should work alongside development from early in a project, rather than testing each build after it is handed over. The economic case is that a defect costs more to fix the later it is found. The specific multipliers in the IBM Systems Sciences Institute figures usually quoted alongside the idea are contested. The direction is not. Does shift-left testing mean writing more unit tests? No. Unit tests already run at the far left, in the editor and on every commit, and a service with high unit coverage can still break the first consumer that calls it. In a microservices system the tests that sit too far right are the integration tests, which wait for a deploy to staging. Moving those left is the change that finds cross-service failures earlier, and it requires an environment with real dependencies per pull request. What is the difference between shift-left and shift-right testing? Shift-left testing finds problems before code merges, with unit, contract, integration, and scoped end-to-end tests at the pull request. Shift-right testing finds problems after release, with canary releases, traffic mirroring, feature flags, and production observability. Both are needed. Shift right catches what only real traffic reveals, and shift left keeps the number of surprises reaching production small enough for shift right to handle. How do you shift integration testing left for microservices? Give each pull request an environment that holds the real current versions of the services the change depends on, run the service's integration tests against it in CI, and make the check required. The environment can be a full copy of the stack, or a lightweight ephemeral environment that deploys only the changed service onto a shared cluster and routes test requests to it. The second spins up in seconds and costs only what changed. Our PR pipeline only runs unit tests. What should we add? In order: contract tests for the APIs the service publishes and consumes, then integration tests that run the changed service against real dependencies in an environment created for the pull request, then a short end-to-end check on the one or two paths the change touches. Keep the whole gate under ten minutes. Contract tests catch shape changes cheaply. Real-dependency integration tests catch the behavior changes contracts cannot describe. Can shift-left testing go further left than the pull request? Yes. With a per-change environment on a shared cluster, a developer can run only the service they are changing as a local process, connected to the real versions of everything it depends on, and run the same integration tests from their terminal before opening a pull request. A coding agent can do the same from inside its iteration loop. The pull request gate then confirms a result the developer or agent has already seen. Does shifting testing left slow development down? It moves time rather than adding it, and usually removes more than it adds. A pull request gate of five to ten minutes replaces a wait for a staging slot that is measured in hours or days, and a bug found before merge is fixed by its author with the context fresh. Where shift left does slow a team down, the cause is nearly always a slow or unreliable environment behind the gate, not the tests themselves. What tools run integration tests against a real Kubernetes environment in CI? Two separate things are needed. A runner executes the tests in or against the cluster: Testkube, a CI job with kubectl and the framework's own runner, or tools such as Playwright and k6. An environment provides the real dependencies the tests hit: Release, Bunnyshell, or Argo CD ApplicationSets for a copy per pull request, or Signadot Sandboxes, which deploy only the changed service onto a shared cluster. Choose the environment model first, because it decides cost and spin-up time. #### Related Articles - [ *Image: Why Signadot Outperforms Other K8s Testing Solutions* ### Why Signadot Outperforms Other K8s Testing Solutions April 30, 2025 ](https://www.signadot.com/articles/why-signadot-outperforms-other-k8s-testing-solutions/) - [ *Image: Best Microservices Testing Tools for Kubernetes in 2026* ### Best Microservices Testing Tools for Kubernetes in 2026 July 17, 2025 ](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) - [ *Image: What Is Environment Parity? How to Make Dev Environments Match Production* Ephemeral Environments ### What Is Environment Parity? How to Make Dev Environments Match Production September 10, 2026 ](https://www.signadot.com/articles/environment-parity/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Local Development on Kubernetes: The Complete Guide Source: https://www.signadot.com/guide-to-local-development-kubernetes/ Summary: The four approaches to Kubernetes local development, local clusters, sync tools, shared remote clusters, and cloud development environments, compared on speed, quality, and cost. ### Local Development on Kubernetes: The Complete Guide Four ways to develop locally when services run on Kubernetes, from local clusters and sync tools to a shared cluster and cloud development environments, weighed on speed, quality and cost. Published July 10, 2026 Reading time 20 min **Local development on Kubernetes** means iterating on a service that will ultimately run in a cluster, with a feedback loop fast enough to keep you in flow and an environment faithful enough that passing locally predicts passing in production. Every way of setting this up is a position on three dimensions: speed, how quickly a change can be tried; quality, how much a passing test actually proves; and cost, what the environment consumes in infrastructure and upkeep. Local development is one stage in the wider practice covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). The shared cluster that third option connects to is usually the staging environment, whose own tradeoffs are covered in the [complete guide to staging environments](https://www.signadot.com/staging-environments/). There are four workable approaches to Kubernetes local development. You can run a scaled-down copy of the system on your machine with a local Kubernetes cluster, automate the rebuild and redeploy loop with a sync tool, run only the service you are changing and connect it to a shared remote cluster, or move the development workspace off the laptop entirely into a cloud development environment. Each approach settles the three dimensions differently, and this guide takes them in order before comparing them directly. 1 · Local clusterlaptopCode: laptopDependencies: laptop2 · Sync toolswatch · build · syncCode: synced to clusterDependencies: cluster3 · Shared clusterlaptopCode: one serviceDependencies: shared cluster4 · Cloud workspaceslaptopCloud workspaceCode: cloud workspaceDependencies: remote The four approaches: where the code runs and where its dependencies run. 1 · Local clusterCode: laptopDependencies: laptop2 · Sync toolsCode: synced to clusterDependencies: cluster3 · Shared clusterCode: one serviceDependencies: shared cluster4 · Cloud workspacesCode: cloud workspaceDependencies: remoteCloud The four approaches: where the code runs and where its dependencies run. #### Why localhost stops working A microservices system outgrows a laptop in predictable stages. Two or three services and a database run fine under [Docker Compose](https://docs.docker.com/compose/). Somewhere between ten and twenty services, the combined footprint of the system, its datastores, its message queues, and its sidecars exceeds what a development machine can hold. Past that point nobody runs the whole system locally, whatever the onboarding wiki claims. The standard response is to mock the dependencies you cannot run. Mocks keep the loop fast, but they encode yesterday’s understanding of a contract. As other teams ship changes, the mocks drift from the behavior of the real services, and the drift surfaces late, in staging or in an incident, where it is most expensive to fix. The [local versus remote environments tradeoff](https://www.signadot.com/blog/local-kubernetes-dev-environments/) has been argued for years, and mock drift is the recurring complaint that drives teams away from pure-local setups. Dependency behaviorreal service, changes each releaseyour mock, frozen at day onerelease 1release 2release 3timealigned on day onedriftthe gap widens with every releaseand surfaces in staging or an incident Time runs left to right, one tick per release of the dependency. The green line is what the real service does, the dashed line is what your mock still does, and the shaded area is drift your local tests cannot see. real servicemock, frozen at day oneDependency behaviorrelease 1release 2timedriftthe gap surfaces in staging Time runs left to right, one tick per release of the dependency. The green line is what the real service does, the dashed line is what your mock still does, and the shaded area is drift your local tests cannot see. Configuration drifts too. In-cluster DNS, service discovery, ConfigMaps, secrets, resource limits, and mesh sidecars have no faithful equivalent in a plain process-on-a-laptop setup. A service that behaves on localhost can fail in the cluster for reasons unrelated to the code change, which trains developers to distrust local results. Mock drift and configuration drift are two of the four ways a laptop loses [environment parity](https://www.signadot.com/articles/environment-parity/) with production, and each is detected and recovered by a different means. The outcome is a familiar failure mode. The inner loop stays on the laptop and becomes less trustworthy, integration testing moves to a shared staging environment and becomes more contended, and the time between writing a change and trusting it grows. Each approach below is an attempt to shorten that distance from a different direction. #### Option 1: Run a local Kubernetes cluster on your machine The most direct answer is to run a real cluster locally. Several mature tools do this well, and they differ mainly in how they trade startup time and resource use against similarity to a production cluster. - [minikube](https://minikube.sigs.k8s.io/) runs a full single-node cluster, optionally multi-node, inside a virtual machine or containers, with an addon system covering ingress, dashboards, and load balancer emulation. - [kind](https://kind.sigs.k8s.io/) runs Kubernetes nodes as Docker containers and was built for testing Kubernetes itself, which makes it fast to create and destroy and a common choice in continuous integration (CI). - [k3s](https://k3s.io/) is a certified lightweight distribution packaged as a single binary, aimed at edge and resource-constrained machines. [k3d](https://k3d.io/) wraps it in Docker containers for the same convenience kind offers. - [Docker Desktop](https://docs.docker.com/desktop/features/kubernetes/) bundles a one-checkbox cluster, provisioned as kind nodes since early 2025, so images built locally are immediately visible to it. The differences between these distributions are covered in more depth in [the guide to lightweight Kubernetes environments](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/), and which tests a local cluster can and cannot run is covered in the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/). Some teams sidestep the cluster entirely and keep local development on Docker Compose, defining the service set as plain containers. This is the fastest option available and the furthest from production. Compose now ships [a bridge](https://docs.docker.com/compose/bridge/) that generates Kubernetes manifests from Compose files, which narrows the format gap but not the behavioral one, since none of the cluster machinery listed above exists in a Compose stack. A local cluster is the right tool in specific situations: - a small number of services that one machine can hold comfortably - platform components like admission controllers, operators, or [Helm](https://helm.sh/) charts, where the cluster itself is the subject of the work - offline development - CI pipelines that need a disposable cluster per run It breaks down on system size and on state. The resource ceiling is absolute, and no local distribution changes the arithmetic of running fifty services on sixteen gigabytes of memory. Local datastores start empty or rely on seed scripts that age poorly. And the local cluster’s manifests, versions, and configuration drift from production unless someone actively maintains parity, a cost that lands on every individual developer. #### Option 2: Automate the inner loop with sync tools The second approach keeps your code running inside a cluster, local or remote, and automates the loop that gets changes there. These tools watch your source tree and rebuild, redeploy, or sync files into running containers on every change. Editorsource treechangeSync toolSkaffold · TiltDevSpacebuild · syncCLUSTER, LOCAL OR REMOTEyour applogs and status back to the developer Sync tools automate the watch, build, deploy cycle so every save lands in the cluster. Editorsource treefile changeSync toolSkaffold · Tilt · DevSpacebuild or live-syncCLUSTERyour applogs and status Sync tools automate the watch, build, deploy cycle so every save lands in the cluster. - [Skaffold](https://skaffold.dev/), from Google, pipelines the watch, build, tag, deploy cycle. Google will archive the Skaffold repository on January 29, 2027, and Cloud Deploy now runs its own fork. See the [Skaffold alternatives](https://www.signadot.com/comparison/skaffold/). - [Tilt](https://tilt.dev/) defines the development environment as code in a Starlark Tiltfile and adds live-update rules that sync files into running containers to skip image rebuilds, plus a web interface showing service health and logs. Docker acquired Tilt in 2022 and continues to maintain it, though feature velocity has been low since. See the [Tilt alternatives](https://www.signadot.com/comparison/tilt/). - [DevSpace](https://www.devspace.sh/), a Cloud Native Computing Foundation (CNCF) sandbox project, deploys your application and then swaps a development container into the cluster with bidirectional file sync and port forwarding, so changes hot-reload in place. - [Okteto](https://www.okteto.com/) builds a development platform around the same loop, provisioning a namespace-isolated copy of the application per developer on your own cluster and syncing local files into a development container inside it. It has repositioned in recent years around provisioning those environments for coding agents as well as humans. Sync tools solve a real problem, which is that the unassisted build, push, redeploy loop on Kubernetes is far too slow for iterative work. Applied to a local cluster they make Option 1 more livable. Applied to a remote cluster they remove the laptop resource ceiling, since your code runs in the cluster rather than on your machine. What they do not answer is where everything else runs, and, for remote clusters, who else is using them. A sync tool deploys your in-progress code into whatever cluster it points at, so that cluster must provide isolation. Either each developer or coding agent gets a dedicated cluster, or each gets a dedicated namespace holding a copy of the system’s services. Point two developers at the same namespace and they collide: one engineer’s half-finished deployment starts answering the other’s requests. Okteto is the clearest expression of the isolated model, one namespaced environment per developer, and it works. But it is isolation by duplication: every developer needs their own copy of the dependencies, which sets up the cost problem the comparison below returns to. #### Option 3: Develop locally against a shared Kubernetes cluster The third approach inverts the problem. Instead of bringing the system to your machine, you run only the service you are changing, as a normal local process under your debugger, and connect it into a shared cluster where every dependency is real and current. Tools in this category differ in how they wire the connection. YOUR LAPTOPcheckout (modified)under your debuggertunnelSHARED CLUSTER · DEPENDENCIES AT CURRENT VERSIONSfrontendorderspaymentsother requestscheckoutstablereal datacalls checkoutto your laptop Your requests are redirected down the tunnel to the modified checkout on your laptop. Everyone else's traffic keeps using the stable checkout in the cluster. SHARED CLUSTERfrontendorderspaymentscheckoutstableother requeststunnelYOUR LAPTOPcheckout (modified) Your requests are redirected down the tunnel to the modified checkout on your laptop. Everyone else's traffic keeps using the stable checkout in the cluster. - [Telepresence](https://telepresence.io/), a CNCF sandbox project, creates a virtual network interface connecting your workstation to the cluster and intercepts traffic destined for an in-cluster deployment through a sidecar agent, redirecting it to your local process either globally or per request header. The open source project remains actively maintained. Its commercial story changed substantially, with Ambassador Labs folding the commercial edition into its Blackbird platform in 2025 before being acquired by Gravitee, a shift that pushed many teams to [reevaluate the Telepresence-style toolchain](https://www.signadot.com/blog/the-ultimate-guide-to-telepresence-alternatives-in-2025/). Operationally, the daemon requires elevated privileges for network configuration, and virtual private network (VPN) software on corporate machines is a recurring source of conflict with its network interface. - [mirrord](https://mirrord.dev/), from MetalBear, takes a process-level approach instead of a network-level one. It injects a layer into your local process that hooks system calls, so the process reads the target pod’s environment variables and files and receives mirrored or stolen traffic through an in-cluster agent, without a VPN-style interface or root access. Its paid plans add an in-cluster operator so multiple engineers can share one target, each receiving only the requests that match their header filter. If you also need an environment for every pull request, see the [mirrord alternative comparison](https://www.signadot.com/comparison/mirrord/). - [Gefyra](https://gefyra.dev/) takes a related approach for containerized local workloads, bridging local containers into the cluster network. - Signadot connects the workstation to the cluster through an authenticated tunnel and applies the same run-one-service-locally pattern, with the difference that the connection is expressed as a sandbox, a declarative unit the platform manages, rather than an ad hoc intercept. How that model handles many concurrent developers is covered in the sharing section below, and the practical differences from Telepresence and mirrord are broken out in the [Telepresence](https://www.signadot.com/comparison/telepresence/) and [mirrord](https://www.signadot.com/comparison/mirrord/) comparison pages. The strengths of this category are exactly what Options 1 and 2 lack. The feedback loop is a local process restart, seconds rather than a rebuild cycle. Dependencies are the real services at their current versions, with real data and real configuration, so mock drift disappears for everything you did not change. Laptop resource use is one service instead of fifty. The costs are organizational more than technical. There must be a shared cluster that reflects production, and someone must own keeping it healthy, typically a platform team. And once more than a handful of developers connect to the same cluster, their changes can collide, one developer’s modified service answering another developer’s requests. Solving that collision problem is what separates the tools in this category at team scale, and it is where this guide ends up in the final section. #### Option 4: Cloud development environments (CDEs) The fourth approach relocates the development workspace itself. Cloud development environments, often shortened to CDEs and also called remote development environments, give each engineer, or each task, a cloud machine holding the source tree, toolchain, and editor backend, accessed from a thin client or the browser. - [GitHub Codespaces](https://github.com/features/codespaces) provisions per-branch cloud machines defined by devcontainer configuration. It is not Kubernetes-aware by itself, so teams either run kind or minikube inside the workspace or point it at a remote cluster, inheriting the tradeoffs of whichever they choose. - [Coder](https://coder.com/) is a self-hosted platform that provisions workspaces on your own infrastructure from Terraform templates, aimed at organizations that want development environments behind their own network boundary, and increasingly at running fleets of coding agents inside them. - [Ona](https://ona.com/), formerly Gitpod, provisions ephemeral cloud development environments defined as code, with prebuilt workspaces that start in seconds, and has expanded toward orchestrating fleets of coding agents in those same environments. CDEs fit teams that value fast onboarding, standardized toolchains, and code never residing on laptops, which matters in regulated environments. They also fit the emerging pattern of running many coding agents, since agents need environments provisioned programmatically and laptops are not where that happens. What a CDE does not do is answer the testing question. It moves the development environment into the cloud, and nothing else: the services your code depends on still have to run somewhere, and that somewhere is still one of the first three options, a local cluster inside the workspace, a sync tool pointed at an isolated namespace, or a connection to a shared cluster. Choosing a CDE changes where you develop, not how you validate, so the test environment decision remains open and just as consequential as before. #### Choosing an approach: cost, quality, and speed Strip away the tooling details and every Kubernetes development environment is a position on three dimensions. **Speed** is the pace of the inner loop: the time from saving a change to seeing it behave, including any waiting on rebuilds, provisioning, or other people. **Quality** is the fidelity of the signal: how well a pass in the development environment predicts a pass in production. **Cost** is what the environment consumes in infrastructure spend and platform upkeep, multiplied by every person and coding agent that needs one. No traditional setup delivers all three. Each of the standard positions sacrifices a different dimension to get the other two. Speedfast feedbackCostcheap to runQualityhigh fidelityLocal environmentsquality lost to mocksDuplicated environmentscost explodes per copyShared stagingspeed lost to queueingMulti-tenantshared clusterspeed, quality, and cost Traditional setups pick two of the three dimensions and sacrifice the third. A multi-tenant shared cluster is the position that holds all three. Speedfast feedbackCostcheap to runQualityhigh fidelityLocal environmentsquality lost to mocksDuplicated environmentscost explodes per copyShared stagingspeed lost to queueingMulti-tenantshared clusterspeed, quality, and cost Traditional setups pick two of the three dimensions and sacrifice the third. A multi-tenant shared cluster is the position that holds all three. **Local environments choose speed and cost.** Everything from Docker Compose to a kind cluster on your machine (Option 1) is free to run and fast to iterate in. Quality is what gets sacrificed: past a handful of services, most dependencies must be mocked or seeded, and the mocks drift from reality as the real system ships. The faster the organization ships, the faster that drift accumulates. Event-driven and data-heavy systems feel it earliest, because brokers like [Kafka](https://kafka.apache.org/) and realistic datastores are exactly what mocks and seed scripts reproduce worst. **Duplicated environments choose speed and quality.** Give every developer a private copy of the system, a namespace each in the sync tool model (Option 2) or a full cluster each, and iteration is fast against real dependencies with nobody contending. Cost is what gets sacrificed, and it scales the wrong way: linearly with the number of environments. That was survivable when one engineer meant one environment. In the agentic era it is not. Each engineer now runs multiple coding agents producing changes in parallel, every agent needs its own environment to validate in, and duplication multiplied by agents produces a number no budget approves. **Shared staging chooses cost and quality.** One live, continuously deployed copy of the system is the cheapest high-fidelity environment an organization can operate, which is why nearly every organization has one. Speed is what gets sacrificed: with no isolation between changes, developers queue to validate their work, contend for the environment, and a single broken deploy blocks everyone behind it. Connecting laptops to a shared cluster (Option 3) inherits exactly this failure mode once more than a handful of developers connect at the same time, unless something keeps their changes from colliding. Cloud development environments (Option 4) do not occupy a corner of this triangle at all. They relocate the workspace and then inherit whichever position they are paired with, so adopting a CDE still means choosing among the three positions above, or the fourth one below. The fourth position is the center of the triangle: share one live environment and make it multi-tenant, so every change is isolated inside it and sharing does not mean colliding. That keeps the shared model’s cost and quality while removing the queue that made it slow. The final section covers how it works. First, the summary: Approach Cost Quality Speed Breaks down when Local environment (local cluster or Compose) Low. Laptop only Low. Mocked dependencies drift from reality High. Everything at hand The system outgrows the laptop and mocks outnumber real services Environment per developer (sync tools, namespaces) High. One system copy per developer or agent High. Real dependencies in every copy High once provisioned. Copies drift between syncs Copy count multiplies with the team, then again with coding agents Shared staging without isolation Low. One environment for everyone High. Real and continuously deployed Low. Queueing and contention Every concurrent change fights for the same environment Multi-tenant shared cluster Low. One environment plus one service instance per change High. Real dependencies at current versions High. Local process restart, no queue Services do not propagate trace headers Cloud development environment Per-seat workspace pricing Set by the paired approach Set by the paired approach The test environment question is still open #### Getting all three: a multi-tenant shared cluster Every path through the trilemma points at the same destination. A shared cluster is already the cheapest environment an organization runs and the highest-fidelity one. The only thing wrong with it is contention, and contention is fixable. The fix is to make the shared environment multi-tenant, isolating each change inside it rather than duplicating infrastructure around it. For the synchronous traffic between services, the isolation comes from request routing. The cluster runs one stable version of every service, continuously deployed. A developer working on a change runs only the modified service, typically as a local process under a debugger, registered as an additional version alongside the stable one. Requests carry a routing key in a header, propagated across service hops through standard context propagation such as [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) or [B3 headers](https://github.com/openzipkin/b3-propagation), and the routing layer sends requests tagged with your key to your modified service while all other traffic flows through the stable versions. SHARED CLUSTER · ONE STABLE VERSION OF EVERY SERVICEAB · stableB · modifiedCkeyno keyrouting key propagated in headers across hops Request routing: one live system, with each change reachable only by requests that carry its key. keyno keySHARED CLUSTERAB · stableB · modifiedCrouting key propagated in headers Request routing: one live system, with each change reachable only by requests that carry its key. Measured against the three dimensions, this is the center of the triangle rather than another compromise: - **Cost stays low.** The whole team, and its coding agents, share one cluster. Each concurrent change adds one service instance, not one system copy, so the marginal cost of the fiftieth simultaneous change is a single process. - **Quality stays high.** Every dependency is the real service at its current version, with real data and real configuration. Nothing is mocked, so nothing drifts. - **Speed stays high.** The inner loop is a local process restart, seconds rather than a rebuild or a redeploy, and there is no queue, because requests isolated by key cannot collide with anyone else’s changes. Two requirements follow from the mechanism. Services must [propagate trace context headers](https://www.signadot.com/docs/guides/set-up-context-propagation), which most teams already have through distributed tracing instrumentation. And something in the data path must make routing decisions per request, either an existing service mesh or dedicated sidecars. ##### How Signadot implements the pattern Signadot packages this model as its core primitive. A [sandbox](https://www.signadot.com/docs/concepts/sandbox) is a lightweight environment defined declaratively, holding the changed version of one or more services while everything else resolves to the stable versions in the shared cluster. For local development specifically, [the CLI connects your workstation to the cluster](https://www.signadot.com/docs/tutorials/quickstart/local-development) over an authenticated tunnel, cluster service names resolve from the laptop as if the process were running inside the cluster, and the sandbox maps an in-cluster workload to the local process, pulling down its environment variables and files so the local service starts with in-cluster configuration. Requests carrying the sandbox’s routing key reach the laptop, and all other traffic never sees it. Control planeauth and SSOYOUR LAPTOPmodified serviceSignadot CLItunnelKUBERNETES CLUSTERSignadot Operatorsvc Asvc Bsvc Csidecarroutingsandboxmaps svc B to the local process The sandbox maps one in-cluster workload to the process on your laptop. Only requests with its routing key are sent down the tunnel. Control planeauth and SSOKUBERNETES CLUSTERSignadot Operatorsidecar routingsvc Asvc Bsvc Csandboxmaps svc B to the local processtunnelYOUR LAPTOPmodified serviceSignadot CLI The sandbox maps one in-cluster workload to the process on your laptop. Only requests with its routing key are sent down the tunnel. [Routing](https://www.signadot.com/docs/guides/request-routing) runs through a built-in sidecar or through an existing [Istio](https://istio.io/) or [Linkerd](https://linkerd.io/) mesh. [Route groups](https://www.signadot.com/docs/reference/route-groups/spec) combine multiple sandboxes under a single routing key, so a change spanning several services, some local and some in-cluster, tests as one unit. Access flows through a central control plane with single sign-on rather than through per-developer cluster credentials, which keeps the shared cluster workable for teams that must keep credentials and data centrally controlled. Request routing is only the mechanism for synchronous traffic, not the whole model. For [message queues like Kafka](https://www.signadot.com/docs/guides/set-up-message-queue-isolation), isolation moves into the messages themselves: routing keys carried in message headers decide whether the stable consumer or a sandboxed one processes each message. For databases, sandboxes take [different approaches](https://www.signadot.com/docs/guides/set-up-data-isolation) depending on the engine and the isolation a change needs, from an ephemeral schema or instance per sandbox to branches on copy-on-write databases like [Neon](https://neon.com/). And because these mechanisms sit on an [extensible plugin framework](https://www.signadot.com/docs/reference/resource-plugins), any system in the stack, including internal ones no vendor ships support for, can be made multi-tenant in the same shared environment rather than duplicated alongside it. The same primitives serve the [local development workflow](https://www.signadot.com/solutions/local-development/) and the pull request validation workflow described in the [ephemeral environments guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), so the environment a developer debugs in and the environment CI validates in are the same kind of thing. Running the same integration tests from the laptop first, and again at the pull request gate, is how [shift-left testing on Kubernetes](https://www.signadot.com/articles/shift-left-testing/) moves feedback earlier. The honest tradeoffs run in the same directions as the rest of Option 3. There is an operator to run and a platform team decision to make, header propagation is a prerequisite, and isolating shared state like databases and message queues requires explicit configuration per resource rather than coming free with the routing. Teams whose systems are small enough to hold on a laptop do not need any of it yet. Develop locally against your real cluster Signadot connects your laptop to a shared Kubernetes cluster so you can run one service under your debugger while every dependency stays real, with sandbox isolation keeping your changes invisible to the rest of the team. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Related reading - [Local development solution overview](https://www.signadot.com/solutions/local-development/) - [Ephemeral Environments: The Complete Kubernetes Guide](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) - [The Ultimate Guide to Lightweight Kubernetes Environments](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) - [Kubernetes Development Environments: Local, Remote, and Production-Like Options](https://www.signadot.com/blog/local-kubernetes-dev-environments/) - [The Ultimate Guide to Telepresence Alternatives in 2026](https://www.signadot.com/blog/the-ultimate-guide-to-telepresence-alternatives-in-2025/) - [Signadot vs Telepresence](https://www.signadot.com/comparison/telepresence/) - [Signadot vs mirrord](https://www.signadot.com/comparison/mirrord/) #### Frequently asked questions What is the best way to develop locally against a shared Kubernetes cluster? Run only the service you are changing on your machine and connect it to the cluster with a tool that routes requests by header, so your changes are isolated from other developers using the same cluster. Telepresence, mirrord, and Signadot all implement variants of this, differing in connection mechanics, concurrency handling, and how much platform tooling surrounds the core workflow. Can I run Kubernetes locally on my laptop? Yes. minikube, kind, k3s, and Docker Desktop all run real clusters on a development machine, and they are the right choice for small systems and for platform-level work like operators and Helm charts. The limit is system size, since a laptop cannot hold a large microservices deployment with its datastores, so most teams outgrow purely local clusters as the service count climbs into the tens. Do I need a service mesh to share a development cluster? No. Header-based routing needs something making per-request decisions in the data path, and an existing Istio or Linkerd mesh can serve that role, but platforms in this space also ship their own lightweight sidecar routing for clusters without a mesh. The harder requirement is context propagation, since services must forward tracing headers for routing keys to survive multi-hop calls. Should every developer get their own namespace instead? Per-developer namespaces work at small scale and are the natural first architecture to try. Their cost grows linearly with team size, each copy takes minutes to provision and drifts from the live system between syncs, and datastores and third-party dependencies usually end up shared anyway. Teams typically move to request-routed sharing of one environment when the namespace fleet's cost or staleness becomes visible. Is Docker Compose good enough for Kubernetes development? For a small number of services it is often the fastest loop available, and that speed is worth real fidelity costs. What Compose cannot reproduce is cluster behavior, including service discovery, ConfigMaps and secrets, resource limits, ingress, and sidecars, so services pass locally and fail in the cluster for environment reasons. Teams usually pair it with a real cluster stage or replace it once those failures become frequent. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Staging Environment Challenges: The Bottleneck and How to Fix It Source: https://www.signadot.com/blog/the-staging-bottleneck-why-your-engineering-team-is-slow-and-how-to-fix-it/ Summary: Why one shared staging environment becomes the bottleneck for scaling teams, how per-PR sandboxes on a staging baseline remove the contention, and how duplicating only what each change needs cuts Kubernetes test environment costs. ### Staging Environment Challenges: The Bottleneck and How to Fix It Why a shared staging environment turns into a queue and a blame game as teams grow, why locking, cloning and mocks fail, and how isolating each change helps developers and AI agents. Author Arjun Iyer Published September 4, 2025 Updated August 31, 2026 Reading time 14 min Topics [Ephemeral Environments](https://www.signadot.com/blog/category/ephemeral-environments/) A few weeks ago, I had a conversation that I can’t stop thinking about. I was talking to a Director of Engineering who leads a 100-person team, and I asked him about his biggest development bottleneck. I expected to hear the usual suspects: CI/CD performance, code review cycles, or maybe deployment complexity. His answer was immediate and emphatic: “It’s the staging environment”. He went on to describe a scenario that is painfully familiar to anyone who has worked in a scaling engineering organization. “We have one staging environment,” he said. “At any given time, 3-4 teams are trying to push their changes in for testing. It’s constantly breaking, and we spend hours trying to figure out whose change caused the issue”. I summed it up: “So it’s a race to merge, followed by a blame game?”. “Exactly,” he confirmed. “Devs get frustrated. QA gets blocked. We either delay releases or ship things with less confidence because we couldn’t get a clean testing window”. This story was so resonant that I shared it on LinkedIn. The reaction was overwhelming, sparking a massive discussion with hundreds of comments from engineers, architects, and leaders across the industry. It was clear this wasn’t an isolated problem; it’s a classic scaling challenge that I’ve seen grind velocity to a halt at countless companies. The traditional model of a single, shared, long-lived staging environment simply breaks down under the pressure of multiple teams moving in parallel. It creates a zero-sum game where one team’s progress often comes at the expense of another’s. The very environment meant to ensure quality becomes the biggest single point of failure and contention. Our [complete guide to staging environments](https://www.signadot.com/staging-environments/) covers what the stage is actually for, the three ways the shared copy fails, and the four alternatives compared. But what if we could fundamentally change the game? #### The usual workarounds, and why they don’t scale High-performing teams deploy code [973 times more often](https://cloud.google.com/resources/state-of-devops) than low performers, so the pressure on a single shared environment is relentless. (If you are still standing up the environment itself, start with our guide to [Kubernetes staging environments](https://www.signadot.com/blog/kubernetes-staging-environment/).) Before teams rethink the model, they usually try to manage that pressure. None of these workarounds holds up as the team grows: - **Locking staging.** Someone posts “locking staging for the next hour, do not touch” in Slack. Fine for a team of three. Past that it just turns testing into a queue, and the next team still has to reconcile against whatever you merged. - **Cloning staging.** Give each team its own copy. Now you are paying to run, and to keep in sync, several near-production environments. Before long you need a “staging 2” just to test how the clones behave together. - **Spinning up more lower environments.** This is like solving a traffic jam by building more highways. It helps for a week, then you have the same contention at a larger scale, plus mocks standing in for third-party services and a synchronization nightmare. - **Leaning on mocks.** Quick to start, but mocks drift from the real services they imitate, and the false confidence they create is its own source of bugs. (More on that in [why developers shouldn’t write mocks](https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing/).) *Image: Spinning up more lower environments is like solving a traffic jam by building more highways* The irony is that each of these, in trying to fix staging, creates a new problem. What teams need is not more environments. It is a smarter way to use the one they have. #### A New Mental Model: From Scarcity to Abundance The old model is built on scarcity, one environment that everyone must share and fight over. The new model is one of abundance. The solution is to give every developer a personal, ephemeral “sandbox” for their feature within the _same_ shared environment. This isn’t about duplicating your entire stack a hundred times over. That would be a cost and complexity nightmare, a valid concern many people raised in the discussion. Instead, the modern approach is far more elegant and efficient. Through [smart request routing](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), we can isolate each developer’s changes from everyone else’s, even though they share the same underlying infrastructure. Imagine Developer A is working on a feature. They can deploy their pull request into a personal sandbox, a [preview environment](https://www.signadot.com/solutions/preview-environments/) that exists just for that change. When they send a test request to the staging cluster with a specific header, the request is intelligently routed to their modified service. For all other dependencies, the request is routed to the stable, baseline services running in the main staging environment. This means Dev A can test their PR without ever being affected by Dev B’s work, and vice-versa. The analogy to source control is almost perfect. The main staging environment is your trunk or main branch, the stable baseline. Each developer’s sandbox is like a feature branch, isolated and independent until you’re ready to merge. Our [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) walks through this model end to end. The results we’re seeing from this shift are profound: - **Testing Contention:** Eliminated. - **Environment-Related Release Delays:** Down by over 90%. - **Time Spent Debugging “Who Broke Staging”:** Near zero. The Director’s follow-up question perfectly captured the promise of this new model: “You mean my teams can test in parallel without stepping on each other’s toes?”. Yes. That’s the power of modern development infrastructure. #### Answering the Hard Questions from the Community The vibrant online discussion that followed my post surfaced a number of valid and challenging questions. For this approach to be viable, it has to hold up to real-world scrutiny. Let’s tackle the most common discussion topics head-on. ##### Discussion 1: “This only works if you don’t have shared resources like a database.” This was, by far, the most critical counterpoint. One commenter articulated the problem perfectly: even with temporary environments, you cannot spin up a temporary database with large test datasets for every developer, so “all these environments end up sharing the same database”. This becomes a huge issue, “especially when a developer needs to make schema changes that are not backward compatible”. This is absolutely correct. The data layer is a crucial piece of the puzzle. The solution requires a multi-faceted approach: 1. **For Most Cases, Share the Database:** For the majority of changes that don’t involve schema modifications, sandboxes can share the main staging database. This works well as long as teams follow best practices, such as not mutating data they didn’t create. The rich, representative data in a shared staging database is incredibly valuable for testing. 2. **For Schema Changes, Isolate the Database:** For cases where full data isolation is required, you must spin up a temporary database with the sandbox. This could be a containerized DB, a separate schema, or a branch if you’re using a modern database that supports it. This temporary DB can be seeded with a snapshot of production data to ensure realistic testing. 3. **Automate the Cleanup:** A valid concern was the cost and complexity of managing these temporary resources, with developers potentially leaving them running after a merge. The key is automation. These temporary resources must be linked to the lifecycle of the sandbox itself. When a sandbox is deleted (either automatically on PR merge, via a TTL, or manually), the associated temporary database is automatically cleaned up with it. ##### Discussion 2: “This sounds expensive and complex to set up.” Many commenters worried about the infrastructure costs and setup complexity of ephemeral environments. One was blunt, suggesting “it increases costs a gazillion times”. This is a misconception rooted in the traditional approach to ephemeral environments, where you duplicate the entire stack. The sandbox model I’m proposing is fundamentally different and far more cost-efficient. You are not cloning 15+ services for every developer. You are only spinning up pods for the 1-2 services that are actually changing in a given PR. The rest of the stack is shared. The cost of running a few extra pods in your Kubernetes cluster is minimal compared to the staggering cost of developer downtime, blocked QA teams, delayed releases, and hours spent in “blame game” war rooms. The long-term savings in developer productivity far outweigh the infrastructure costs. As for complexity, building this from scratch is indeed a significant undertaking. You’d need to manage shadow deployments, integrate with a service mesh for routing, handle header propagation via OpenTelemetry, and build a system for spinning up temporary resources. This is precisely why we built Signadot, to provide this capability as a managed platform so teams can focus on their own products. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Discussion 3: “Doesn’t this just kick integration problems down the road to production?” Several people argued that by testing in isolation, you miss the very integration issues that staging is meant to catch. They suggested this might “shift the integration pain to the right” and that “If the changes are still separated, you don’t know if they work _together_ when you push them to production”. This is a critical point, and it highlights a misunderstanding. This approach does **not** eliminate the staging environment or the need for integration testing. It supercharges it. The goal is to “shift left”, to find and fix as many bugs as possible _before_ code is merged into the main branch. By performing integration testing against a stable baseline in a pre-merge sandbox, you gain a high degree of confidence. Post-merge testing on the main staging environment becomes much shorter and smoother because you’ve already ironed out the majority of issues. Furthermore, what if a single feature requires changes across multiple PRs from different teams? Sandboxes can be grouped together, allowing you to test a collection of interdependent PRs as a single, cohesive feature before any of them are merged. You’re still performing integration testing, but you’re doing it earlier, faster, and in a controlled, isolated manner. ##### Discussion 4: “Why not just fix the process? Use Trunk-Based Development, better CI, or run everything locally.” A number of comments suggested this was a process or architecture problem, not an environment problem. Suggestions included Trunk-Based Development (TBD), better CI pipelines, or simply having developers run everything on their local machines. While all these practices have value, they don’t fully solve this specific problem at scale: - **Local Development:** Running a complex, distributed system with 20+ microservices, databases, message queues, and cloud resources locally is often impossible. As your system grows, local development can’t replicate the production environment’s complexity, leading to the dreaded “it works on my machine” scenario. - **Trunk-Based Development:** TBD is a fantastic practice, but it’s orthogonal to the issue of testing. You still need a way to validate changes before they land on the trunk, especially when those changes have complex interactions with other services. This sandbox approach provides a powerful way to do pre-merge validation, which perfectly complements TBD. - **Fixing the Architecture:** While better architecture and loose coupling are always good goals, even well-architected systems have dependencies. The need to test how services interact doesn’t disappear. The bottleneck isn’t just about code; it’s about providing a realistic, scalable environment for integration testing. #### How Uber, Lyft, and DoorDash actually solved it This is not theoretical. The companies that hit microservices scale first all landed on the same answer: isolate the change, not the whole environment. - **Uber** built SLATE (Short-Lived Application Testing Environment), which lets developers spin up on-demand environments that talk to production instances of their dependencies instead of standing up a full stack each time. - **Lyft** built a control plane for its shared development environment. As its engineering team put it, “instead of providing fully isolated environments, we isolated requests within a shared environment.” That one sentence is the whole idea. - **DoorDash** built a fast feedback loop that lets developers test Kubernetes changes against a realistic environment without waiting on a shared one. The instinct to “just run everything locally” does not scale either. As distributed-systems engineer Cindy Sridharan [observed](https://copyconstruct.medium.com/testing-microservices-the-sane-way-9bb31d158c16), “trying to spin up the full stack on developer laptops is fundamentally the wrong mindset to begin with.” Modern systems are too interconnected: a failure in one service, say payments, often surfaces as a bug in another, like orders, which is what makes interdependencies the hard part of testing. *Image: An e-commerce microservices architecture, where payment, orders, shipping, and other services all depend on each other* The payoff is measurable. AWS [documented](https://aws.amazon.com/blogs/enterprise-strategy/business-value-of-developer-experience-improvements-amazons-15-9-breakthrough/) a 15.9% year-over-year cost reduction and 433% three-year ROI from developer-experience improvements, and Gartner [expects](https://devops.com/platform-engineering-the-2024-game-changer-in-tech/) 80% of large engineering organizations to run platform-engineering teams by 2027, up from 45% in 2022. What once took a dedicated platform team at Uber or Lyft to build is now available off the shelf, so teams of any size can isolate each change on a shared cluster today. #### The staging bottleneck just got bigger: now AI is writing the code Since I first wrote this, the ground has shifted. A large and growing share of code is now written by AI coding agents like Claude Code, Codex, and Cursor. A single engineer can open more pull requests in a day than a small team used to open in a week. The part we spent a decade trying to speed up, writing the code, is no longer the thing holding us back. The constraint has moved downstream, to validation. When an agent produces ten changes in the time it used to take to hand-write one, every one of those changes still has to be proven correct against the rest of the system. For a cloud-native application that means running it in a realistic environment with its real dependencies, and that is exactly where the old staging model buckles. A laptop cannot hold the full stack, so local testing stays low fidelity. Shared staging can hold it, but now it has even more changes queued for the same single lane. The faster agents write, the longer the line for staging gets. This is why the sandbox model matters more now than when I first described it. Agents do their best work when they can close the loop themselves: make a change, test it against real dependencies, read the result, and try again, without waiting on a person or a shared environment to free up. A per-pull-request sandbox gives them exactly that, an environment that is quick and cheap to create but still high fidelity because it runs against the real baseline. The same shared-cluster model that lets a hundred developers test in parallel, each duplicating only what they changed, lets a fleet of agents work at once without stepping on each other. An environment on its own is only half the answer. To let agent-written code merge with confidence, you need a validation layer sitting on top of it. At Signadot that layer is Sandboxes for the environment, plus [SmartTests](https://www.signadot.com/docs/concepts/smart-tests-and-jobs), [Jobs](https://www.signadot.com/docs/reference/jobs), and [Plans](https://www.signadot.com/docs/reference/plans) for the checks that run inside it. SmartTests catch behavioral regressions by comparing the baseline against the change, Jobs run the test suites you already trust against the sandbox, and Plans tie them into one repeatable workflow an agent can run and act on. Together they take a change all the way from generated to verified. I wrote up the longer version of this argument in [validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/). None of this replaces the staging environment or the integration testing it stands for. It does what shifting left always promised: it moves validation earlier, makes it cheap enough to run on every change, and keeps it high fidelity. For where sandboxes fit in the full lifecycle, see our complete guide to [microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). That was worth doing when humans wrote the code. Now that agents write most of it, it is the difference between shipping faster and drowning in unverified pull requests. #### The Way Forward: Platform Thinking and Empowered Teams Ultimately, solving the staging bottleneck is about more than just tools; it’s a cultural shift. It’s about moving from a rigid, sequential process to a flexible, parallel one. It requires adopting a platform engineering mindset, where the platform team provides self-service tools that empower feature teams to move faster. Instead of a centralized gatekeeper (the staging environment), you create an ecosystem where every developer and every PR gets an isolated slice of the testing workflow, on-demand. The blame game disappears, confidence goes up, and teams can finally realize the core promise of microservices: the ability to develop, test, and ship smaller code changes to production independently and rapidly. The journey from a congested, single-lane road to a multi-lane superhighway for development is challenging, but the destination, faster releases, higher quality, and happier developers, is more than worth the effort. How does your team manage the staging environment bottleneck? I’m keen to hear your strategies. #### Frequently asked questions How do teams share one staging environment without blocking each other? By giving each change its own sandbox on the shared baseline. The sandbox deploys only the services the change modifies, requests carrying that sandbox's routing key are steered to the new versions, and everything else uses the baseline. Teams and coding agents test in parallel on the same cluster without queueing or stepping on each other's changes. Why does staging break so often in a microservices organization? Because dozens of teams deploy half-finished changes into one long-lived environment. Any single bad deploy blocks everyone, failures are hard to attribute across overlapping changes, and the environment drifts from production over time. The fix is to keep the baseline continuously deployed from the main branch and move per-change testing into isolated sandboxes instead of deploying unmerged code into the shared environment. How do you make test environments match production more closely? Stop cloning environments and test on a continuously deployed shared baseline instead. A baseline that tracks the main branch with production-like configuration stays current by construction, and sandboxes on top of it give each change isolation without drift. Cloned environments, by contrast, start diverging from the moment they are created. How do you reduce the cost of Kubernetes test environments? Replace per-team or per-PR copies of the stack with sandboxes on one shared cluster, so each test environment deploys only the services that changed. Teams that make this switch report 85 to 90 percent lower test-infrastructure cost, and Brex saves about $2 million annually after replacing duplicated environments with sandboxes. #### Related Posts - [ *Image: How Uber & DoorDash empower developers to preview every code change in production* Ephemeral EnvironmentsCase Studies ### How Uber & DoorDash empower developers to preview every code change in production June 23, 2023 ](https://www.signadot.com/blog/how-uber-and-doordash-enable-developers-to-test-in-production/) - [ *Image: The Pitfalls of Using Feature Flagging for Testing in Shared Staging Environments* Ephemeral Environments ### The Pitfalls of Using Feature Flagging for Testing in Shared Staging Environments July 9, 2023 ](https://www.signadot.com/blog/pitfalls-of-using-feature-flagging-for-testing-in-shared-staging-environments/) - [ *Image: Microservices Testing: 4 Use Cases for Sandbox Environments* Ephemeral Environments ### Microservices Testing: 4 Use Cases for Sandbox Environments March 28, 2025 ](https://www.signadot.com/blog/microservices-testing-4-use-cases-for-sandbox-environments/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Kubernetes Staging Environments: How to Build, Run, and Share One Source: https://www.signadot.com/blog/kubernetes-staging-environment/ Summary: An opinionated guide to staging on Kubernetes: why most companies need exactly one pre-production environment, a reference architecture and data strategy, staging SLOs, and how test tenants share one environment safely across services, message queues, and databases. ### Kubernetes Staging Environments: How to Build, Run, and Share One An opinionated guide to Kubernetes staging: why one pre-production environment is enough, how to run it like a product with SLOs, and how test tenants let teams and agents share it. Author Arjun Iyer Published July 17, 2026 Updated July 18, 2026 Reading time 9 min Topics [Ephemeral Environments](https://www.signadot.com/blog/category/ephemeral-environments/) **A Kubernetes staging environment** is a pre-production cluster, or a dedicated slice of one, that mirrors production closely enough that a change passing there is expected to be safe to ship. It is where integration bugs are supposed to die. This guide is the operations manual: how to build staging, how to run it like a product with SLOs, what your data strategy should be, and how to share one environment across every team and coding agent without the usual queueing and breakage. Staging is one stage in the wider practice covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### What staging is for Staging exists to answer one question with high confidence: will this change work in production? Unit tests and mocks cannot answer it, because in a microservices system most failures live in the interactions, the real datastores, the real message queues, and the real configuration. A good staging environment is the first place a change meets all of those at once. For the wider picture of where staging sits among dev, QA, UAT, and pre-production, and why the single shared copy stops working at microservices scale, see the [complete guide to staging environments](https://www.signadot.com/staging-environments/). That gives you the design principle for everything below: staging earns its keep exactly to the degree that it resembles production and stays current. Every decision that follows, cluster layout, data, sync strategy, sharing model, is in service of that fidelity. #### How many environments do you need before production? This is the question we hear most often, so here is the opinionated answer: for most companies, one. The traditional ladder of dev, QA, staging, and preprod exists mostly as a workaround for unsafe sharing. Each rung was added because the previous environment got too crowded or too broken to trust, and each rung costs real money, drifts from production on its own schedule, and adds a hop between a developer and the truth. Teams do not actually want four environments; they want four moments of isolation, and cloning infrastructure was historically the only way to get them. It no longer is. If one staging baseline can be shared safely, with every change getting its own isolated slice (more on how below), the intermediate rungs stop paying rent. One environment to keep in sync means one environment that is actually in sync, and the money saved on the ladder funds making that single environment excellent. The honest exceptions are specialized environments with a narrow job: sustained load and performance testing that would disturb functional testing, compliance or certification environments that regulators require to be frozen, and the occasional hardware-specific rig. Keep those when they earn their place. What you should not keep is a general-purpose dev or QA clone whose only justification is that staging is too contested to use. #### A reference architecture for staging on Kubernetes Run staging in its own cluster rather than a namespace inside production, so a bad deploy, a runaway job, or a misconfigured operator can never reach production. From there, the architecture question is what to copy exactly and what to shrink. The rule: scale down capacity, never fidelity. Safe to scale down: - **Replica counts.** Two replicas prove the same things twenty do, including that the service survives a pod restart. - **Node size and pool count.** Test traffic is a trickle next to production load. - **Off-hours capacity.** Autoscale toward zero on nights and weekends; non-production environments are a large share of most cloud bills, and idle staging capacity is the easiest cut in the budget. Never cut: - **The Kubernetes version and add-ons.** Same minor version, same service mesh, same ingress controller, same operators. Version skew here is a class of production surprise you paid for staging to catch. - **The configuration shape.** Same ConfigMap and secret layout, same resource limit patterns, same network policies. A change that breaks on config should break in staging. - **The dependency graph.** Real datastores, real message queues, real third-party integrations in test mode. The moment a dependency becomes a mock, deploy-time bugs get a place to hide. #### The promotion pipeline: staging tracks main Deploy to staging through the same GitOps or CI/CD path that deploys to production, promoting the same artifacts, automatically on every merge. Hand-applied manifests are how configuration drift starts, and [drift is what makes staging results untrustworthy](https://www.signadot.com/blog/the-staging-bottleneck-why-your-engineering-team-is-slow-and-how-to-fix-it/). Automatic sync matters more than teams expect. A staging environment that is three days behind main is testing a system that no longer exists, and every test that passes against it is a small lie. Measure the gap explicitly: sync lag, counted in merges behind main, should hover near zero. #### A data strategy in three tiers Most deploy-time surprises come from data and dependencies, not code, so an empty database is a fidelity hole. In practice you need three tiers, applied per datastore: 1. **Synthetic seed data** for the baseline: generated fixtures that match production’s schema and shape. Cheap, safe, and good enough for most functional testing. 2. **Anonymized production snapshots** on a refresh schedule for the datastores where distribution matters: the search index whose relevance depends on real cardinality, the orders table whose edge cases only production has generated. Anonymize on export, refresh on a cadence you alert on, because stale snapshots drift like everything else. 3. **Ephemeral per-change databases** for destructive work: schema migrations and data rewrites get a temporary database or schema provisioned for the change and destroyed after, so the shared baseline’s data stays trustworthy. Tier three is also where sharing gets interesting, which is the next-but-one section. #### Give staging an SLO Here is the reframe that separates teams whose staging works from teams whose staging is a running joke: treat staging as a product with an SLO, not a shared resource with a Slack channel. Three numbers cover it: - **Baseline-green rate**: the percentage of time the staging baseline passes its smoke checks. If this is not alerted on, staging breakage becomes something developers discover, one wasted afternoon at a time. - **Sync lag**: merges behind main, as above. Target near zero; alert past a handful. - **Time-to-test**: how long a developer or coding agent waits between wanting to validate a change and having an environment for it. This is your earliest bottleneck signal, and when it starts stretching, the sharing model, not the cluster size, is usually what needs to change. Put the three on a dashboard, page someone when baseline-green drops, and staging stops being a joke. None of this is heavy: the smoke checks already exist in most CI setups, and the sync lag is a one-liner against the Git history. #### Sharing staging: the test tenant model Now the part the rest of this guide has been building toward. One excellent staging environment is only viable if many teams, and now fleets of coding agents, can use it at once without corrupting it for each other. Locks, calendars, and clones all fail at this in familiar ways: locks become queues, calendars make testing a scarce resource, and clones resurrect the environment ladder you just tore down. (Our [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) compares these sharing models in depth.) The model that works is the **test tenant**: each change in flight gets a [logical tenant on the shared baseline](https://www.signadot.com/solutions/kubernetes-test-environments/), identified by a routing key. The tenant is not a copy of anything. It is a thin overlay, only the services the change touches get deployed as sandboxed workloads, and the routing key makes the overlay behave like a private environment across every kind of interaction: - **Synchronous services** use request routing. The routing key travels in request headers via context propagation, and the mesh or a lightweight proxy steers any request carrying the key to the sandboxed version of a changed service. Everything else flows to the baseline. - **Asynchronous flows** use selective consumption, because a message queue has no request path to route. Messages produced under a tenant carry the routing key in their metadata, and sandboxed consumers process only their tenant’s messages while baseline consumers skip them. The same pattern covers [Kafka, SQS, and other brokers](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/) and [Google Pub/Sub](https://www.signadot.com/docs/tutorials/testing-google-pub-sub-flows). - **Databases and stateful dependencies** are isolated only as far as a tenant needs. Most tenants read and write the shared baseline datastores, which is exactly the fidelity you want. A tenant doing destructive work gets its own ephemeral database or schema, provisioned through [resource plugins](https://www.signadot.com/docs/concepts/resources) for the tenant’s lifetime. One concept, applied to every interaction type. That is what makes it powerful: the tenant boundary is logical, so its marginal cost is near zero and hundreds can coexist on one cluster, yet every tenant tests against the real baseline, so fidelity is not traded away for isolation. Isolation stops being something you buy with infrastructure and becomes something the platform grants per change. This is the model behind a [Kubernetes sandbox](https://www.signadot.com/kubernetes-sandbox/), and our guide to [ephemeral environments in Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers it as duplicating only what changed. Nothing about it removes staging. The baseline is still there and still synced. What disappears is the contention over it, which matters twice as much now that coding agents multiply the number of changes needing a production-like environment at the same time. #### Best practices checklist - Run staging in its own cluster, matched to production’s version, add-ons, and config shape. - Scale down replicas, node sizes, and off-hours capacity; never scale down fidelity. - Promote the same artifacts through the same pipeline as production, on every merge. - Seed synthetic data everywhere, refresh anonymized snapshots where distribution matters, and give destructive changes ephemeral databases. - Set staging SLOs: baseline-green rate, sync lag, and time-to-test, with alerts. - Share the environment through test tenants, not locks, calendars, or clones. - Keep specialized environments only where they earn their place, like performance and compliance. One staging environment, a tenant for every change Signadot turns the staging cluster you already run into a multi-tenant platform: every developer and coding agent gets an isolated sandbox across services, message queues, and databases. The free tier is open to every developer. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Frequently asked questions What is a Kubernetes staging environment? A Kubernetes staging environment is a pre-production cluster, or a dedicated slice of one, that mirrors production closely enough that a change passing there is expected to be safe to deploy. It typically runs the same services, configuration, and infrastructure primitives as production, with test or anonymized data. How many environments do you need before production? For most companies, one. A single well-run staging environment, shared safely through per-change test tenants, replaces the traditional dev, QA, staging, and preprod ladder, which mostly exists to work around unsafe sharing. Specialized environments can still earn their place for narrow jobs like sustained performance testing or compliance certification. How do I share a staging Kubernetes environment across teams? Give each change a test tenant on the shared baseline: a routing key that follows the change through synchronous calls via header-based routing, through message queues via selective consumption, and through data via ephemeral per-tenant databases or schemas. Teams and coding agents then test in parallel without redeploying or breaking the shared environment. How big should a staging cluster be? Match production's Kubernetes version, add-ons, and configuration shape exactly, then scale down what does not affect fidelity: replica counts, node sizes, and node pool counts. Test traffic is a trickle compared to production load, so staging typically runs well at a fraction of production capacity. How often should staging sync with the main branch? On every merge, automatically, through the same delivery pipeline as production. A useful way to track it is sync lag measured in merges behind main; if staging is more than a handful of merges behind, test results there describe a system that no longer exists. What is the difference between staging and preprod? In most organizations they are the same idea: a production-like environment that sits after CI and before production. Some teams run both, using staging for integration testing and a stricter preprod for release rehearsal, but any long-lived shared environment becomes a queue as team count grows, which is why fewer, better-shared environments win. #### Related Posts - [ *Image: Smart Ephemeral Environments: Share More, Copy Less* Ephemeral Environments ### Smart Ephemeral Environments: Share More, Copy Less January 31, 2025 ](https://www.signadot.com/blog/smart-ephemeral-environments-share-more-copy-less/) - [ *Image: How to Improve the Microservice Lifecycle with Developer Environments* Ephemeral Environments ### How to Improve the Microservice Lifecycle with Developer Environments June 2, 2022 ](https://www.signadot.com/blog/improving-software-development-lifecycle-with-microservice-sandbox-environments/) - [ *Image: How Bitso Is Scaling Branch-Based Development with Signadot and Neon* Case StudiesEphemeral Environments ### How Bitso Is Scaling Branch-Based Development with Signadot and Neon January 7, 2026 ](https://www.signadot.com/blog/how-bitso-is-scaling-branch-based-development-with-signadot-and-neon/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments Source: https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/ Summary: How namespaces compare to separate clusters and vClusters as test environment boundaries, where namespace-based environments break down, and what teams move to. ### Kubernetes Namespace vs Cluster: Pros and Cons for Test Environments Namespaces, separate clusters and vCluster compared as Kubernetes test environments: where namespaces work, where they break as teams grow, and why teams move to routing on a shared baseline. Author Nica Mellifera Published July 5, 2023 Updated July 17, 2026 Reading time 10 min Topics [Ephemeral Environments](https://www.signadot.com/blog/category/ephemeral-environments/) **Namespace-based environments** give each team or developer a logical slice of a shared Kubernetes cluster to test in. They are cheap, they are built into Kubernetes itself, and a small team can be up and running with them in an afternoon. They are one of four ways to replace a single shared environment, compared in the [complete guide to staging environments](https://www.signadot.com/staging-environments/), and the [complete guide to Kubernetes testing](https://www.signadot.com/kubernetes-testing/) sets them beside local clusters, virtual clusters and full copies. The trouble arrives later, when the number of services and teams grows and keeping every namespace in sync quietly becomes a full-time job. This is an honest look at what namespaces do well as [test environments](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/), where they break down, and what teams adopt when they outgrow them. Pros Cons Simple to set up, core to Kubernetes Each namespace needs its own datastores and seed data Cheap logical isolation within one cluster No built-in way to keep namespaces in sync Per-namespace RBAC and resource quotas Spin-up time grows with app complexity Familiar kubectl workflow Often unmaintained and unreliable at scale #### What a namespace actually gives you A namespace is a logical partition inside a single Kubernetes cluster. It groups a set of resources, pods, services, volumes, and so on, so they can be managed and access-controlled as a unit while still sharing the cluster with everything else. Creating one is about as easy as anything gets in Kubernetes: define the namespace in a YAML file, run `kubectl apply -f namespace.yaml`, and deploy your application into it with the `--namespace` flag. Every kubectl command can be pointed at your namespace, or default to it, so you can experiment freely without worrying about wrecking another team’s services. You get per-namespace RBAC and resource quotas, and because namespaces are core to the Kubernetes spec, you are not depending on obscure add-ons. Namespaces are not fully sealed off from each other either, which is handy for testing: every service gets a DNS entry of the form `..svc.cluster.local`, so if you know a service’s name and namespace, you can reach it from anywhere in the cluster. Namespaces have plenty of uses, but this piece is about one in particular: functional testing, where each engineer gets a production-like environment to try their code in before it merges. *Image: Diagram of four developers each deploying updated services into their own namespace copy of the full environment* _Four developers, four namespaces, four copies of the environment to keep up to date._ #### Namespace vs cluster vs vCluster Namespaces are one of three isolation boundaries teams reach for, so it is worth placing them side by side. A namespace is a logical partition inside one cluster: cheap, simple, and isolated at the application level. A separate cluster is the opposite extreme, a fully independent control plane and data plane, with the strongest isolation and the highest cost. A virtual cluster (vCluster) sits in between: a virtual control plane running inside a host cluster, which gives each tenant cluster-level isolation without another physical cluster to pay for. Namespace vCluster Separate cluster Isolation level Application / resource Control plane (virtual) Full (physical) Relative cost Lowest Low to medium Highest Blast radius Shared control plane Isolated control plane Fully isolated Setup and management Simplest Moderate Most complex Best for Lightweight separation in one cluster Strong isolation without new clusters Hard multi-tenancy or compliance In practice, namespaces and vClusters solve similar problems. Both partition a shared cluster, both support quotas and role-based access control, and both let teams make changes independently. The difference is depth: a namespace shares one control plane with every other tenant, while a vCluster gives each tenant its own API server, operators, and CRDs, at the price of more moving parts. For PR environments, the [vCluster alternatives](https://www.signadot.com/comparison/vcluster/) page compares a virtual cluster per pull request with environments that copy only what changed. A separate physical cluster remains the strongest boundary, and the right one for hard multi-tenancy or compliance, but it is the most expensive and the slowest to provision. Here is the thing, though: for testing, the choice of boundary matters less than you would hope. Whichever unit you pick, namespace, vCluster, or cluster, a test environment still needs its own copies of services, datastores, and seed data, and something has to keep all of it in sync. That problem is the real subject of this post, and it is why many teams eventually skip stronger partitioning altogether in favor of [sandboxes](https://www.signadot.com/kubernetes-sandbox/) on a shared cluster that duplicate only what changed. More on that at the end. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Where namespaces start to break down First, remember what the namespace is for. The point of handing a team its own namespace is more accurate testing: spin up the services and resources you need, run your changes against them, and trust the result. Some teams deploy everything into the namespace; others deploy a subset and mock the rest to save money. That accuracy is exactly what erodes first. Say you create namespaces for two large teams, team-jacob and team-edward. There is no built-in way to keep those namespaces in sync. Changes to datastores, changes to third-party APIs, even routine updates to services neither team is touching, all of it has to be propagated by hand. And while running many namespaces is certainly simpler than running many clusters, it is not much cheaper: the storage and compute for all those duplicated services cost nearly the same wherever the boundary sits. I got a vivid picture of how this plays out from an enterprise operations team that had used namespaces heavily for several years. Early on they were delighted: every team had free rein in its own namespace, and testing accuracy went up. Then the app grew. Spinning up all the services in a namespace took longer and longer, and the needs of different teams caused their namespaces to diverge. There was no simple way for a developer to tell whether their namespace was up to date, so a passing test stopped meaning much about what would happen on staging. Ownership made it worse. Product teams are clearly delineated; operations teams usually are not. Individual namespaces had no owner on the platform side, so when one went stale, nobody was answerable for it. The end state was a collection of half-maintained namespaces that nobody trusted. One by one, developers stopped using them and went back to feature flags and testing on staging, which is exactly the situation the namespaces were supposed to fix. #### The mocking trap Faced with that upkeep, any sensible team starts looking for things to stop maintaining. Instead of updating the namespace every time a third-party API changes, why not mock its responses? The response format changes far more slowly than the service itself. Instead of running a pile of unrelated services, why not emulate them with a localstack-style tool? Each of these swaps buys stability and costs fidelity. The stack gets easier to maintain and less like production at the same time, and the risk of surprises between development, staging, and production goes right back up. Test fidelity was the whole reason we handed out namespaces in the first place. Follow the mocking logic far enough and you end up with an environment so hollowed out that you might as well mock everything and run it on a laptop. Which layers of testing can safely use a mock, and which need the real dependency, is the subject of our [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). #### Deploy-time bugs live in data and dependencies There is a reason fidelity keeps coming up. The bugs that surface at deploy time overwhelmingly come from two places: data and dependencies. Namespaces make the compute side easy, since your services run in the real cluster alongside everyone else’s. Data is the hard part. To be isolated, each namespace needs its own datastore instance, whether that is a managed cloud database or a containerized one inside the cluster, and something has to create those temporary databases and load the right seed data every time an environment spins up. That is a real system you have to build and operate. Third-party dependencies are no better off; a namespace does nothing special to help you test against them. Who hasn’t had code fail in production because only the production database contained [little Bobby Tables](https://xkcd.com/327/)? #### What teams move to To be fair to namespaces: they work well for simple apps with a handful of services, they work for teams with dozens of engineers, and they can keep working at large scale if someone actively maintains them. But the failure mode is a vicious cycle. The moment maintenance slips, sync problems multiply, developers hit bugs that are really environment drift, and they quietly route around the whole thing. The teams that scale past this tend to land on the same insight: deploy only what changed, and share everything else. In a discussion on a private Slack (quoted with permission), Mason Jones described how his previous company got there: > At my previous company as we scaled up we used a shared dev/test cluster and built our own tooling to provide on-demand namespaces. A namespace isn’t a “dedicated instance of infra”, it’s just a logical space in which devs can deploy their own version of something. We often used it with one namespace per team. They only deployed the containers they were changing, and used the shared instances of everything else, so it was actually quite economical. \[…\] there are “hidden” complications to making this work, no question. We customized our service mesh so that requests to another service would first try to target a version of the service in the same namespace, and if there wasn’t one then the request would go to the shared namespace. That allowed devs to run a custom version of a service privately, while others could use the shared one. Notice what did the heavy lifting there. It was not the namespace; it was the request routing layered on top of a shared baseline. [Lyft](https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-3-extending-our-envoy-mesh-with-staging-fdaafafca82f), [Razorpay](https://www.youtube.com/watch?v=1hZY-vg7kSs), and [DoorDash](https://doordash.engineering/2022/06/23/fast-feedback-loop-for-kubernetes-product-development-in-a-production-environment/) all arrived at versions of the same approach, letting individual developers run just the service under test against a high-fidelity shared environment. That approach, duplicating only what changed on a shared baseline, is what our guide to [ephemeral environments in Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers in depth. If you want to talk about high-fidelity testing on Kubernetes at scale, or you hated this article and want to tell Nica about it, [join the Signadot Slack](https://join.slack.com/t/signadotcommunity/shared_invite/zt-1estxm8pv-qfiaNfiFFCaW~eUlXsVoEQ) and talk directly with the team. #### Frequently asked questions What is a namespace-based test environment? It is a testing setup where each team or developer gets a dedicated Kubernetes namespace within a shared cluster to deploy and test their services, isolated at the application level from other namespaces. What are the drawbacks of using namespaces for testing? Each namespace needs its own datastores and seed data, there is no built-in way to keep namespaces in sync with each other or with production, spin-up time grows with application complexity, and namespaces often go unmaintained and unreliable as teams scale. Do namespaces scale for large teams? They work well for simple apps or smaller teams, and can work for large teams if actively maintained. At scale, sync and maintenance problems usually push teams toward a shared baseline instead, where each change duplicates only the services it touched and request routing sends its traffic to them. What is the difference between a Kubernetes namespace and a cluster? A namespace is a logical partition inside a single Kubernetes cluster, isolating resources at the application level. A cluster is a complete, independent Kubernetes environment with its own control plane and nodes. Namespaces are cheaper and simpler, while separate clusters give stronger isolation at higher cost. What is a vCluster? A virtual cluster (vCluster) is a virtual Kubernetes control plane that runs inside a namespace of a host cluster. It gives each tenant cluster-level isolation, including their own API server and CRDs, without the cost of provisioning a separate physical cluster. Is a namespace the same as a virtual cluster? No. A namespace isolates resources at the application level within one shared control plane, while a virtual cluster provides a separate virtual control plane, which is a much stronger isolation boundary. #### Related Posts - [ *Image: It’s Time To Kill Staging: The Case for Testing in Production* Ephemeral Environments ### It’s Time To Kill Staging: The Case for Testing in Production December 3, 2025 ](https://www.signadot.com/blog/its-time-to-kill-staging-the-case-for-testing-in-production/) - [ *Image: Microservices Testing: 4 Use Cases for Sandbox Environments* Ephemeral Environments ### Microservices Testing: 4 Use Cases for Sandbox Environments March 28, 2025 ](https://www.signadot.com/blog/microservices-testing-4-use-cases-for-sandbox-environments/) - [ *Image: Multitenancy Is Fundamental to Shift-Left Testing* Ephemeral Environments ### Multitenancy Is Fundamental to Shift-Left Testing July 5, 2024 ](https://www.signadot.com/blog/multitenancy-is-fundamental-to-shift-left-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## How to Test Event-Driven Architectures Without Duplicating Infrastructure Source: https://www.signadot.com/blog/testing-event-driven-architectures-with-signadot/ Summary: The general model for testing event-driven systems with message-level isolation (a routing key propagated inside every message, consumer-side selective consumption, and a central Routes API), applied to pub/sub fan-out, work queues, sagas, and CDC, cron, and batch flows. ### How to Test Event-Driven Architectures Without Duplicating Infrastructure A model for testing event-driven systems on one shared baseline: carry a routing key in every message, let consumers choose what to process, and apply it to pub/sub, CDC and batch flows. Author Anirudh Ramanathan Published August 16, 2024 Updated August 31, 2026 Reading time 7 min Topics [Integration/E2E Testing](https://www.signadot.com/blog/category/integration-e2e-testing/) Event-driven architectures decouple services beautifully and complicate testing in equal measure. In a request-driven system, a test is a request and a response, and isolation is a routing problem that service meshes solved years ago. In an event-driven system, a single change fans out: a producer publishes an event, a broker distributes it, several consumers react, and some of those consumers publish events of their own. Testing a change to any one of those pieces means exercising that whole flow, and the traditional answer, duplicating the broker and every consumer per test environment, is [expensive, slow, and permanently out of date](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/). There is a better model. It tests every change against one shared baseline environment, adds no infrastructure per test, and extends to every event-driven pattern in production use. This article describes the model in general terms, then walks through how each common pattern maps onto it. The [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) covers how event-driven flows fit with synchronous calls and workflows in a wider test strategy. #### The general model: message-level isolation Instead of isolating tests at the infrastructure level (separate brokers) or the channel level (separate topics), isolate at the level of individual messages. Three mechanisms make it work. **1\. Propagate the routing key inside every message.** Each [sandbox](https://www.signadot.com/kubernetes-sandbox/) is assigned an opaque routing key. Synchronous requests already carry it in headers through OpenTelemetry context propagation. The asynchronous extension is simple: when a service publishes a message while handling a request that carries a routing key, it copies the key into the message’s own metadata, such as Kafka headers, RabbitMQ headers, Pub/Sub attributes, or SQS message attributes. The key now travels with the message wherever it goes, independent of the transport path, and [OpenTelemetry auto-instrumentation](https://opentelemetry.io/blog/2022/instrument-kafka-clients/) can do the copying without application code changes. **2\. Decide on the consumer side which instance processes each message.** Every variant of a consumer, the baseline `C1` and any sandboxed versions like `C1'`, subscribes to the same message stream, each through its own consumer group or subscription, so every variant sees every message. A small consumption check then ensures exactly one of them acts: a sandboxed consumer processes only messages whose routing key is in its set (its sandbox’s key, plus the keys of any route groups containing that sandbox), and the baseline consumer processes only messages that carry no routing key or a key no sandbox claims. *Image: Two messages pass through the queue, and only the message carrying routing key rk1 is processed by the sandboxed consumer while the baseline consumer handles the other* **3\. Resolve routing keys against a central Routes API.** The consumption check needs the current mapping between routing keys and sandboxes, and that mapping lives in one place. The Signadot Operator serves it inside the cluster through the [Routes API](https://github.com/signadot/routesapi), a set of gRPC and REST endpoints that consumers can poll or stream and cache locally. Platform teams typically wrap the lookup and the consumption check in a small client library, so application teams get message isolation by importing it. That is the whole model: one broker, one baseline environment, and any number of concurrent tests whose cost does not grow with the number of testers. For the full routing key contract and a complete Kafka implementation, see the [complete guide to testing Kafka-based microservices](https://www.signadot.com/docs/tutorials/testing-kafka-microservices). Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### How each event-driven pattern maps to the model Event-driven systems are built from a handful of recurring patterns, and each one reduces to the same three mechanisms. ##### Pub/sub fan-out: one event, many subscribers A checkout event fans out to payment, inventory, and notification services. Each subscribing service applies the consumption check independently, so you sandbox only the subscriber you changed: your test event reaches every subscriber, the sandboxed version of the one service under test processes it, and the baseline versions of every other subscriber handle it as usual. Because the routing key lives in the message rather than the transport path, fan-out needs no special handling at all. ##### Work queues and competing consumers In a work queue, worker instances compete and exactly one takes each job. That guarantee normally comes from sharing a consumer group, which is precisely what sandboxed variants must not do. Instead, each variant (the baseline worker pool and each sandboxed worker) gets its own consumer group or subscription, so each variant observes every job, and the consumption check preserves the one-processor guarantee across variants: the matching variant processes the job, and the others skip it and acknowledge. ##### Multi-hop chains and choreographed sagas Flows like order placed, then payment captured, then shipment created span several producer-consumer hops. The routing key must survive every hop, so each consumer copies the key from the message it processes into any requests or messages it emits, which is the same context propagation rule the synchronous half of the system already follows. When a change spans multiple services in the chain, a [route group](https://www.signadot.com/docs/reference/route-groups/spec) assigns one routing key to several sandboxes, so the whole modified path is exercised together while everything else stays baseline. ##### Flows that do not start with a request: CDC, cron, and batch Change data capture pipelines and scheduled jobs have no incoming request to carry a routing key, so the key has to be established at the source. For CDC tools like Debezium, that usually means a metadata column in the source rows that the producer copies into message headers. For a cron job that reads rows and publishes them, the job tags each message according to the tenant the row belongs to. Batch processors add one more rule: group messages by routing context, so a single batch never mixes messages that belong to different sandboxes. Once the key is in the message, everything downstream works exactly as in the request-driven case. #### The same model across brokers The three mechanisms rely on broker features that are nearly universal: metadata attached to each message and a way for multiple consumer variants to observe the same stream. In Kafka that is headers and consumer groups; in RabbitMQ, headers and per-variant queues bound to the same exchange; in Google Pub/Sub, attributes and per-sandbox subscriptions; in SQS, message attributes. The primitives differ but the contract does not, and the broker-specific details are covered in [testing microservices with message isolation for Kafka, SQS, and more](https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/). #### Seeing it work The fastest way to make this concrete is to run it. The [complete Kafka guide](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) includes a hands-on tutorial with a small demo application where you can watch each case live: a baseline message handled by the baseline consumer, a sandboxed consumer picking up only its own messages, an unmatched routing key falling through to the baseline, and a route group carrying a change through a forked producer and a forked consumer together. #### Conclusion Testing event-driven architectures does not require duplicating brokers, topics, or consumers per environment. Three mechanisms, a routing key propagated inside every message, a consumer-side check that decides which variant processes it, and a central Routes API that holds the mapping, are enough to give every developer and every pull request an isolated end-to-end test path through the real system. The model covers pub/sub fan-out, work queues, multi-hop sagas, and even flows that start in a database log or a cron job, and it works across Kafka, RabbitMQ, Pub/Sub, and SQS. Signadot provides the pieces so you do not build them yourself: sandboxes with assigned routing keys, route groups for multi-service changes, and the Routes API served by the operator in your cluster. [Sign up](https://www.signadot.com/signup/) to try it on your own cluster, or start with the [hands-on Kafka tutorial](https://www.signadot.com/docs/tutorials/testing-kafka-microservices). #### Frequently asked questions How do you test event-driven architectures? Test against a shared baseline environment instead of duplicating brokers and consumers for every test. Each change runs in a sandbox: messages triggered by a test carry the sandbox's routing key in their metadata, every consumer variant sees the message stream, and a small consumption check ensures exactly one variant processes each message. This gives isolated end-to-end tests across producers, the broker, and consumers on one shared infrastructure. How does selective consumption decide which consumer processes a message? Each sandboxed consumer processes only messages whose routing key belongs to it, meaning its own sandbox's key plus the keys of any route groups that include it. The baseline consumer processes messages that carry no routing key or a key no sandbox claims. Consumers make this decision locally against a cached copy of the routing-key mapping served by a central Routes API. Does this approach work with brokers other than Kafka? Yes. The model relies on two broker features that are nearly universal: message metadata (headers or attributes) to carry the routing key, and a way for each consumer variant to see the message stream, such as consumer groups, subscriptions, or queue bindings. It applies to Kafka, RabbitMQ, Google Pub/Sub, AWS SQS, and similar systems. How do you test event-driven flows that do not start with a request, like CDC or cron jobs? There is no incoming request to carry the routing key, so embed the routing metadata at the source instead: a metadata column that the CDC producer copies into message headers, or per-tenant tagging in the job that generates the messages. Once the key is in the message, the same selective consumption applies downstream. #### Related Posts - [ *Image: What's the real ROI of writing Mocks?* Integration/E2E Testing ### What's the real ROI of writing Mocks? September 26, 2023 ](https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing/) - [ *Image: The wrong way to use DORA Metrics* Integration/E2E Testing ### The wrong way to use DORA Metrics April 1, 2024 ](https://www.signadot.com/blog/the-wrong-way-to-use-dora-metrics/) - [ *Image: Why AI Features Break Microservices Testing* Integration/E2E Testing ### Why AI Features Break Microservices Testing July 11, 2025 ](https://www.signadot.com/blog/why-ai-features-break-microservices-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Testing Microservices: Message Isolation for Kafka, SQS, More Source: https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/ Summary: How the message-isolation pattern maps onto specific brokers, including Kafka, RabbitMQ, Google Pub/Sub, and AWS SQS. ### Testing Microservices: Message Isolation for Kafka, SQS, More How the message isolation pattern maps onto Kafka, RabbitMQ, Google Pub/Sub, AWS SQS and NATS, the SNS fan-out SQS needs, and the reusable pieces platform teams can build to support it. Author Arjun Iyer Published April 4, 2025 Reading time 7 min Topics [Integration/E2E Testing](https://www.signadot.com/blog/category/integration-e2e-testing/) _Originally posted on_ [_The New Stack_](https://thenewstack.io/testing-microservices-message-isolation-for-kafka-sqs-more/)_, by Arjun Iyer._ The message isolation pattern works across most popular message brokers, while implementation details vary based on architecture and terminology. When I talk to engineering leaders about testing event-driven microservices, I hear a common pain point: “Our async flows are nearly impossible to test reliably.” Message isolation is one technique within the wider practice covered in the [complete guide to microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/). Message isolation is one of the stateful problems the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers, alongside databases and third-party APIs. Previously, we explored how [OpenTelemetry and message isolation](https://www.signadot.com/docs/tutorials/testing-kafka-microservices) enable testing Kafka-based workflows without duplicating infrastructure. The approach resonated with many teams facing the challenge of testing asynchronous flows cost-effectively, but I’ve received numerous questions about extending this pattern to other message brokers. - “Does this work with AWS SQS?” - “Can we apply this to RabbitMQ?” - “What about Google Pub/Sub?” The short answer is yes. The message isolation pattern works across most popular message brokers but implementation details vary based on each system’s architecture and terminology. #### The Universal Pattern: Message Isolation Before diving into specific platforms, let’s recap the core problem and our approach: ##### The Problem Testing microservices that communicate via message queues presents several challenges: - Setting up separate test environments is expensive and difficult to maintain. - Async workflows are hard to verify without full-system tests. - Integration issues often appear only after code is merged. - Mocks frequently drift from reality, leading to false positives. ##### The Solution Our message isolation approach addresses these challenges through these key principles: 1. **Share infrastructure:** Use a single baseline environment accessible to all sandboxes. 2. **Propagate context:** Pass routing keys in message metadata via OpenTelemetry. 3. **Duplicate messages:** Allow both baseline and sandbox services to receive copies of the same messages. 4. **Selectively process:** Filter messages in consumers based on sandbox routing keys. The goal is straightforward: Enable multiple developers to test their changes in isolation without interference while avoiding the cost and complexity of duplicating your entire messaging infrastructure. #### Apache Kafka: The Reference Implementation Kafka remains our reference implementation for message isolation testing due to its robust consumer group model. In Kafka, multiple consumer groups can independently process the same messages from a topic, creating a natural pattern for our testing approach. With Kafka, each producer includes a routing key in message headers, and consumer groups provide the isolation mechanism. When a developer deploys their service version to test, it joins a unique consumer group (such as `my-service-group-sandbox123`), receiving a copy of all messages while the baseline version continues operating in its original consumer group. *Image: Kafka isolation pattern* #### RabbitMQ: Exchanges and Bindings RabbitMQ’s architecture differs from Kafka because it uses exchanges and bindings as intermediaries between producers and consumers, allowing more flexible routing patterns. In RabbitMQ, exchanges distribute messages to queues through bindings. The message isolation pattern works by creating temporary queues for sandbox services that are bound to the same exchanges as the baseline queues. For example, if your baseline service consumes from a queue bound to a topic exchange, your sandbox version would create a new temporary queue bound to the same exchange with matching routing keys. While RabbitMQ offers broker-level filtering through binding patterns, we still need consumer-side filtering to check the sandbox routing key against a central mapping service. The consumer would extract the routing key from the message headers, call a mapping service to determine if it should process the message and then proceed accordingly. *Image: RabbitMQ isolation diagram* #### Google Cloud Pub/Sub: Subscriptions Google Cloud Pub/Sub organizes messages around topics and subscriptions, making message isolation straightforward. A topic can have multiple subscriptions, and each subscription receives a copy of every message published to the topic – aligning perfectly with our sandbox model. For sandbox testing, create temporary subscriptions to the same topics your baseline services use. The subscriber would implement consumer-side filtering to extract the routing key from message attributes, check against the mapping service and process messages accordingly. *Image: Pub/Sub isolation diagram* #### AWS SQS: The Special Case AWS SQS presents a unique challenge for message isolation because it doesn’t natively support the fan-out pattern we rely on in other brokers. In SQS, messages are consumed by a single consumer; there’s no built-in mechanism for multiple consumers to receive the same message, and messages are automatically deleted after being processed. Therefore, we need to consider alternate approaches: Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Option 1: SNS + SQS Pattern Use Amazon SNS (Simple Notification Service) as the entry point to fan out messages to multiple SQS queues. Publishers send messages to an SNS topic, which delivers copies to multiple subscribed SQS queues — one for baseline, and separate ones for each sandbox environment. This approach requires no changes to existing consumers as they continue to read from their dedicated queues. ##### Option 2: Dedicated Test Queues Create temporary SQS queues for testing with sandboxed producers and consumers. Deploy a new instance of your producer that sends messages to the temporary test queue and configure your test consumer to read from this new queue. Each sandbox gets its own isolated queue, producer and consumer. For most teams, Option 1 provides the cleanest solution. Setting up an SNS topic that publishes to both your baseline SQS queue and any temporary testing queues maintains the decoupled architecture while enabling the isolation needed for testing. The overhead of message duplication is typically negligible compared to the cost of duplicating your entire infrastructure. *Image: SQS isolation diagram* #### NATS: Subjects and Queue Groups [NATS](https://nats.io/), with its lightweight and high-performance design, offers features well-suited for sandbox testing. We can leverage NATS queue groups, which function similarly to Kafka consumer groups. By creating sandbox-specific queue groups, test services receive copies of messages without interfering with baseline services. Your test service would join a unique queue group (such as `service-sandbox123`), subscribe to the same subjects as your baseline service and implement consumer-side filtering based on routing keys. *Image: NATS isolation diagram* #### Platform Team Implementation The message isolation approach becomes much more maintainable when platform teams build reusable components that handle the sandbox-specific logic. ##### Creating Sandbox-Aware Consumer Libraries Platform teams can create thin wrappers around standard client libraries that automatically handle sandbox filtering: Application developers use this exactly like the standard consumer: This approach makes the sandbox filtering completely transparent to application teams. ##### Sandbox Life Cycle Management Platform teams can provide standardized hooks around the life cycle of sandboxes that automate resource management: *Image: Diagram of sandbox lifecycle hooks that register the sandbox ID in the mapping service, create queue groups and subscriptions, and clean up all sandbox resources on deletion* When a developer creates a new sandbox for testing: - The platform automatically registers the sandbox ID in the central mapping service. - Necessary consumer groups, queues or subscriptions are created with consistent naming `(service-{sandbox-id})`. - When testing completes, all sandbox-specific resources are automatically removed. This hook-based approach makes sandbox isolation part of the platform’s core capabilities rather than a burden on each development team. #### Conclusion Message isolation patterns work across all major message brokers, though implementation details vary based on each system’s architecture. The approach consistently delivers three key benefits: 1. **Cost efficiency:** Share infrastructure instead of duplicating it. 2. **Testing fidelity:** Test against real dependencies instead of mocks. 3. **Developer velocity:** Catch issues earlier in the development process. At [Signadot](https://www.signadot.com/), we’ve implemented these patterns across dozens of customer environments using different messaging systems. Companies like [Brex](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/), [Earnest](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) and [ShareChat](https://www.signadot.com/case-studies/sharechat-chooses-signadot-giving-devs-high-quality-testing-feedback/) have used these approaches to transform their microservice testing. If you’re interested in exploring how to apply these concepts to your specific environment, sign up and join our [community Slack channel](https://signadotcommunity.slack.com/join/shared_invite/zt-1estxm8pv-qfiaNfiFFCaW~eUlXsVoEQ#/shared-invite/email). As microservice architectures continue to evolve, effective testing of asynchronous workflows becomes increasingly critical. The message isolation pattern provides a practical, scalable approach regardless of which message broker you’ve chosen. #### Related Posts - [ *Image: How to Test Event-Driven Architectures Without Duplicating Infrastructure* Integration/E2E Testing ### How to Test Event-Driven Architectures Without Duplicating Infrastructure August 16, 2024 ](https://www.signadot.com/blog/testing-event-driven-architectures-with-signadot/) - [ *Image: Who Should Run Tests? On the Future of QA* Integration/E2E Testing ### Who Should Run Tests? On the Future of QA June 28, 2024 ](https://www.signadot.com/blog/who-should-run-tests-on-the-future-of-qa/) - [ *Image: Introducing a New Paradigm for End-to-End Feature Testing In Microservices* Integration/E2E Testing ### Introducing a New Paradigm for End-to-End Feature Testing In Microservices January 25, 2023 ](https://www.signadot.com/blog/new-paradigm-for-end-to-end-feature-testing-in-microservices/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## What Is Shadow Testing? Four Ways to Bulletproof Your APIs Source: https://www.signadot.com/blog/shadow-testing-superpowers-four-ways-to-bulletproof-apis/ Summary: What shadow testing is, how it compares to canary releases and feature flags, whether to run it in staging or production, and four ways teams use it before merge. ### What Is Shadow Testing? Four Ways to Bulletproof Your APIs What shadow testing is, how it differs from canary releases and feature flags, its four uses (API contracts, performance, log patterns and security), and why it suits AI coding agents. Author Arjun Iyer Published April 21, 2025 Updated August 31, 2026 Reading time 12 min Topics [Integration/E2E Testing](https://www.signadot.com/blog/category/integration-e2e-testing/) Shadow testing means running a new version of a service alongside the current version in an isolated pre-production environment, sending both the same representative traffic, and comparing their responses to catch regressions before release. No real user ever sees the unreleased version, which is what makes it safe to test changes that would be too risky to canary. I’ve spent the last 20+ years building distributed systems, and one truth remains constant: [microservices testing](https://www.signadot.com/the-complete-guide-to-microservices-testing-from-local-development-to-production/) is challenging. Shadow testing is a powerful approach to validate microservice changes safely because engineering leaders are eager for better ways to test complex [microservice architectures](https://www.signadot.com/blog/how-microservices-communicate-with-each-other/). Before getting into what shadow testing unlocks, it is worth being precise about what it is, and what it is not. Shadow testing covers what a contract cannot, and the [complete guide to contract testing](https://www.signadot.com/contract-testing/) explains where that boundary sits. #### What Is Shadow Testing? The detail that matters most for everything below is where this runs. Shadow testing happens in an isolated, pre-production sandbox, not against live production. The “same traffic” sent to both versions is usually synthetic or replayed requests crafted to closely mimic real conditions, routed to an isolated copy of the changed service running against the shared baseline. The closer that representative traffic is to the real thing, the more useful the result, but no real user is ever in the loop. ##### Shadow Testing vs. Canary Releases and Feature Flags That is what separates shadow testing from two approaches it often gets confused with: - **Canary releases** roll a change out to a small slice of real users and watch for problems. Failures still reach those users. Shadow testing runs before anyone sees the change. - **Feature flags** turn functionality on and off at runtime. They are useful for controlled rollouts, but they sit late in the lifecycle and do not give you a clean comparison of old behavior against new. Shadow testing is a shift-left technique: it catches regressions, contract breaks, and performance issues early, inside a sandbox, without the brittleness of hand-maintained integration tests. The real magic happens when we leverage this parallel execution to derive actionable insights that traditional testing methods simply can’t provide. Let’s explore four powerful testing approaches that shadow testing enables, each providing unique signals that help developers ship with confidence. #### 1\. API Contract Testing: Detecting Breaking Changes API contract testing is perhaps the most immediately valuable application of shadow testing. Traditional [contract testing relies on mock services](https://www.signadot.com/blog/why-developers-shouldnt-write-mocks-a-guide-to-modern-testing/) and schema validation, which can miss subtle compatibility issues. Shadow testing takes contract validation to the next level by comparing actual API responses between versions. Here’s how it works: 1. Deploy your updated service alongside the baseline version. 2. Route identical traffic to both versions. 3. Compare the responses in real time. 4. Identify any divergence that indicates potential contract breaks. This approach catches breaking changes that would be invisible to conventional testing: field type changes, new required fields, altered error responses and even performance degradation that might violate service-level agreements (SLAs). This approach combines the best of specification-based contract testing with runtime validation. Rather than just checking if your new API implementation conforms to a static specification, you’re directly measuring compatibility with everything currently consuming it. *Image: API Contract Testing with Shadow Testing* For instance, when a team at a fintech company implemented this approach, it caught a subtle breaking change where a field was being converted from string to integer, something its OpenAPI validators hadn’t flagged, but that would have broken several downstream consumers. The shadow testing system highlighted the discrepancy immediately, allowing the team to fix it before merging. #### 2\. Performance Testing: Compare Side by Side Performance testing is another area where shadow testing shines. [Traditional performance testing usually happens late in the development cycle](https://www.signadot.com/articles/shift-left-testing/) in dedicated environments with synthetic loads that often don’t reflect real-world usage patterns. With shadow testing, you can: 1. Run your new code against realistic, production-like traffic patterns. 2. Compare key metrics against the baseline version in real time. 3. Identify performance regressions before they affect users. 4. Validate that optimizations actually deliver expected improvements. This approach is particularly powerful when combined with canary analysis tools like [Kayenta](https://github.com/spinnaker/kayenta), which can automatically analyze statistical significance in performance differences. *Image: Performance/Load Testing with Shadow Testing* A retail platform I worked with used this approach to validate a major database query optimization. While its synthetic benchmarks showed a 40% improvement, [shadow testing against real traffic](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) revealed edge cases where certain user queries were performing worse. The team was able to address these issues before deploying, avoiding what would have been a serious production incident during the company’s peak shopping season. The beauty of this approach is that it doesn’t just test whether your code functions correctly, it tests whether it functions correctly under real-world conditions with diverse traffic patterns that would be nearly impossible to simulate accurately. #### 3\. Log Analysis: Uncovering Hidden Issues Log analysis is often overlooked in traditional testing approaches, yet logs contain rich information about application behavior. Shadow testing enables sophisticated log comparisons that can surface subtle issues before they manifest as user-facing problems. With log-based shadow testing, you can: 1. Capture logs from both baseline and test versions. 2. Apply machine learning (ML) clustering to identify new or changed log patterns. 3. Highlight potentially problematic log sequences. 4. Discover errors, warnings or unexpected behaviors that might not trigger test failures. *Image: Log Analysis with Shadow Testing* I’ve seen this approach catch issues that would have slipped through every other testing layer. One engineering team discovered their new code was generating a cluster of database connection warnings that weren’t present in the baseline version. While these warnings didn’t cause immediate failures, they were an early indicator of connection pool exhaustion that would have eventually led to cascading timeouts under load. This approach bridges testing and observability, creating a feedback loop that helps developers understand the operational impact of their changes before deployment. #### 4\. API Security Analysis: Shift-Left Security Validation Perhaps the most innovative application of shadow testing is in the security domain. Traditional security testing often happens too late in the development process, after code has already been deployed. Shadow testing enables a true shift left for security by enabling dynamic analysis against real traffic patterns. The approach works like this: 1. Deploy your service changes to a sandbox environment. 2. Run security scanning tools like OWASP ZAP against the sandboxed version. 3. Identify and fix security vulnerabilities before merging your code. 4. Prevent security issues from reaching production in the first place. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) ##### Runtime Security Analysis With Shadow Testing Zed Attack Proxy (ZAP) is an open source security tool that acts as a “man-in-the-middle” proxy, intercepting and inspecting messages between client and server. It automatically scans for security vulnerabilities by analyzing responses and simulating attacks to discover weaknesses like SQL injection, cross-site scripting and authentication flaws. *Image: Runtime Security Analysis with Shadow Testing* This approach is particularly powerful for detecting issues like new API endpoints that might be missing authentication checks, input validation flaws or information disclosure vulnerabilities, all before your code gets merged. Automated security testing embedded in CI/CD pipelines has moved from a differentiator to table stakes, and shadow testing provides an ideal framework for this kind of automated security validation. By integrating security scanning directly into your pre-merge testing workflow, you catch vulnerabilities during development when they’re easiest and cheapest to fix. This eliminates the traditional security bottleneck where issues are discovered late in the development cycle, requiring costly rework and delaying releases. #### Self-Maintaining Tests: Automatic Pattern Detection What makes these shadow testing approaches particularly valuable is their inherently low-maintenance nature. Unlike traditional testing approaches that require constantly updating test suites, mocks and assertions, shadow testing uses representative traffic and automated comparisons to detect issues with minimal human intervention. The power lies in the baseline comparison: by running both versions side by side and automatically identifying differences in behavior, these tests essentially write themselves. They can detect subtle issues, emerging patterns and regressions without requiring engineers to anticipate every possible edge case. This fundamentally changes the testing paradigm. Instead of spending countless hours maintaining brittle test fixtures, teams can focus on building features while automated differential analysis provides the safety net. The pioneers of microservices at companies like Netflix and Uber have been using variations of these techniques for years. The difference now is that modern tools are making these low-maintenance, high-signal testing approaches accessible to engineering teams of all sizes. #### How shadow testing works in practice The mechanics are the same wherever you run it, and the pattern was pioneered by tools like [Diffy](https://blog.x.com/engineering/en_us/a/2015/diffy-testing-services-without-writing-tests), built at Twitter to test services by comparing responses instead of writing assertions. Three things happen: - **Traffic execution.** The new version (V-Next) runs alongside the current version (V-Current), and both receive the same requests. That gives you a direct, apples-to-apples comparison of behavior. - **Response comparison.** Outputs are captured and diffed to surface regressions. A good implementation uses a [relevancy model](https://www.signadot.com/blog/rest-api-testing-using-relevancy-weighted-diffs/#calculating-diff-operation-relevance) to decide which differences matter and which are benign noise, like timestamps or reordered fields, so you are not buried in false positives. - **Observability and metrics.** Performance and error patterns from both versions are analyzed to confirm the change is stable before it goes any further. As Microsoft’s [Engineering Fundamentals Playbook](https://microsoft.github.io/code-with-engineering-playbook/automated-testing/shadow-testing/) describes it, shadow testing captures the differences between the current environment and a candidate environment, then reduces risk before you introduce the new release. *Image: Shadow testing: a new version runs alongside the current one, both get the same traffic, their behavior is compared, and the new version is validated* ##### Where Do You Run Shadow Testing: Staging or Production? Where you run it is a choice. **Staging shadow testing** is easier to set up, sidesteps compliance and data-isolation concerns, and uses synthetic or anonymized traffic. **Production shadow testing** gives the most accurate signal from live traffic but needs real safeguards for data handling and workload isolation. For most teams an isolated sandbox on a shared pre-production cluster hits the sweet spot. That is exactly what [SmartTests](https://www.signadot.com/docs/concepts/smart-tests-and-jobs) are in Signadot. Shadow testing is the technique; a SmartTest is that technique packaged as a runnable test. Each run spins up an isolated sandbox for a branch, sends representative traffic to the changed service alongside the baseline, and compares request and response behavior in real time, using a relevancy model to separate meaningful regressions from benign noise. There are no assertions to write and no mocks to maintain, and it runs in Kubernetes with Istio, Linkerd, or no service mesh at all. *Image: Signadot SmartTests comparing a baseline and a sandboxed version of a service* #### Why this matters more now that agents write the code A growing share of code is now written by AI coding agents like Claude Code, Codex, and Cursor, and they produce changes far faster than anyone can hand-review them. Writing the change is no longer the hard part. Knowing whether it is safe to merge is. That is exactly the signal shadow testing produces. Shadow testing also fits how agents work better than almost any other method, because it needs no hand-written assertions. The agent makes a change, its version is deployed into an isolated sandbox on the shared cluster, the same representative requests are sent to both the agent’s version and the baseline, and the differences are compared automatically. The agent does not have to anticipate every edge case or maintain a brittle suite. It gets a direct, behavioral answer: here is what your change does differently, and whether that difference is a regression. The isolation is what makes this work at agent speed. Each change is tested against the real, shared baseline without standing up a full environment of its own and without colliding with the other agents and developers doing the same thing. Many changes can be validated in parallel on a single cluster. This is the same model behind [ephemeral environments on Kubernetes](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/), which duplicate only what changed, applied to differential testing. In Signadot this is delivered through [SmartTests](https://www.signadot.com/docs/concepts/smart-tests-and-jobs), which let an agent kick off a shadow comparison the same way it runs any other test. Paired with [Jobs](https://www.signadot.com/docs/reference/jobs) and [Plans](https://www.signadot.com/docs/reference/plans), an agent can run the comparison, read the result, and decide whether its change is ready, all before it opens a pull request. For the wider picture, see [validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/). #### Conclusion Shadow testing is unlocking a new generation of testing approaches that provide deeper insights and more confidence than traditional methods. By detecting API contract issues, performance regressions, suspicious log patterns and security vulnerabilities before they reach production, teams can ship faster with greater confidence. If you’re interested in exploring how shadow testing could transform your microservices testing strategy, I’d love to continue the conversation. Check us out at [signadot.com](https://www.signadot.com/) or join our community discussions. The future of microservices testing isn’t just about running more tests earlier, it’s about getting better signals that truly predict production behavior. Shadow testing is leading this evolution, proving that with the right approach, we can have both speed and quality. #### Frequently asked questions What is shadow testing? Shadow testing runs a new version of a service alongside the current version in an isolated pre-production sandbox, sends both the same representative traffic, and compares their responses to catch contract breaks, performance regressions, and other issues before release. Is shadow testing done in production? Usually not. Shadow testing typically runs in an isolated pre-production sandbox using synthetic or replayed traffic that closely mimics real conditions. Some teams mirror live production traffic into a shadow copy, but that requires strict safeguards for data handling, and no real user ever receives a response from the unreleased version. What is the difference between shadow testing and a canary release? A canary release rolls a change out to a small slice of real users, so failures still reach those users. Shadow testing compares the new version against the baseline on the same traffic before any user sees the change, making it a shift-left technique rather than a rollout strategy. How do Signadot SmartTests relate to shadow testing? SmartTests are shadow testing packaged as a runnable test primitive. Each run spins up an isolated sandbox, sends representative traffic to the changed service alongside the baseline, and uses a relevancy model to surface meaningful differences. AI coding agents use them to validate changes automatically before opening a pull request. #### Related Posts - [ *Image: Why large engineering teams are testing on Kubernetes* Integration/E2E Testing ### Why large engineering teams are testing on Kubernetes May 3, 2023 ](https://www.signadot.com/blog/large-engineering-teams-testing-on-k8s/) - [ *Image: Testing Event-Driven Architectures with OpenTelemetry* Integration/E2E Testing ### Testing Event-Driven Architectures with OpenTelemetry August 9, 2024 ](https://www.signadot.com/blog/testing-event-driven-architectures-with-opentelemetry/) - [ *Image: Improve Developer Velocity by Decentralizing Testing* Integration/E2E Testing ### Improve Developer Velocity by Decentralizing Testing March 23, 2024 ](https://www.signadot.com/blog/improve-developer-velocity-by-decentralizing-testing/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Customer Case Studies ## Brex Source: https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/ Summary: How Brex scaled developer testing across 100s of engineers and coding agents, cutting preview infrastructure costs by 99% and saving about $2 million annually by replacing full environment duplication with Signadot sandboxes. *Image: Brex logo* Case study ### How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents Signadot replaced Brex's namespace-based preview environments across 800+ microservices, cutting infrastructure costs by 99%, taking service previews from 30 to 60 minutes down to under 5, and pushing developer CSAT 28 points higher. Brex ran preview environments the way many fast-growing platform teams do. Any developer who wanted to test a change got a duplicated Kubernetes namespace running the entire application. At 800+ microservices, that model stopped working. The tradeoffs of that model, and the alternatives to it, are covered in the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/). The environments were pushing the scaling limits of a single Kubernetes control plane, the compute bill for running hundreds of services per developer was enormous, and they broke often enough that teams stopped maintaining them and pushed their testing into staging instead. Brex replaced that model with Signadot, which creates isolated sandboxes (lightweight ephemeral environments) inside a single shared cluster. Only the deltas of the one or two services a developer or a coding agent are working on get duplicated. Every other dependency is served by the shared cluster, with real data and real traffic patterns. > “Signadot for the preview environments use case fit what we were trying to do better than anything else. It was a more mature solution than the other stuff that we were looking at. And the return on the investment was obvious… just in infrastructure costs, it saves us about $2 million annually.” Phil Burrows · Head of Platform Engineering, Brex #### About Brex [Brex](https://www.brex.com/) is the finance platform that startups and enterprises use to run corporate cards, expenses, bill pay, banking, and accounting in one system. Their product strategy is built around intelligent finance: AI that reads receipts, drafts expense reports, codes transactions to the general ledger, and enforces spend policy on its own, to the point where the majority of expenses on the platform are prepared with no human involvement at all. Delivering that requires real engineering scale: - **800+ microservices** running on Kubernetes - **300+ engineers** shipping into a shared monorepo - A platform team responsible for the testing and preview infrastructure all of them depend on Brex is also at the forefront of agentic engineering inside its own organization, with autonomous coding agents and agent-driven workflows adopted deeply across the engineering team. Their engineering blog documents that work in posts like [Building autonomous agents for technical tasks](https://www.brex.com/journal/building-autonomous-agents-for-technical-tasks). Signadot is a big part of the infrastructure that supports it, giving engineers and agents alike realistic environments to validate their changes in. #### Results at a Glance Infrastructure Cost−99%about $2 million saved annually Service Preview<5 minsdown from 30 to 60 minutes Developer CSAT+28 ptsversus the previous platform tooling #### The Problem: Preview Environments Nobody Maintained Preview environments in Kubernetes namespaces were powerful when Brex first built them. Each one was a duplicated namespace that gave a developer an isolated copy of the application to test and experiment against. The model degraded as the service count climbed: - Each environment ran enough microservices to push the scaling limits of a single Kubernetes control plane - Availability problems made the environments unreliable, and unreliable environments do not get maintained - Teams routed around them and tested in staging instead, which broke staging more often and blocked everyone else - Replicating 800+ services so a developer could change one or two of them was extraordinarily expensive > “The primary problem is just the cost of getting something wrong or breaking something in a preview environment was never high enough to prevent folks from leaving things broken. And so they were constantly in a state of disrepair.” Connor Braa · Software Engineering Manager, Brex ##### Teams routed around the problem The downstream effect was that entire groups abandoned realistic testing. As Connor Braa, Software Engineering Manager at Brex, described it, “There was a group of teams building very important product features that just eschewed development against preview environments entirely, and instead focused on unit tests. After unit tests, it was tested on Staging, and were required to use feature flags way more heavily. It made the data quality and reliability problems with moving to staging way worse.” When the first high-fidelity test of a change happens in staging, staging breaks more often, other teams get blocked, and delays compound across the org. ##### The cost of duplicating everything The reliability problems came with a bill attached. > “The other big problem is cost, having isolated environments for our internal software that’s hundreds of microservices solely for the purpose of testing gets insanely expensive. We’re talking about replicating 800+ services just so a developer can test changes on 1-2 services. That’s a great deal of compute and memory.” Connor Braa · Software Engineering Manager, Brex There are hidden costs too. When a team does not feel like it is working at its best, that hurts retention, job satisfaction, and performance. Connor describes it as developers not feeling like they have the tools to ship the quality of software they want to ship. With engineering hires costing six figures, it is worth asking whether your team can afford to lose people because they are frustrated at not being able to ship. #### The Solution: Isolation Inside the Shared Cluster Signadot changed what has to be duplicated. Instead of standing up a copy of the application per developer, a sandbox holds only the changed versions of the one or two services being worked on. Requests carrying a routing key are directed through those sandboxed services, and every other call is served by the stable version already running in the cluster. Sandboxes spin up in seconds because there is almost nothing to start, and a time-to-live setting shuts them down automatically, so abandoned environments stop accumulating cost. requestrouting key: alice-1Kubernetes clusterSandboxservice-1'service-1service-2 The routing key sends one hop through the Sandbox. Every other hop is served by the shared stable cluster. requestrouting key: alice-1Kubernetes clusterSandboxservice-1'service-1service-2 The routing key sends one hop through the Sandbox. Every other hop is served by the shared stable cluster. ##### Testing on staging without the blast radius The switch did not require Brex to change how developers work, and that was part of the appeal. What it changed is where the first realistic test happens. Because a sandbox costs almost nothing to create and needs no upkeep, running integration tests alongside unit tests in the inner loop became the default rather than a luxury, and integration problems started surfacing before code review instead of after merge. It also dissolved the maintenance problem that had killed the old preview environments. Staging is an environment with real consequences attached, so it stays healthy as a matter of course, and sandboxes inherit that health rather than depending on each team’s willingness to maintain a separate environment. > “Staging has always been a place where we have a bit higher standard. It needs to be working, and there are real business consequences when it breaks. Being able to run Signadot in Staging alleviates this pressure where no one wants to maintain a separate dev environment just for dev testing.” Connor Braa · Software Engineering Manager, Brex ##### Data quality nobody has to fight for Testing in staging also solved a problem that replicated environments never could: the data. Connor is direct about this. The data quality on staging and production is far higher than anything a replication environment offers, which makes testing there fundamentally more realistic. Brex could have invested in making dev environment data realistic. But that would have required a huge cultural shift across every team to prioritize it, and in his words, changing everyone’s behavior is not a road a platform team wants to go down. With Signadot, realistic data is simply a property of the environment the sandboxes already live in. #### Accelerating Testing: A Tale of Two Loops Any engineering org larger than a few people tests code changes in two loops: - **The inner loop.** A developer or an agent changes the code and sees the result right away, while the change is still fresh and the context is still loaded. - **The outer loop.** Changes are validated after code has been reviewed and merged to main, where a failure costs far more to diagnose and fix. Brex wants as many bugs as possible caught on the inner loop. With Signadot, an individual developer or coding agent creates a sandbox and runs an accurate test of their service against every dependency it needs, on the shared cluster. Combined with unit tests, that pulls a large class of integration problems forward, ahead of the pull request rather than after it. That matters more as agents write more of the code. An agent that can only run unit tests hands its work back to a person for real validation. An agent with its own sandbox can exercise the change against live dependencies and iterate until it passes, so what arrives at the pull request has already been checked. #### The Results ##### Cost With Signadot, only the services being tested are replicated. With duplicated environments, Brex was running 800+ services so that a developer could work on one or two of them. Connor puts the waste in blunt terms: on the margin, 99.8% of a duplicated environment’s infrastructure cost went to services nobody was testing, a figure he notes sounds like an exaggeration but is not. Eliminating that waste is what produces the roughly $2 million in annual infrastructure savings, and it is why the return on investment was obvious to Brex’s platform engineering leadership from the start. ##### Velocity The cost savings changed the budget. The speed changed the day-to-day experience. Standing up a namespace-based preview environment meant deploying hundreds of services and waiting for all of them to become healthy. A sandbox deploys one. > “A typical deploy in our Signadot based tooling, end-to-end, is less than five minutes versus the 30-60 minute deploy times we would see with preview environments. Local sandboxes allow us to take that down to almost instantaneous testing.” John Salem · Senior Software Engineer, Brex A preview in under five minutes keeps a developer in flow while the change is still fresh. For coding agents it matters even more, since a fast environment is the difference between validating every iteration and validating none of them. ##### Satisfaction For a platform team, the development org’s Customer Satisfaction (CSAT) score is the number that says whether the tooling actually works for the people using it. Brex measures CSAT on its internal platform tooling, and John Salem reports that the Signadot-based tooling scored 28 points higher than the older platform engineering tech it replaced. Taken together with the cost and speed numbers, the platform team got something rare: an infrastructure change that saved money, sped developers up, and made them happier at the same time. #### Conclusion: Signadot for Developer Velocity Moving preview environments to Signadot addressed the cost, reliability, and efficiency problems that had made Brex’s previous system unusable. Isolated sandboxes inside a shared staging cluster gave developers realistic testing without the overhead of maintaining hundreds of duplicated services, and without the risk of one person’s broken change blocking everyone else. As coding agents become a bigger part of the workflow at Brex, the same isolation and speed that accelerated human developers now gives agents the infrastructure to validate their own changes autonomously. That is the piece that turns an agent from something that writes code into something that ships it. To see how lightweight ephemeral environments can work for your team, [sign up for free](https://www.signadot.com/signup/) or [book a demo](https://www.signadot.com/schedule-a-call/). #### More customer stories - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development August 27, 2026 ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% June 18, 2026 ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## DoorDash Source: https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/ Summary: How DoorDash developers achieved 10x faster feedback loops on code changes using Signadot's ephemeral sandbox environments. *Image: DoorDash logo* Case study ### How Developers and Coding Agents at DoorDash Get 10x Faster Feedback on Code Changes DoorDash eliminated their staging environment and gave developers the ability to test directly against production from their laptops — cutting feedback time from 30 minutes to 60 seconds. #### DoorDash Overview DoorDash is a technology company that connects consumers with their favorite local businesses in more than 25 countries across the globe. Founded in 2013, DoorDash builds products and services to help businesses innovate, grow, and reach more customers. DoorDash employs a multi-tenant architecture consisting of 100s of microservices operating on Kubernetes. Due to their on-demand service, the company’s extensive engineering team must innovate rapidly while ensuring minimal disruption for customers. To achieve this, DoorDash leverages end-to-end feature testing on their production environment. * * * #### Before Signadot Spinning up new Docker images for changes proved to be a time-consuming process, taking more than 30 minutes for each deployment. Additionally, if a team introduced a new change with regressions, staging would break. This significantly impacted other teams waiting for the change to be fixed or rolled back, causing further delays and inefficiencies. Relying on unit and mocked CI tests proved inadequate since they did not address the dynamic end-to-end functionality. Furthermore, developers wanted greater control over the specific user flows that required testing. Maintaining the accuracy and reliability of the staging environment independently proved to be a challenging task at the scale the team was operating. The difficulties in ensuring the fidelity of the staging environment made it harder for developers and agents to accurately test their changes before deploying them to production. These challenges created a bottleneck in the development process and hindered the team’s ability to deliver features quickly and efficiently. * * * #### After Signadot Signadot’s vision of providing developers and agents with early access to high fidelity testing in the development workflow aligned with DoorDash’s long term vision. Recognizing the potential benefits, DoorDash made the strategic decision to remove the staging environment entirely and adopt Signadot’s approach. As a result, developers and agents at DoorDash now have the ability to safely test their changes directly in the production environment from their laptops or agent runtimes. This shift allows for more efficient and streamlined testing, enabling developers and agents to iterate and refine their work with greater speed and accuracy. * * * #### How Developers use Signadot DoorDash developers make code changes to their microservice(s) locally on their workstations, creating a “Local Sandbox” environment that has a unique identifier. In order to test these changes end-to-end in production, developers access the Web front end and utilize a Chrome Extension to specify the identifier associated with their Local Sandbox. Signadot ensures that these end-to-end flows are routed to the correct local workstation based on the unique identifier. *Image: doordash screenshot 2 copy* When it comes to testing from the mobile App, they use a debug version of the mobile app that enables them to enter the identifier corresponding to their Local Sandbox. By leveraging Signadot’s operator, all requests from this session are routed to the version of the backend Service(s) running on the developer’s workstation. This setup enables the locally running service to interact with the services running in the production environment, facilitating end-to-end testing. *Image: doordash screenshot copy* With this workflow in place, developers now have the capability to thoroughly test specific user flows in the production environment from the front end. This approach gives them a high level of confidence in the functionality and stability of their code changes when deployed to production. By eliminating the need for a separate staging environment, this streamlined process enables developers to iterate and refine their work more efficiently, reducing potential bottlenecks and improving overall development speed. > “Our vision when we started talking to Signadot was that we wanted to have the same stack for testing and production. Signadot has allowed us to do just that, bringing down the lead time for our developers to test changes from 30 minutes to literally 60 seconds.” Amit Gud · Software Engineer, DoorDash * * * #### Results **30 minutes → 60 seconds.** DoorDash developers and agents now test changes end-to-end against the production environment in under a minute — with fewer rollbacks, reduced production incidents, and significantly lower cloud costs. The implementation of Signadot’s workflow has significantly reduced the time required for high-fidelity testing from the front end during each iteration. Previously, developers had to wait for CI or deployments to the staging environment, which took around 30 minutes. Now, with the new approach, developers can complete their testing within a few minutes directly from the front end. Local developmentmocks, unit tests: low fidelity, high speed code Local testinghigh fidelity manual tests commit Pull requestCI: unit, component and automated tests both test against Productionfast feedback for local testing, realistic testing for pull requests Developers benefit from a rapid iteration process on their local workstations. With Signadot, each change is quickly [tested end-to-end against the production environment](https://doordash.engineering/2022/06/23/fast-feedback-loop-for-kubernetes-product-development-in-a-production-environment/) within a few minutes. This also incentivizes developers and agents to engage in more ad-hoc testing, allowing them to validate their code changes effectively. As a result, the number of rollbacks and production incidents has decreased, saving valuable developer time. With fewer disruptions caused by issues or bugs, developers and agents can focus their efforts on building new features, enhancing productivity and enabling teams to make progress in delivering innovation to users. Additionally, the elimination of the pre-production environment has significantly reduced cloud costs for DoorDash. Resources are allocated more efficiently, resulting in a more streamlined and cost-effective development process that scales naturally as agents increase the volume of changes flowing through the pipeline. #### More customer stories - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development August 27, 2026 ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% June 18, 2026 ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Earnest Source: https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/ Summary: How Earnest shifted testing left, empowering developers to catch bugs earlier in the development cycle with isolated sandbox environments. *Image: Earnest logo* Case study ### Shifting Left with Signadot: How Earnest Empowered Developers for Early Testing Earnest automated end-to-end testing with Signadot sandboxes, parallelizing test suites to cut cycle times from 60 minutes to 10 — and ensuring developers catch issues before code reaches staging. ##### Enabling more reliable release process and a better Developer Experience Fintech company Earnest is a technology-lender that brings low-interest loans to high-potential people. With both a platform engineering and developer experience team, Earnest maintains focus on a secure, efficient release process with as few surprises as possible. With Signadot, the Developer Experience (DevEx) team was able to automate end-to-end and integration tests. This gave engineers better insight into how their services are running during a test, and caused more reliable commits to Staging. I recently got the chance to sit down with Early Ehlinger, lead on the Earnest DevEx team, and Daniel Jimenez, Infrastructure DevEx Engineer, to talk about their journey to test automation with Signadot. * * * #### Why Earnest needed Signadot As we at Signadot have written in the past, testing and experimentation can become quite difficult as microservice architecture gets more complex. When you don’t have reliable, high-accuracy integration tests before pushing to staging, the result is more out-of-band bug fixes as your team tries to diagnose problems. Without Signadot, developers at Earnest didn’t have ready access to a testing Sandbox that included up-to-date dependencies for their service. This meant that releases were often handed off without 100% confidence that the code would pass integration testing. This meant having a separate team identify the problem, try to diagnose it, and then reach out to the original product team with a description of the problem — adding friction and hurting developer velocity. > “Signadot is a key part of our solution for the most critical parts of our testing pyramid. For integration and end-to-end testing we need to have a sandbox up and running because they need a live service to talk to.” Early Ehlinger · Dev Experience Lead, Earnest Signadot duplicates only the services a change touches and wires them into the rest of the cluster, letting multiple teams use a single high-fidelity cluster for testing and to try out new features without their changes impacting other teams. For Earnest, their first use of Signadot was to allow users to create sandboxes manually: designating the services that the engineer wanted to ‘fork’ and letting this forked service interact with the baseline environment. * * * #### Moving to automated Sandboxes From manual sandboxes, the Developer Experience team found it easy to move to an automated system: “We’d been trying Signadot, letting people manually create sandboxes, for a while so switching over to automatically creating sandboxes and running tests was a pretty smooth process.” — Daniel Jimenez, Infrastructure DevEx Engineer, Earnest The team was highly motivated to improve the performance of integration testing. As Early put it: “We were looking for what can we do from an infrastructure perspective without going in and rewriting all the tests to make them fast? Previously we were parallelizing tests, but we’re doing it by hand. So you’d still end up with chunks of like 10 or 20 tests that would run in series and so they would take like, you know, 30 minutes to an hour. Whereas, the longest run on any individual test might have been 10 minutes.” This led the team to migrate Testkube so that the same tests, with the same execution time, could run faster by running in parallel. Early described the advantages: “What’s nice about Testkube is you can very easily set things up so that an individual source file in your Cypress repo or your Mocha repo or whatever, correlates to a single test in Testkube. If you design things right, we can run all of the tests in a suite in parallel. If you parallelize it, that whole chunk takes only 10 minutes instead of an hour. And we saw really drastic improvements in terms of speed just by letting something run everything in parallel for us.” Once that work was done, all that was needed to integrate Signadot was to make sure the [routing key](https://www.signadot.com/docs/concepts/sandbox) was available for the test suite, so that it could supply the appropriate request header while testing. As Early noted, “Our Dev Ex team is able to maintain all the test setup and sandbox spec in our Jenkins library. With a standard naming convention, you can automatically flag a deploy to create a sandbox for testing.” Teams leverage Cypress to automate end-to-end testing of applications deployed in Kubernetes clusters, ensuring each component interacts seamlessly across the orchestrated environment. **Before Signadot:** 30–60 minute test cycles, sequential execution, issues discovered post-staging. **After Signadot:** 10-minute parallel test runs, developer-first issue detection, reliable automated integration tests for every commit. * * * #### Testing the critical paths For fintech companies like Earnest, the steps a user takes on a path to conversion can be extremely high-stakes. Earnest helps students apply for financial services, and the steps along that path need to work with every deployment. Daniel Jimenez described the process: “Basically an integration test goes through the flow of any of our products and emulates a user interaction. There are tests to emulate every critical part of our flow, such as document signing. There’s a test to create a loan application, and check to see if the applicant was approved or not, whatever it’s expecting. Then a separate test that will go into an existing application, see that an offer has been made, and verify that you get a .pdf and that you can sign it.” One of the key benefits of Signadot is the amount of insight offered into the services under tests: > “People really like Signadot. They like being able to go in and see their service’s logs, and they find it really useful to go in and see the errors from their service during a test, catching problems before they emerge on Staging.” Daniel Jimenez · Infrastructure DevEx Engineer, Earnest While any organization can take the time for extensive testing on Staging, the key advantage of a Signadot-enabled process is that it’s the developers who are aware of a problem first, before their code reaches staging. This means that the person working on the feature has direct knowledge of the problem and can work on a fix directly, rather than waiting for later feedback from a software testing team. * * * ##### Front-end focused testing with Signadot Signadot isn’t just a tool for backend developers maintaining a microservice, it also helps developers working closer to the frontend to preview changes that involve one or multiple updated microservices. Kai Rillet, a software engineer at Earnest, led an effort to get more developers to preview changes using Signadot’s routing key right from their browser. By adding a header with a simple browser plugin, any developer can preview a sandbox based on their or another developer’s branch. This ability to preview frontend changes with all the normal dependencies hosted in a shared cluster means that developers can be more confident their changes interact with every other service correctly. * * * #### Giving testing power to the developers While the initial setup of the Signadot integration and end-to-end tests was completed by the developer experience team, the Earnest engineering org would like to move toward a model where developers manage these more complete tests directly. > “Developer Experience had a really easy time setting up the integration and migrating our current suite of integration tests to using Signadot. As an org we want to migrate to having the developers maintain these. Individual developers don’t mind writing unit tests, but it requires a cultural shift to have them write and maintain integration tests.” Early Ehlinger · Dev Experience Lead, Earnest However this breakdown of work develops, Signadot will remain a key component of the process for creating highly accurate test environments to test new changes. * * * #### Conclusion: meeting testing and reliability challenges Earnest’s integration of Signadot into their release process represents a significant leap forward in the realm of automated testing and reliable deployment strategies. By leveraging Signadot’s capabilities for creating isolated sandboxes and automating end-to-end and integration tests, Earnest has not only streamlined their development workflow but also enhanced the reliability and security of their releases. This approach has empowered developers with direct insights into their service’s performance during tests, enabling quicker identification and resolution of issues before they escalate to staging environments. The shift towards automated sandbox creation and the emphasis on testing critical user paths reflect a proactive approach to software development that prioritizes user experience and system reliability. Tools like Signadot are not just facilitating more efficient testing processes; they are reshaping the very dynamics of team collaboration and product development. #### More customer stories - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development August 27, 2026 ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% June 18, 2026 ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Bitso Source: https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/ Summary: How Bitso scaled branch-based development for 250+ engineers and 200+ microservices, with coding agents validating changes in isolated environments per PR. *Image: Bitso logo* Case study ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% Bitso replaced broken shared staging with Signadot's lightweight ephemeral environments, giving 250+ engineers and their coding agents isolated per-PR sandboxes across 200+ microservices, cutting change failure rate by 83% while more than doubling deploy frequency. [Bitso](https://bitso.com/), a leader in digital financial services in Latin America, spent years trying to solve the shared staging problem. With 200+ microservices, ~100 databases, and 250+ engineers all pushing changes into the same environments, something was always broken. Their homegrown [preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) solution required replicating the full stack per PR. This was already too expensive and too unreliable to sustain at scale, and then coding agents quickly made it impossible. The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) covers the tradeoffs between duplicating a stack per PR and sharing one instead. Signadot gave them a different model. Instead of duplicating environments, Bitso now forks only the changed services per PR and routes traffic through a single, stable cluster. This became the foundation of an internal initiative the team calls RISE (Robust Isolated Staging Environment), consolidating four fragile environments into one. As AI coding agents have driven a surge in the volume of changes flowing through the pipeline, this architecture has proven essential, giving Bitso the validation infrastructure to keep shipping safely at a pace that would have overwhelmed their previous setup. The [complete guide to software factories](https://www.signadot.com/software-factory/) uses Bitso’s results to show what happens when verification scales with agent volume. Open-to-Deploy62% faster12.1 days → 4.6 days Deploy Frequency2.1×664/week → 1,400/week Change Failure Rate−83%5.2% → 0.9% across all severities #### The Challenge: Shared Environments Buckling Under Agentic Scale Bitso’s platform is built on roughly 200 Java Spring Boot microservices communicating over gRPC, ~100 Postgres databases, 20+ Redis caches, and asynchronous messaging across Kafka, SQS, and Redis Streams. It is not feasible to run this stack locally. Engineers and their coding agents depend on remote environments to validate their work. 250+ engineersand their coding agents push changes Bitso's platform 200+microservices (Java, gRPC) ~100Postgres databases 20+Redis caches Async messagingKafka, SQS, Redis Streams As the team scaled to over 250 engineers and adopted agentic coding tools across the organization, that dependency became a bottleneck. Bitso maintained four separate non-production environments, each with varying degrees of stability, and none of them delivered the confidence engineers needed to ship with speed. The core problem was contention. Hundreds of engineers were pushing changes into shared environments where one faulty deployment could break the pipeline for everyone. The development environment was built around an unprotected branch that any engineer could push to directly. > “Everyone pushing changes onto a shared development branch… it was constantly broken. And if you break something, you break it for everyone using that environment.” Joe Horsnell · Principal Platform Engineer, Bitso Coding agents compounded the problem. Over the past year, the number of PRs flowing through the pipeline per developer increased by 60%, adding load to their testing infrastructure that was already bottlenecked. > “We spent a few years implementing our own preview environments solution, and it was always very expensive, both in time and cost, because we had been trying to replicate full environments per PR. We had a working solution, but in terms of speed and reliability, it was not great.” Marcus Tavares · Staff Software Engineer, Bitso The deployment process compounded the problem further. Engineers and agents would deploy changes to staging, where they mingled with other people’s changes, and then deploy a separate build to production. The code running in production was never identical to what had been validated in staging. Bitso’s vision was clear: collapse down to a single stable non-production environment where engineers and their agents could validate changes in isolation without stepping on each other. They needed infrastructure that could provide isolation at the scale of their architecture, and at the velocity of agentic development, without replicating it entirely. #### The Solution: Signadot and RISE Signadot’s lightweight ephemeral environments gave Bitso the next step forward: per-PR sandboxes that fork only the changed services while routing traffic through a stable shared cluster. Instead of replicating the entire environment, Signadot uses routing keys to direct traffic to only the modified services, with all other dependencies served by the stable cluster. This approach spins up in seconds rather than minutes and costs a fraction of full-environment duplication. Before · a full copy per PR PR #1200+ services, ~100 databases, 20+ caches PR #2200+ services, ~100 databases, 20+ caches × every open PR After · fork only what changed PR #1service + Neon branch PR #2service + Neon branch Stable environmentshared by every PR Before: minutes to spin up a full copy for every PR, at full cost. After: seconds to fork only the changed service, at a fraction of the cost. Because each PR gets its own isolated environment automatically, the system scales linearly with the increase in changes flowing through the pipeline. There is no shared environment to bottleneck, no queue of developers waiting to deploy to staging. This was critical for Bitso as agent-driven PRs increased the volume of changes far beyond what shared environments could absorb. Critically, this also unlocked artifact promotion. > “When you test, you are testing your changes in isolation, in your sandbox. When we deploy to production, it is precisely the same artifact that was tested in the sandbox, the container image, that we promote.” Joe Horsnell · Principal Platform Engineer, Bitso Bitso formalized this new architecture under an internal initiative called RISE: Robust Isolated Staging Environment. The architecture pairs Signadot for compute-layer isolation with [Neon](https://neon.com/) for data-layer isolation, providing instant Postgres database branches per sandbox for services that require schema changes or write-heavy testing. The combination gives engineers and agents production-like fidelity without resource contention. > “We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast.” Marcus Tavares · Staff Software Engineer, Bitso Signadot has become a core part of Bitso’s agent workflows. All affected services within a PR receive a sandbox automatically, and some services have either deterministic integration tests or custom agent skills that validate the behavior of those sandboxes. The result is a tighter feedback loop: agents generate code, Signadot provisions the isolated environment, and validation runs against it before the change is merged. Looking ahead, the team is exploring [Signadot’s Plans feature](https://www.signadot.com/blog/signadot-plans/), which provides agent-native validation workflows purpose-built for the inner loop, giving coding agents the ability to validate their own work inside sandboxes before a PR is published. #### Using Signadot for the Inner Loop with Coding Agents As Bitso adopted coding agents across the organization, a pattern emerged. The agents could generate large volumes of code and clear the cheap checks on their own (unit tests, lints, and mocked dependencies), but anything beyond that had to be handed back to engineers. In a system of 200+ interdependent microservices, the behavior that actually matters only shows up when a change runs against real services, real data, and real messaging. Mocks cannot represent that, so the agent’s work stalled at the point where a human had to take over and validate it in a shared environment. Signadot enabled Bitso to move more of that validation surface into the inner loop for their coding agents. Using Signadot’s [local development capabilities](https://www.signadot.com/solutions/local-development/), an engineer or an agent can run the service being changed locally and connect to the staging cluster for every other dependency, without the sidecar overhead or deployment restarts that came with Telepresence, which they were using previously. The agent gets the same routing keys, isolation model, and sandbox infrastructure that drives per-PR validation in CI, which means it can exercise its change against realistic dependencies as it writes it. Agent writes code Runs the service locally Validates against real dependenciesin a sandbox on the cluster iterate until it passes PR opens validated The effect is a tighter, more realistic feedback loop. Instead of generating code, clearing only the shallow checks, and handing off, agents can validate against production-like behavior before they open a PR. More of the work arrives already validated, and engineers spend less time confirming what the agent could have confirmed itself. #### The Results Since launching RISE and adopting Signadot alongside AI-assisted development and broader developer platform investments, Bitso has seen dramatic improvements across their key engineering velocity and quality metrics: Metric Before After Change Open-to-Deploy time 12.1 days **4.6 days** 62% faster Deploy frequency 664/week **1,400/week** 2.1× increase Change Failure Rate 5.2% **0.9%** 83% reduction These gains reflect the combined impact of Signadot, AI coding agents, and broader platform investments. But the relationship between these factors is reinforcing, not incidental. Coding agents drive velocity. Signadot provides the validation infrastructure that makes shipping at that velocity safe. The change failure rate improvement is especially striking. Even as deployment frequency more than doubled, incidents dropped from 5.2% to 0.9% across all severity levels (P4 through P1). Per-PR isolation means engineers and agents catch problems before code reaches the shared cluster, rather than discovering them in production. #### Conclusion Signadot gave Bitso isolated validation across a complex distributed architecture, at the velocity that coding agents demand. Agents and engineers can test changes against realistic dependencies in the inner loop before a PR exists, every PR gets its own sandbox automatically in CI, and the exact artifact that was validated is what ships to production. The result is a team that ships more than twice as often, with a fraction of the failures, on a foundation built to scale with the next wave of agentic development. > “Signadot has been key in our strategy to provide a faster, better, and more secure pipeline for our projects.” Marcus Tavares · Staff Software Engineer, Bitso To see how lightweight ephemeral environments can work for your team, [sign up for free](https://www.signadot.com/signup/) or [book a demo](https://www.signadot.com/schedule-a-call/). #### More customer stories - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development August 27, 2026 ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers* ### How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers July 15, 2025 ](https://www.signadot.com/case-studies/how-signadot-helped-wealthsimple-speed-up-microservice-testing-and-unblock-developers/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Laurel Source: https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/ Summary: How Laurel brings production-ready validation to AI-native development inside its own cloud, giving every pull request a production-like sandbox on its own EKS cluster and cutting change failure rate from 1.1% to 0.2% as agents author 76% of engineering PRs. *Image: Laurel logo* Case study ### How Laurel Brings Production‑Ready Validation to AI‑Native Development Laurel, an AI-native company where agents author 76% of engineering PRs, uses Signadot to give every pull request a production-like sandbox on its own EKS cluster, cutting change failure rate from 1.1% to 0.2% while keeping all code and customer data inside its own AWS account. [Laurel](https://www.laurel.ai/), a leading AI-native company building AI timekeeping and work intelligence software for professional services, faced a validation problem most engineering organizations will meet soon enough: coding agents were authoring an ever-larger share of its code, and every one of those changes needed production-grade validation before merge, without code or customer data leaving Laurel’s own cloud. Across a team of approximately 60 engineers, agents now author 76% of Laurel’s engineering pull requests, and the team merges about 9 PRs per engineer per week. With a rising volume of code changes moving through the pipeline, Laurel needed a dependable way to validate multi-service changes with production-like fidelity before they merged. Two constraints made the usual options hard. Laurel’s previous approach, using [Vercel](https://vercel.com/) preview environments, could not accurately replicate the full application stack and production infrastructure, which caused regular issues. And because many of Laurel’s clients are law and accounting firms, sending code or customer data to an external vendor’s cloud meant subprocessor reviews and compliance friction the team wanted to avoid. With Signadot, automation owned by Laurel’s infrastructure team stands up [per-PR sandboxes](https://www.signadot.com/docs/guides/set-up-pr-sandboxes) directly on its own [Amazon EKS](https://aws.amazon.com/eks/) cluster, managed with infrastructure as code and GitOps, and never leaving Laurel’s AWS account. Demand for this capability was high and the configuration work was light, so adoption at Laurel has been fast. Agent-Authored PRs76%up from 55% three months earlier Change Failure Rate−82%1.1% → 0.2% Throughput9 PRsmerged per engineer, per week, across ~60 engineers #### The Challenge: Validating a Flood of AI-Generated Changes, Without Leaving Their Own Cloud Laurel runs a Kubernetes stack on Amazon EKS: roughly 40 backend services (mostly Node.js, with some Python services for data enrichment) plus a single front-end monorepo of four services, which is its largest Signadot use case. Data lives in [MongoDB Atlas](https://www.mongodb.com/atlas) and Postgres on [AWS Aurora](https://aws.amazon.com/rds/aurora/), messaging runs on [Kafka](https://kafka.apache.org/), and observability is [OpenTelemetry](https://opentelemetry.io/)\-based. The team runs no service mesh. The demand for validation came from three directions at once: engineers wanting to spin up a change and preview it manually, a push to run automated end-to-end tests in CI, and, most importantly, a need to give coding agents a tight, realistic validation loop. As a frontier AI-native engineering team, Laurel was generating more change than a divergent preview setup could safely absorb. > “We were using Vercel for preview environments, and it was just different enough from our production, staging and development environments that it caused regular issues.” Naomi Klein · Lead Infrastructure Engineer, Laurel The deeper issue was where that validation could run. Serving law and accounting firms, Laurel keeps customer data inside its own AWS environment wherever possible, and every external vendor that touches that data is another subprocessor conversation the team would rather avoid. So the selection criteria were strict: run inside their own AWS, work with GitOps, and build directly off their own development Kubernetes cluster. Signadot met all three, which made it the only obvious choice. #### The Solution: Per-PR Sandboxes on Their Own EKS Cluster Instead of duplicating environments, Signadot forks only the changed services into a [sandbox](https://www.signadot.com/docs/concepts/sandbox), a lightweight ephemeral environment, and uses [routing keys](https://www.signadot.com/docs/guides/request-routing) to send the relevant traffic to them, while every other dependency is served by a stable shared environment, all running in Laurel’s own cluster. Sandboxes are provisioned per pull request, and everything runs on their EKS cluster, owned by the infrastructure team and provisioned using the same GitOps, infrastructure-as-code approach as the rest of their platform. Laurel's AWS account dev EKS cluster · GitOps managed PR #241 sandbox1 changed service PR #242 sandbox2 changed services \+ every PR routed via DevMesh Stable shared environmentevery other service and data store No code or customer data leaves Laurel's AWS account. Because Laurel hasn’t adopted a service mesh, Signadot’s [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) filled that gap directly. To coordinate changes that span multiple repositories, the team ties a routing key to each [Linear](https://linear.app/) ticket, so a single key can bring up matching sandboxes across the front-end monorepo and the backend services a change touches. > “We regularly create Signadot environments on every PR for almost every microservice. Our engineers can build and test changes across multiple services directly on top of our development environment, both locally and through our CI tooling.” Naomi Klein · Lead Infrastructure Engineer, Laurel #### Testing and Coding Agents on the Same Path Two of Laurel’s biggest use cases run on this foundation. On the testing side, Laurel rebuilt its E2E suite on Signadot, carrying over some parts of its existing [Playwright](https://playwright.dev/) logic, using Signadot’s [Smart Tests](https://www.signadot.com/product/smart-tests/) and routing keys. > “We have built a new suite for end-to-end testing on top of Signadot, work that would have been significantly more complex beforehand. Multiple engineers have discussed how Signadot helped significantly in uncovering bugs in their changes or in validating their features.” Naomi Klein · Lead Infrastructure Engineer, Laurel On the agentic side, Laurel is seeing an influx of fully AI-generated PRs alongside AI-assisted and human-written ones. Because every PR gets its own production-like sandbox automatically, all three go through the same validation path. The coding agents, [Claude Code](https://www.claude.com/product/claude-code), [Codex](https://openai.com/codex/), and others, wired in through Laurel’s own CLI tooling and AI skills, get the same isolated, realistic environment a human engineer would. That is the point: one place to validate a change, whoever or whatever wrote it. engineerengineer + AIcoding agentsClaude Code · Codexpull requestper-PR sandboxon Laurel's EKSmanual previewPlaywright e2e suiterun as Smart Testsmerge · 0.2% change failure rate Every change, human- or agent-written, gets the same per-PR sandbox and the same validation before merge. engineerengineer + AIcoding agents · Claude Code · Codexpull requestper-PR sandboxon Laurel's EKSmanual previewPlaywright e2e suite · Smart Testsmerge · 0.2% change failure rate Every change, human- or agent-written, gets the same per-PR sandbox and the same validation before merge. #### The Results: More AI-Written Code, Fewer Failures The qualitative wins landed first: easier end-to-end testing, a new test suite built on Signadot, and engineers catching bugs and validating features earlier than they could before. The numbers now back them up. For context on the scale Signadot now supports: over a recent 30-day window, Laurel’s team of roughly 60 engineers merged 2,873 pull requests. Setting aside automation traffic, about 20% of the total, that is roughly 2,300 engineering PRs: 538 per week, or about 9 merged PRs per engineer, per week. Metric Before After Change Agent-authored engineering PRs 55% **76%** +21 pts Change Failure Rate 1.1% **0.2%** 82% reduction The change failure rate is what makes the throughput credible. Over the same three months in which the agent-authored share of engineering PRs climbed from 55% to 76%, change failure rate did not just hold under the added volume. It fell from 1.1% to 0.2%. That was the whole point of the investment: push a higher volume of increasingly AI-generated changes without a matching rise in incidents. Laurel is shipping more AI-written code and breaking production less. #### Conclusion For an AI-native team carrying the compliance obligations of serving law and accounting firms, Signadot gave Laurel what the other options couldn’t: production-grade validation it could own and run entirely inside its own AWS and Kubernetes, scaling per PR, covering nearly every microservice in its most critical repos, and providing a single validation surface for both engineers and coding agents. > “Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made both human and AI-written code easier to validate and test.” Naomi Klein · Lead Infrastructure Engineer, Laurel To see how Signadot can help your team ship safer agent-written code at scale, [sign up for free](https://www.signadot.com/signup/) or [book a demo](https://www.signadot.com/schedule-a-call/). #### More customer stories - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% June 18, 2026 ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) - [ *Image: How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers* ### How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers July 15, 2025 ](https://www.signadot.com/case-studies/how-signadot-helped-wealthsimple-speed-up-microservice-testing-and-unblock-developers/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## ShareChat Source: https://www.signadot.com/case-studies/sharechat-chooses-signadot-giving-devs-high-quality-testing-feedback/ Summary: How ShareChat eliminated shared staging bottlenecks and enabled parallel development with Signadot sandboxes. *Image: ShareChat logo* Case study ### ShareChat Chooses Signadot, Giving Devs High Quality Testing Feedback Enabling Faster Time-to-Market ShareChat integrated Signadot into their CI pipeline to give 300+ developers independent sandboxes for 100+ microservices — dramatically reducing production issues and enabling faster, more frequent releases. #### The Challenge ShareChat is an Indian social media startup that caters to over 1 billion wireless network users. It offers a content consumption and sharing platform in 15 vernacular Indian languages. ShareChat has an engineering team of more than 300 members and manages over 100 microservices running on Kubernetes. They needed a way to test every feature independently before merging to production. They wanted a separate environment similar to staging to integration test their microservices and ensure that API contracts weren’t broken. Knowing the cost and development effort required to build a testing solution in-house, ShareChat was looking for a turnkey solution that could easily integrate into their existing architecture. * * * #### The Solution Signadot’s approach to ephemeral environments ultimately drove ShareChat to purchase Signadot as it allowed them to automate integration testing in their staging environment for 100+ microservices. They also didn’t have to worry about runaway infrastructure costs or operational burden. ShareChat integrated Signadot into their CI tool to automate integration testing for every PR. Developers were also able to write integration tests using their framework of choice running on Sandbox Environments. > “Signadot helps ShareChat to set up an extremely efficient rollout of an end-to-end integration test environment and allows developers to test for backward compatibility without having to worry about maintaining various versions of their services.” Harshal Vora · Sr. Engineering Manager, ShareChat * * * #### The Results **Independent testing for every PR.** ShareChat developers can now spin up a sandbox for every pull request, self-identify errors, and fix them before deploying to production — leading to more frequent deployments with fewer rollbacks and fewer production incidents. By adding Signadot to their deployment and review pipeline, ShareChat is able to spin up a Sandbox for every PR for developers to test their features independently. Developers are now able to self identify errors and fix them before deploying to production. Automated integration testing with Signadot has led to more frequent deployments while minimizing rollbacks and decreasing issues in production. To read or download the entire ShareChat case study, [click here](https://signadot-public.s3.us-west-2.amazonaws.com/sharechat-case-study.pdf). #### More customer stories - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development August 27, 2026 ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot July 29, 2026 ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% June 18, 2026 ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## All Customers Source: https://www.signadot.com/customers/ Summary: Customer success stories and video testimonials from engineering teams using Signadot. ### Case Studies See how engineering teams at leading companies use Signadot to accelerate development, reduce costs, and ship with confidence. - [ *Image: How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents* ### How Brex Uses Signadot to Scale Developer Testing Across 100s of Engineers and Coding Agents Signadot replaced Brex's namespace-based preview environments across 800+ microservices, cutting infrastructure costs by 99%, taking service previews from 30 to 60 minutes down to... ](https://www.signadot.com/case-studies/brex-uses-signadot-to-scale-developer-testing-across-100s-of-engineers/) - [ *Image: How Laurel Brings Production‑Ready Validation to AI‑Native Development* ### How Laurel Brings Production‑Ready Validation to AI‑Native Development Laurel, an AI-native company where agents author 76% of engineering PRs, uses Signadot to give every pull request a production-like sandbox on its own EKS cluster, cutting change... ](https://www.signadot.com/case-studies/how-laurel-brings-production-ready-validation-to-ai-native-development/) - [ *Image: How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot* ### How Miro's AI Team Builds and Tests Agentic Features at Scale with Signadot Miro's AI team adopted Signadot as the runtime validation layer for agentic development, testing agent tools and agent-generated code against real services. Adoption spread across... ](https://www.signadot.com/case-studies/how-miro-builds-and-tests-agentic-features-at-scale/) - [ *Image: How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83%* ### How Bitso Scaled Delivery with Coding Agents While Cutting Change Failure Rate by 83% Bitso replaced broken shared staging with Signadot's lightweight ephemeral environments, giving 250+ engineers and their coding agents isolated per-PR sandboxes across 200+... ](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) - [ *Image: How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers* ### How Signadot Helped Wealthsimple Speed Up Microservice Testing and Unblock Developers Wealthsimple cut weekly staging deployments nearly in half by adopting Signadot, eliminating staging conflicts and giving developers fast, isolated sandboxes that integrate with... ](https://www.signadot.com/case-studies/how-signadot-helped-wealthsimple-speed-up-microservice-testing-and-unblock-developers/) - [ *Image: How Arkestro Supercharged Developer Efficiency and Slashed Costs with Signadot* ### How Arkestro Supercharged Developer Efficiency and Slashed Costs with Signadot Arkestro's 40-person engineering team saves over $1M in annual developer bandwidth by running pre-merge tests in Signadot sandboxes — reducing regressions and freeing engineers to... ](https://www.signadot.com/case-studies/how-arkestro-supercharged-developer-efficiency-and-slashed-costs-with-signadot/) - [ *Image: Shifting Left with Signadot: How Earnest Empowered Developers for Early Testing* ### Shifting Left with Signadot: How Earnest Empowered Developers for Early Testing Earnest automated end-to-end testing with Signadot sandboxes, parallelizing test suites to cut cycle times from 60 minutes to 10 — and ensuring developers catch issues before code... ](https://www.signadot.com/case-studies/how-earnest-empowers-developers-for-early-testing/) - [ *Image: Yellow.ai and Signadot: Pioneering Parallel Feature Development for Faster Releases* ### Yellow.ai and Signadot: Pioneering Parallel Feature Development for Faster Releases Yellow.ai improved developer productivity 2–3x by replacing per-developer virtual clusters with Signadot sandboxes that deploy only what changed, letting 100+ developers test on... ](https://www.signadot.com/case-studies/how-yellow-ai-enabled-parallel-feature-development-using-signadot/) - [ *Image: How Developers and Coding Agents at DoorDash Get 10x Faster Feedback on Code Changes* ### How Developers and Coding Agents at DoorDash Get 10x Faster Feedback on Code Changes DoorDash eliminated their staging environment and gave developers the ability to test directly against production from their laptops — cutting feedback time from 30 minutes to 60... ](https://www.signadot.com/case-studies/how-developers-at-doordash-get-10x-faster-feedback/) - [ *Image: Levo Saves 90% of Time on Integration Tests, Improves Developer Efficiency with Signadot* ### Levo Saves 90% of Time on Integration Tests, Improves Developer Efficiency with Signadot Levo achieved 90% time savings on integration testing by running automated end-to-end tests in Signadot sandboxes before every merge — eliminating post-merge bugs and rollbacks. ](https://www.signadot.com/case-studies/levo-saves-time-on-integration-tests-improves-developer-efficiency-with-signadot/) - [ *Image: ShareChat Chooses Signadot, Giving Devs High Quality Testing Feedback Enabling Faster Time-to-Market* ### ShareChat Chooses Signadot, Giving Devs High Quality Testing Feedback Enabling Faster Time-to-Market ShareChat integrated Signadot into their CI pipeline to give 300+ developers independent sandboxes for 100+ microservices — dramatically reducing production issues and enabling... ](https://www.signadot.com/case-studies/sharechat-chooses-signadot-giving-devs-high-quality-testing-feedback/) #### And many more Signadot is trusted by engineering teams at companies of all sizes - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Integrations Source: https://www.signadot.com/integrations/ Summary: Signadot integrations for service meshes, CI/CD, testing frameworks, coding agents, and observability tools. Integrations ### All your critical tools and infrastructure in one place Work with the tools your team already uses: GitHub, GitLab, Jenkins and more. - *Image: Apollo GraphQL logo* ### Apollo GraphQL API Frameworks Enables safe testing of federated GraphQL schema changes in isolated, per-Pull Request environments. By combining Apollo's Variants with Signadot's Sandboxes, developers can validate subgraphs and router changes without affecting the production supergraph. [Learn more about Apollo GraphQL](https://www.signadot.com/docs/tutorials/apollo-graphql-schema-testing) - *Image: Backstage logo* ### Backstage Internal Developer Portals A plugin to bring your ephemeral test environments directly into Backstage. Get a unified view of both production and sandboxes, enabling your developers to manage and view environments in one place. [Learn more about Backstage](https://github.com/signadot/backstage-plugin) - *Image: Bitbucket logo* ### Bitbucket CI/CD Tools Use Bitbucket pipelines to automate the creation and testing of sandboxes for your commits and pull requests, securing your API key with environment variables. [Learn more about Bitbucket](https://www.signadot.com/docs/guides/integrate-ci/bitbucket) - *Image: Claude Code logo* ### Claude Code AI Coding Use Signadot's MCP server with Claude Code to manage sandboxes, run tests, and interact with your ephemeral environments directly from your AI coding assistant. [Learn more about Claude Code](https://www.signadot.com/docs/integrations/mcp/claude-code) - *Image: Codex logo* ### Codex AI Coding Integrate Signadot's MCP server with OpenAI Codex to automate sandbox creation, run tests, and manage ephemeral environments from your AI development workflow. [Learn more about Codex](https://www.signadot.com/docs/integrations/mcp/other-clients) - *Image: Cursor logo* ### Cursor AI Coding Connect Signadot's MCP server to Cursor and let your AI coding assistant create sandboxes, check test results, and manage ephemeral environments without leaving your editor. [Learn more about Cursor](https://www.signadot.com/docs/integrations/mcp/cursor) - *Image: Cypress logo* ### Cypress Test Frameworks Integrate with your CI/CD pipelines to run Cypress tests in ephemeral environments, using a routing key to ensure each test runs against its own isolated sandbox. [Learn more about Cypress](https://www.signadot.com/docs/guides/testing-frameworks/cypress-integration) - *Image: GitHub Actions logo* ### GitHub Actions CI/CD Tools Integrate with your CI workflow to create and test sandboxes directly from pull requests, with automatic status updates and cleanup. [Learn more about GitHub Actions](https://www.signadot.com/docs/guides/integrate-ci/github) - *Image: GitLab logo* ### GitLab CI/CD Tools Easily set up CI/CD pipelines to manage sandboxes for your merge requests, using the Signadot CLI to apply, test, and delete environments. [Learn more about GitLab](https://www.signadot.com/docs/guides/integrate-ci/gitlab) - *Image: Google Pub/Sub logo* ### Google Pub/Sub Message Queues Perform isolated end-to-end testing for your asynchronous Google Pub/Sub flows. Sandbox-specific routing keys are used to ensure that messages are only processed by the correct environment. [Learn more about Google Pub/Sub](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - *Image: Istio logo* ### Istio Infrastructure Route each ephemeral environment's requests through Istio ambient mode, providing a cleaner and more efficient way to test your applications without the "sidecar tax." [Learn more about Istio](https://www.signadot.com/blog/signadot-operator-cli-hit-1-0-now-with-istio-ambient-support-and-enhanced-local-dev/) - *Image: Jenkins logo* ### Jenkins CI/CD Tools Configure your Jenkins free-style projects to manage sandboxes using a simple build script, securely accessing the API key via Jenkins credentials. [Learn more about Jenkins](https://www.signadot.com/docs/guides/integrate-ci/jenkins) - *Image: Kafka logo* ### Kafka Message Queues Enable isolated end-to-end testing for your Kafka-based microservices. Use routing keys and selective message consumption to test changes in sandboxes without interfering with other environments. [Learn more about Kafka](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) - *Image: Kubernetes logo* ### Kubernetes Infrastructure Seamlessly integrate with your Kubernetes clusters to establish a secure connection, enabling authenticated previews and isolated ephemeral environments. [Learn more about Kubernetes](https://www.signadot.com/docs/guides/admin/security#kubernetes) - *Image: Linkerd logo* ### Linkerd Infrastructure Leverage DevMesh to set up sandbox routing for your Linkerd-powered applications, creating high-fidelity testing environments with ease. [Learn more about Linkerd](https://www.signadot.com/docs/installation/signadot-operator#linkerd) - *Image: MySQL logo* ### MySQL Databases Spin up temporary MySQL databases linked to the lifecycle of Sandboxes using Resource Plugins. [Learn more about MySQL](https://github.com/signadot/plugins) - *Image: Neon logo* ### Neon Databases Spin up temporary Neon databases linked to the lifecycle of Sandboxes using Resource Plugins. [Learn more about Neon](https://github.com/signadot/plugins) - *Image: Open Telemetry logo* ### Open Telemetry Distributed Tracing Integrate with the OpenTelemetry standard for automatic context propagation. Use the sd-routing-key within baggage and tracestate headers to ensure your test traffic is routed correctly. [Learn more about Open Telemetry](https://www.signadot.com/docs/guides/set-up-context-propagation) - *Image: Playwright logo* ### Playwright Test Frameworks Run your end-to-end tests in isolated sandboxes. Configure Playwright to use a routing key, enabling you to test changes on a per-pull request basis. [Learn more about Playwright](https://www.signadot.com/docs/guides/testing-frameworks/playwright-integration) - *Image: Postgres logo* ### Postgres Databases Spin up temporary PostgreSQL databases linked to the lifecycle of Sandboxes using Resource Plugins. [Learn more about Postgres](https://github.com/signadot/plugins/tree/main/postgres-vault) - *Image: Postman logo* ### Postman Test Frameworks Use Postman's CLI, Newman, to run API tests against sandboxes. By passing a routing key, you can automatically route test traffic to the correct isolated environment. [Learn more about Postman](https://www.signadot.com/docs/guides/testing-frameworks/postman-integration) - *Image: RabbitMQ logo* ### RabbitMQ Message Queues Isolate RabbitMQ consumers for safe, parallel testing. By creating temporary, sandbox-specific queues and exchanges, you can test changes in an isolated environment without the overhead of duplicating your infrastructure. [Learn more about RabbitMQ](https://www.signadot.com/docs/tutorials/testing-rabbitmq-microservices) - *Image: SQS logo* ### SQS Message Queues Test your SQS-based microservices by using a dedicated SQS queue for each sandbox. This approach, which often leverages SNS for message fan-out, ensures proper isolation and simplifies testing. [Learn more about SQS](https://github.com/signadot/plugins/tree/main/amazon-sqs) - *Image: Temporal logo* ### Temporal Workflow Systems Test Temporal services and workflows in isolation. This integration enables you to safely validate new worker code in a sandbox environment without needing to redeploy full environments. [Learn more about Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers) - *Image: Windsurf logo* ### Windsurf AI Coding Connect Signadot's MCP server to Windsurf and manage sandboxes, run tests, and interact with ephemeral environments from within your AI coding assistant. [Learn more about Windsurf](https://www.signadot.com/docs/integrations/mcp/other-clients) - *Image: WunderGraph logo* ### WunderGraph API Frameworks Allows teams to safely test GraphQL schema changes in isolated environments. By combining WunderGraph's feature flags with Signadot's sandboxes, developers can validate schema updates and new subgraphs without disrupting the main API. [Learn more about WunderGraph](https://www.signadot.com/docs/tutorials/wundergraph-schema-testing) - *Image: Zed logo* ### Zed AI Coding Use Signadot's MCP server with Zed to manage sandboxes, run tests, and interact with your ephemeral environments directly from your AI-powered editor. [Learn more about Zed](https://www.signadot.com/docs/integrations/mcp/other-clients) No integrations found. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Comparisons ## The Ultimate Guide to Telepresence Alternatives in 2026 Source: https://www.signadot.com/blog/the-ultimate-guide-to-telepresence-alternatives-in-2025/ Summary: mirrord vs Telepresence vs Signadot in a 2026 table, plus Gefyra, retired tools, Telepresence pricing and status after Gravitee acquired Ambassador, what coding agents change, and how to choose. ### The Ultimate Guide to Telepresence Alternatives in 2026 mirrord, Gefyra and Signadot compared as Telepresence alternatives, with pricing and support status, what coding agents change, and how to move off Telepresence. Author Arjun Iyer Published August 27, 2025 Updated October 10, 2026 Reading time 15 min Topics [Tutorials/Guides](https://www.signadot.com/blog/category/tutorials-guides/) The main Telepresence alternatives in 2026 are mirrord, which runs a local process as if it were inside the cluster, Gefyra, which does the same for a local Docker container, and Signadot, which gives local development, every pull request and every coding agent one model: ephemeral environments on a shared cluster. mirrord and Telepresence connect one machine or process to a cluster. Signadot adds environments for every pull request and every coding agent, with databases, queues and other dependencies included. Telepresence itself is still released as an open source project. What changed is that no company stands behind it any more, which is why many teams are looking again. For the wider field, including environment platforms, see the [comparison of Kubernetes development tools](https://www.signadot.com/blog/telepresence-mirrord-okteto-alternatives/). For single competitors in depth, see the [Okteto alternatives](https://www.signadot.com/comparison/okteto/) and [mirrord alternatives](https://www.signadot.com/comparison/mirrord/) comparisons. #### Telepresence alternatives at a glance Telepresence alternatives in 2026 Tool What it does Best for Price Status [Telepresence](https://telepresence.io/) Connects your workstation to the cluster and routes a workload's traffic to your machine One developer debugging a service against a live cluster Free, open source Active, CNCF Sandbox, v2.32 (Oct 2026). No commercial support [mirrord](https://metalbear.com/mirrord/) Runs a local process as if it were in a pod by hooking its network and file calls Low-setup local development, and shared clusters on paid tiers Free for individuals, Team $50 per seat per month Active, near-daily releases [Gefyra](https://gefyra.dev/) Runs a local Docker container on the cluster network over WireGuard Teams that already develop in containers Free, open source Active, v2.5.6 (Oct 2026) [Okteto](https://www.signadot.com/comparison/okteto/) Syncs your code into a development container in the cluster, one namespace per developer Cloud development environments Free for 5 seats, paid tiers on request Active. Free hosted tier ended Jan 2024 [Signadot](https://www.signadot.com/) Ephemeral environments on a shared cluster for every pull request, developer and coding agent, with local development as one way to use them Pull request environments, data and dependencies, coding agents Free Starter plan, usage-based paid plans Active Tilt and Skaffold also show up on lists of alternatives. They rebuild and redeploy your code to a cluster on every change rather than connecting a local process, so they suit a different workflow. Both are active. #### What Telepresence does in Kubernetes, and what changed Telepresence connects a developer’s machine to a Kubernetes cluster. A service running on your laptop can call cluster services by their DNS names, and traffic meant for a workload in the cluster can be sent to your laptop instead. It is a [CNCF Sandbox project](https://www.cncf.io/projects/telepresence/), accepted in 2018 and originally created by Ambassador Labs. ##### How Telepresence works According to the [Telepresence architecture docs](https://telepresence.io/docs/concepts/architecture), three parts work together: - **A Traffic Manager** runs once in the cluster, usually installed by an administrator. - **A traffic agent** attaches to the workload you work on, injected as a sidecar or, since version 2.30, through an optional node agent that does not restart pods. - **Daemons on your machine** create a virtual network interface and DNS resolver. The root daemon asks for administrator rights unless you install from a package or use the `--docker` mode. It has [four ways to attach](https://telepresence.io/docs/concepts/attachments) to a workload: `intercept` sends the workload’s traffic to your machine while the remote container keeps running, `replace` swaps the container out entirely, `wiretap` sends you a read-only copy of the traffic, and `ingest` gives you the workload’s environment and volumes without any traffic. Since version 2.25, open source Telepresence can filter an intercept by HTTP header or path, so several developers can work on the same service at once. Two friction points remain: the virtual network interface can conflict with corporate VPNs, and every developer needs their own credentials for the cluster. ##### Telepresence pricing and status in 2026 Telepresence is free and open source under the Apache 2.0 license, and the project is busy. Version 2.32.2 [shipped on October 8, 2026](https://github.com/telepresenceio/telepresence/releases), with releases every two to four weeks through the year. The commercial story is what changed. Ambassador moved its paid Telepresence features into its Blackbird platform in early 2025, then [Gravitee acquired Ambassador in July 2025](https://www.gravitee.io/blog/ambassador-labs-gravitee-what-our-acquisition-means-for-you). As of October 2026, the old getambassador.io product pages redirect to Gravitee, Gravitee does not offer Blackbird or a paid Telepresence, and no company sells support or an SLA for it. The project is kept up mostly by one maintainer, who wrote about 90% of the commits since April, and its only sponsor [ended its sponsorship](https://github.com/telepresenceio/telepresence/discussions/4334) on October 7, 2026. Help comes from GitHub and the CNCF Slack. Teams that need a vendor behind their tooling are the ones looking hardest at alternatives. #### mirrord vs Telepresence vs Signadot These are the three tools buyers compare most. The table covers the questions that usually decide it. mirrord vs Telepresence vs Signadot (2026) Telepresence mirrord Signadot Primary use case Connect a developer's local process to a remote cluster Run a local process as if it were inside the cluster, plus shared-cluster development on paid tiers An ephemeral environment for every pull request, developer and coding agent on a shared cluster How isolation works Intercepts a workload's traffic and sends it to your machine. Header filters give each developer only their own requests Mirrors traffic by default, or takes it over with HTTP filters. Sharing one service among developers needs a paid plan Duplicates only what a change needs and links it into the shared environment through request routing, message routing or configuration Environment per pull request No Preview environments run the changed services as isolated pods (Enterprise, beta) Yes, created from CI in seconds Databases, queues and workflow systems None built in. `replace` makes your machine the only consumer of a queue Database branching and queue splitting for Kafka, SQS, RabbitMQ, Pub/Sub, NATS, Temporal and more (Team and above) Resource plugins or branching providers for databases, and a routing API and libraries for Kafka, RabbitMQ, NATS, SQS, Pub/Sub and Temporal task queues Workloads and resources outside Kubernetes Your own machine only Your own machine. Database branches can be copied from managed databases such as RDS A service on your machine, and resource plugins that create cloud resources such as managed databases, queues and buckets with each environment Coding agents Built-in MCP server for Claude Desktop, Cursor and VS Code Agent skills for Claude Code, Codex and OpenCode. Cloud agents on Enterprise MCP server and agent skills. Each agent run gets its own ephemeral environment Deployment model Open source. Traffic Manager in your cluster and daemons on each machine, no vendor service Open source CLI. Paid operator in your cluster, licensed against MetalBear's servers on Team Operator in your cluster and a thin SaaS control plane that receives only resource metadata by default. SOC 2 Type II Local development Yes, the core use case Yes, the core use case Yes, on the same model as pull request environments. Code on your laptop, or in a coding agent's cloud sandbox, joins the environment Price Free Free for individuals, Team $50 per seat per month, Enterprise on request Free Starter plan, usage-based Business plan, Enterprise Commercial support None. A community project, kept up mostly by one maintainer Sold by MetalBear on the Team and Enterprise plans Sold by Signadot The short version: mirrord and Telepresence connect one process or machine to a cluster. Signadot gives local development, pull requests and coding agents one model, the ephemeral environment, with databases, queues and other dependencies in it, so the environment you develop in is the one your pull request and tests use. ##### mirrord vs Telepresence The two tools solve the same problem in different ways. - **Mechanism.** Telepresence creates a virtual network interface on your machine, so everything on the machine can reach the cluster. [mirrord loads a library into one process](https://metalbear.com/mirrord/docs/reference/architecture) and hooks its network and file calls, so only that process is connected. That is why mirrord usually needs less setup and conflicts less with corporate VPNs. - **Default behavior.** mirrord mirrors incoming traffic by default and only takes it over when you ask. Telepresence intercepts by default, and added a read-only `wiretap` mode in 2025, so this is no longer a difference in capability, only in defaults. - **Sharing a cluster.** Both can filter traffic by HTTP header so several developers work on one service. In Telepresence this is free. In mirrord, concurrent use of one service needs the Team plan. - **Data and queues.** mirrord’s paid plans add [database branching](https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching) and [queue splitting](https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting). Telepresence has neither. - **Beyond the laptop.** mirrord offers CI runs and preview environments on Enterprise. Telepresence stays a developer tool. - **Price.** Telepresence is free. mirrord is free for individuals and $50 per seat per month for teams. So Telepresence vs mirrord comes down to setup and budget. If you want a free tool and are comfortable with the networking setup, Telepresence is fine. If you want the lowest-friction local workflow and will pay for team features, mirrord is the stronger choice. If you are looking for a mirrord alternative because you need an environment for every pull request rather than a faster local loop, the [mirrord alternative comparison](https://www.signadot.com/comparison/mirrord/) covers where each tool fits, and [Signadot vs Telepresence](https://www.signadot.com/comparison/telepresence/) does the same for Telepresence. #### The other local-to-cluster tools ##### Gefyra [Gefyra](https://gefyra.dev/) runs your code in a local Docker container attached to the cluster network over WireGuard. Your container can call cluster services as if it were a pod. To receive cluster traffic, Gefyra’s bridge [patches the target workload](https://gefyra.dev/en/information/run-vs-bridge) and routes requests to your container. Since version 2.4, personal bridges match on HTTP headers or paths, so developers can share a workload. Gefyra is free, open source and maintained, with version 2.5.6 released on October 6, 2026. The catch is the container requirement: if you prefer to run processes directly on your machine, mirrord or Telepresence fit better. Its Docker Desktop extension still runs the older Gefyra 1. ##### Retired and slowing tools Some tools that appear on older lists of Telepresence alternatives are gone or fading: - **Bridge to Kubernetes** from Microsoft was [retired on April 30, 2025](https://learn.microsoft.com/en-us/visualstudio/bridge/overview-bridge-to-kubernetes). Microsoft named mirrord and Telepresence as replacements. - **ksync** is archived on GitHub, and **kt-connect** has not had a release since 2022. - **Okteto Cloud**, the free hosted tier, closed on January 15, 2024. Okteto itself is active as a self-hosted platform, covered in the [Okteto alternatives comparison](https://www.signadot.com/comparison/okteto/). - **DevSpace** is still a CNCF Sandbox project, but its last stable release was in April 2026 and activity has slowed. Test your next change against real dependencies Signadot spins up isolated sandboxes on the Kubernetes cluster you already run, so every change is validated against real services before it merges. The free tier is open to every developer. [Start free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) #### Signadot: ephemeral environments for every change Signadot is not a tunnel. It gives every pull request, developer and coding agent an ephemeral environment on a shared Kubernetes cluster, and it covers the local workflow the tools above are known for as one way of using that environment. **How it works.** The stable version of every service runs once in the cluster. For each change, Signadot duplicates only the services and components that need isolating, usually just the changed services, and plugs them into the rest of the shared environment. It links them in three ways: request routing for synchronous traffic, message routing for asynchronous and batch work, and configuration changes that point a service at a new version. How much each change duplicates is up to you, from one service to the whole stack. Because so little is copied, environments start in seconds. Request routing runs through the mesh you already have. On Istio, Signadot updates your VirtualService rules and Istio’s own sidecars or ambient mode do the routing, with no extra sidecar. It also works with the Gateway API, and with Linkerd or no mesh a lightweight DevMesh sidecar routes requests. **Data and dependencies.** This is where Signadot differs most from the local-to-cluster tools. - **Databases.** A change can share a database, or get its own through a [resource plugin](https://www.signadot.com/docs/concepts/resources) or a branching provider such as [Neon](https://www.signadot.com/docs/guides/set-up-data-isolation). The plugin creates the resource with the environment and deletes it afterwards. - **Message queues.** Kafka, RabbitMQ, NATS, SQS and Pub/Sub stay shared. Producers tag messages with the environment’s routing key, and a [routing API and libraries](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) tell each consumer which messages are its own. - **Workflow engines.** The same approach works for [Temporal workers on a shared task queue](https://www.signadot.com/docs/tutorials/testing-temporal-workers). - **Resources outside Kubernetes.** Resource plugins can create and delete managed databases, queues, buckets and other cloud resources with each environment, written with Helm, Terraform or a cloud CLI. **Local development.** With [local development](https://www.signadot.com/docs/guides/use-cases/set-up-local-testing), you run the service you are changing on your machine. Requests in the cluster that carry your environment’s routing key reach your local process, and your process reaches cluster services, databases and queues by their usual Kubernetes DNS names. Many developers share one cluster without stepping on each other. **Pull requests and tests.** CI [creates an environment for every pull request](https://www.signadot.com/docs/guides/set-up-pr-sandboxes) in seconds. Your existing tests run against it, from CI or inside the cluster as [Jobs](https://www.signadot.com/docs/concepts/smart-tests-and-jobs), and [RouteGroups](https://www.signadot.com/docs/reference/route-groups) combine environments when a feature spans several services and pull requests. **Deployment model.** Signadot is hybrid. The operator runs in your cluster, with a thin SaaS control plane. By default the control plane receives only resource metadata, such as workload names, namespaces, image references and status, never source code, image contents or application data. Signadot runs in pre-production, so it does not see production PII. Opt-in features such as Smart Diff traffic capture and Job logs send more, encrypted at rest. Signadot is SOC 2 Type II, and the [data and integrations page](https://www.signadot.com/docs/guides/admin/security/data-integrations) has the details. Other environment platforms take a different route. Release and Bunnyshell give each environment a copy of the application, and so does Okteto by default, though its Divert feature can share services from another namespace. Full copies work well for smaller stacks and get expensive as the number of services grows. The [Okteto alternatives](https://www.signadot.com/comparison/okteto/) and [Release (ReleaseHub) alternative](https://www.signadot.com/comparison/release/) pages compare those approaches. #### What coding agents change Coding agents change who needs an environment, and how many need one at once. The [CircleCI 2026 State of Software Delivery report](https://www.signadot.com/blog/circleci-2026-report-agent-validation-gap/) found that average throughput grew 59% year over year, while roughly 30% of merge attempts now fail and median recovery takes 72 minutes. A 50-engineer team built for 100 to 150 pull requests a day is now [looking at 1,000](https://www.signadot.com/blog/the-agent-pr-flood-is-here-if-you-run-istio-youre-halfway-to-solving-it/). Local-to-cluster tools were designed around one person at one workstation. That shows up in three ways once agents do part of the work. 1. **Agents run where there is no laptop.** Claude Code, Codex, Cursor and GitHub Copilot’s coding agent increasingly run in cloud sessions and CI runners. A tool whose main workflow is a daemon or a process on a developer’s machine has to be adapted to run there. 2. **Agents run in parallel.** One developer may have four agent sessions open, each changing a different service or the same service in different ways. Each needs its own isolated copy of the change, its own data where the change touches a schema, and its own routing so that one agent’s test traffic never reaches another agent’s code. 3. **Agents need a closed loop.** An agent can only fix what it can observe. It needs to create an environment, run the tests or requests that exercise its change, read the result and try again, without asking a human to open a tunnel. Local development is turning into agentic development: most code is now written by an agent, on a laptop or in the cloud. All three tools now have something for agents. Telepresence has a built-in MCP server, and mirrord ships agent skills and offers cloud agents on Enterprise. Both still center on connecting one machine to a cluster. Signadot gives each agent run its own ephemeral environment through its [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills), with only the agent’s changes duplicated, so dozens of agents can test against the same cluster at once. That includes background agents running in cloud sandboxes, as in our [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial). The agent and your tests do the checking. Signadot provides the environment they check in. The guide to [validating AI-generated code on Kubernetes](https://www.signadot.com/validate-ai-generated-code-kubernetes/) covers the full loop. #### How to choose **Stay on Telepresence when** it already works for your team, you want a free open source tool, community support is enough, and your need is one developer at a time debugging a service against a cluster. **Choose mirrord when** one developer at a time needs a quick link from a laptop process to a cluster. Pay for the Team plan if several developers share a cluster or need database branching and queue splitting. **Choose Gefyra when** your team already develops inside Docker containers and wants a free, container-based workflow. **Choose Signadot when** you need an environment for every pull request and coding agent, your changes touch databases, queues or workflow engines such as Temporal, or many developers and agents must test on one cluster at the same time. Local development, pull requests and coding agents share one model, the ephemeral environment, so Signadot replaces a local-to-cluster tool rather than sitting beside it. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). #### Moving off Telepresence Signadot’s routing relies on the same thing Telepresence’s header-based intercepts need: each service passes a header such as W3C `baggage` on to the services it calls. Teams that already use personal intercepts usually have that in place. The [technical guide to migrating from Telepresence to Signadot](https://www.signadot.com/blog/how-to-migrate-from-telepresence-to-signadot-a-technical-guide/) walks through the one-time setup, moving the local workflow over, adding pull request environments, and how Telepresence concepts map to Signadot ones. #### Frequently asked questions What is the best alternative to Telepresence? The like-for-like swaps are mirrord, which runs a local process as if it were inside the cluster without a VPN-style tunnel, and Gefyra, which does the same for a local Docker container. Signadot goes further with one model, the ephemeral environment, for local development, every pull request and every coding agent, with databases and queues isolated per change. The environment you or your agent develop in is the one your pull request and tests use. mirrord vs Telepresence: which is better? Both connect a local process to a Kubernetes cluster and both are actively maintained. mirrord is usually quicker to set up because it hooks the process's network and file calls instead of creating a network interface, and its paid tiers add database branching, queue splitting and preview environments. Telepresence is free and open source, a CNCF Sandbox project, with header-based personal intercepts and a built-in MCP server. Pick by setup friction and whether you will pay for team features. What are the best alternatives to mirrord? Telepresence and Gefyra are the like-for-like alternatives to mirrord: all three connect a locally running process or container to a remote Kubernetes cluster. Signadot is the alternative when you need more than the inner loop: an ephemeral environment for every pull request and coding agent, with databases, message queues and Temporal workflows isolated per change. Signadot's mirrord comparison page covers the two side by side. Is Telepresence still maintained? The code is, but no company supports it. Version 2.32.2 shipped on October 8, 2026, and it remains a CNCF Sandbox project. Ambassador Labs moved its paid Telepresence features into Blackbird, Gravitee bought Ambassador in July 2025, and today no one sells support or an SLA. Most recent work comes from one maintainer, and the project's only sponsor ended its sponsorship in October 2026. How much does Telepresence cost? Telepresence is free and open source under the Apache 2.0 license. As of October 2026 no paid Telepresence tier is offered: the commercial features moved into Ambassador's Blackbird platform, and Gravitee, which acquired Ambassador in July 2025, does not market either one. For comparison, mirrord's Team plan costs $50 per seat per month, and Signadot has a free Starter plan and usage-based paid plans. What does Telepresence do in Kubernetes? Telepresence connects your workstation to a Kubernetes cluster so a service running on your laptop can call cluster services by their DNS names and receive traffic meant for a workload in the cluster. A Traffic Manager runs in the cluster, a traffic agent attaches to the workload, and daemons on your machine create a virtual network interface. Modes include intercept, replace, wiretap and ingest. Can coding agents use Telepresence or mirrord? Partly. Telepresence has a built-in MCP server that Claude Desktop, Cursor and VS Code can use to drive it, and mirrord ships agent skills for Claude Code, Codex and OpenCode. Both still center on one machine connected to a cluster. Agents running many tasks in parallel, on laptops or as background agents in the cloud, need an environment per task, which Signadot creates through its MCP server and agent skills, with only the agent's changes duplicated. Does Signadot support local development like Telepresence does? Yes. With Signadot you run the service you are changing on your machine inside an ephemeral environment. Requests in the cluster that carry your environment's routing key reach your local process, and your process reaches the cluster's real services, databases and queues by their Kubernetes DNS names. Many developers share the same cluster without interfering, and the same environment then serves pull request previews and coding agents. #### Related Posts - [ *Image: Signadot vs. Telepresence: A Platform Upgrade for Enterprise Teams* Tutorials/Guides ### Signadot vs. Telepresence: A Platform Upgrade for Enterprise Teams August 27, 2025 ](https://www.signadot.com/blog/signadot-vs-telepresence-a-platform-upgrade-for-enterprise-teams/) - [ *Image: How to Run a Local Kubernetes Cluster with Minikube* Tutorials/Guides ### How to Run a Local Kubernetes Cluster with Minikube April 23, 2023 ](https://www.signadot.com/blog/beginner-minikube-guide/) - [ *Image: What’s the Difference Between an API Gateway and a Service Mesh?* Tutorials/Guides ### What’s the Difference Between an API Gateway and a Service Mesh? June 26, 2023 ](https://www.signadot.com/blog/service-mesh-vs-api-gateway-on-kubernetes/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## mirrord Alternative for Local Dev and PR Environments (Signadot vs mirrord) Source: https://www.signadot.com/comparison/mirrord/ Summary: mirrord connects a developer's local process to a remote cluster; Signadot gives local development, every PR and every coding agent (on a laptop or in the cloud) one model, the ephemeral environment, with databases, message queues and Temporal. Six key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs mirrord ### The mirrord alternative for local dev and PR environments mirrord connects a program on your laptop to a remote cluster. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs mirrord: six key differences Both let you test against real services instead of mocks, but they are built on different foundations. mirrord hooks into a program on your laptop, and not every program can be hooked. Signadot is built on the ephemeral environment, one model for local development, PRs, tests and agents, whether a person or a coding agent writes the code. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares mirrord previews with the other platforms. Feature Signadot mirrord Foundation One model, the ephemeral environment, for local dev, PRs and coding agents. Hooks into local programs. Every laptop needs cluster access. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. Skills for agents on a laptop. Cloud agents on Enterprise only. No test runtime. PR environments The core of the product. Hundreds share one cluster. An Enterprise add-on, with data isolation still in Alpha. Databases, queues and more Any database or queue. Resource plugins add anything else. Added one tool at a time. No plugin framework. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. Its own agent, beside the mesh. Previews handle only Istio and Linkerd. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. None. You run tests on your own. #### When to choose mirrord A fit when one developer needs a quick link from a laptop to a cluster. - You only need to run one program against a cluster - You do not need an environment per PR or per agent - You need a fully air-gapped install #### When to choose Signadot The choice when local dev, PRs and coding agents should share one model. - Local dev and PR environments on one model - An ephemeral environment for every PR and coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Signadot runs your tests too. mirrord does not. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions What is the best mirrord alternative for local dev and PR environments? Signadot. Local development, every pull request and every coding agent get their own ephemeral environment in your cluster. Only the changed services are copied, so hundreds of environments run at once, each with its own data, and your tests and agents run against them before merge. mirrord vs Signadot: what's the difference? They are built on different foundations. mirrord hooks into a program on your laptop and connects it to the cluster. Signadot is built on the ephemeral environment, one model for local development, PRs, tests and coding agents, so the environment you or your agent develop in is the one your PR and tests use. Can I use Signadot for local development? Yes, on the same model as PR environments. Run the service you are changing on your laptop with [`signadot local connect`](https://www.signadot.com/docs/tutorials/quickstart/local-development), or let your coding agent do it, on your laptop or in the cloud. Requests for your change reach it and nobody else's do, and the same ephemeral environment then serves your PR and tests. Our [Telepresence alternatives guide](https://www.signadot.com/blog/the-ultimate-guide-to-telepresence-alternatives-in-2025/) compares mirrord, Telepresence and Signadot side by side, and [Signadot vs Telepresence](https://www.signadot.com/comparison/telepresence/) covers Telepresence on its own. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Does Signadot see my code or data? No, not by default. Only resource metadata, such as names and status, leaves your cluster: never source code, images or application data. Optional features send more, encrypted at rest. Signadot is SOC 2 Type II. See the [security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations). Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. mirrord adds support one tool at a time. Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Okteto Alternatives (Signadot vs Okteto) Source: https://www.signadot.com/comparison/okteto/ Summary: Okteto copies your app into a namespace per developer and PR, with Divert as a limited opt-in for sharing; Signadot gives local development, PRs and coding agents one model, the ephemeral environment, which copies only what changed and shares the rest, with databases, message queues and Temporal. Six key differences, the other Okteto alternatives, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Okteto ### The Okteto alternative that copies only what changed Okteto gives each developer and PR a namespace with its own copy of your app. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Okteto: six key differences Both give developers and PRs environments on Kubernetes, but they start from opposite ends. Okteto copies your app into a namespace, and its Divert option shares some services, with limits. Signadot is built on copying only what changed, and extends it to databases, queues and tests. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares Okteto previews with the other platforms. Feature Signadot Okteto Foundation One model, the ephemeral environment, for local dev, PRs and agents. Copies only what changed. A namespace per developer or PR, with a full copy of your app. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. Skills for laptop agents, CI for unattended ones. Each run gets a full namespace. Sharing services The default, for requests, messages and workflows. Divert is opt-in, HTTP only, and the shared copy must be deployed by Okteto. Databases, queues and more Any database or queue. Resource plugins add anything else. Copy them into each namespace, or share them with no isolation. No plugin framework. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. Divert works with nginx or Istio, not the Gateway API. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. okteto test runs your test commands. No agent test workflows. #### When to choose Okteto A fit when each developer can have a full copy of the app. - Your app is small enough to copy into every namespace - Developers want their code synced into a container in the cluster - You need a fully self-hosted or air-gapped install #### When to choose Signadot The choice when copying the app for every developer, PR or agent is too slow or too costly. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Okteto runs your test commands. Signadot runs them too, and adds Plans that agents build. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions What are the best alternatives to Okteto? Signadot, which gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest, so it suits apps with many services. Bunnyshell and Release also create PR environments, but copy the whole environment, as Okteto does by default. mirrord and Telepresence connect a laptop to a cluster and create no environments. Okteto vs Signadot: what's the difference? They start from opposite ends. Okteto gives each developer or PR a namespace, by default with a full copy of your app. Signadot copies only what changed and shares the rest, with databases, queues and tests built in. Okteto suits small apps and cloud development containers. Signadot suits apps with many services. How is Okteto Divert different from Signadot? Divert lets an Okteto namespace hold only the services you changed and borrow the rest from a shared namespace. It is opt-in, routes HTTP requests only, and the shared namespace must be one Okteto deployed. Databases and queues stay shared unless you copy them, and queue isolation is code you write yourself. In Signadot, sharing is the default, and it covers requests, messages and Temporal workflows. How much does Okteto cost? Okteto's [Starter plan](https://www.okteto.com/pricing/) is free for five seats and self-hosted. Scale and Enterprise are priced on request, per seat. The free hosted Okteto Cloud closed in January 2024. Signadot has a free Starter plan and usage-based paid plans. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Okteto copies them into each namespace or shares them with no isolation. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Signadot vs Telepresence: Local Dev and PR Environments Source: https://www.signadot.com/comparison/telepresence/ Summary: Telepresence is a network tunnel from a laptop to a cluster, with no commercial support today; Signadot gives local development, every PR and every coding agent (on a laptop or in the cloud) one model, the ephemeral environment. Seven key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Telepresence ### One model for local development and PR environments Telepresence tunnels traffic between your laptop and a remote cluster. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Telepresence: seven key differences Both let you test against real services instead of mocks, but they work at different levels. Telepresence works at the network level and has no concept of an environment. Signadot is built on the ephemeral environment, one model for local development, PRs, tests and agents, whether a person or a coding agent writes the code. For the wider field, see our [Telepresence alternatives guide](https://www.signadot.com/blog/the-ultimate-guide-to-telepresence-alternatives-in-2025/). Feature Signadot Telepresence Foundation One model, the ephemeral environment, for local dev, PRs and coding agents. A network tunnel. Every laptop needs cluster access. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. An MCP server drives the tunnel from a desktop. Nothing for cloud agents or tests. PR environments The core of the product. Hundreds share one cluster. None. Databases, queues and more Any database or queue. Resource plugins add anything else. None. Every laptop uses the same shared data. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. Adds its own agent beside the mesh sidecar. No mesh routing. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. None. You run tests on your own. Commercial support Sold and supported by Signadot. None. A community project, kept up mostly by one maintainer. #### When to choose Telepresence A fit when a free network tunnel is all you need. - You only need to reach the cluster from your laptop - You do not need an environment per PR or per agent - Community support is enough for you #### When to choose Signadot The choice when local dev, PRs and coding agents should share one model. - Local dev and PR environments on one model - An ephemeral environment for every PR and coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Signadot runs your tests too. Telepresence does not. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Can Signadot replace Telepresence for local development? Yes. Run the service you are changing on your laptop with [`signadot local connect`](https://www.signadot.com/docs/tutorials/quickstart/local-development), or let your coding agent do it, on your laptop or in the cloud. Your code joins an ephemeral environment, so requests for your change reach it and nobody else's do. The same environment then serves your PR and tests: one model instead of two tools. Telepresence vs Signadot: what's the difference? Telepresence is a network tunnel: it connects a program on your laptop to the cluster and sends it the cluster's traffic. Signadot is built on the ephemeral environment, one model for local development, PRs, tests and coding agents. The environment you or your agent develop in is the one your PR and tests use. Is Telepresence still supported? The open source project still ships releases, but no company sells support or an SLA. Ambassador Labs moved its paid Telepresence features into Blackbird, and Gravitee, which bought Ambassador in July 2025, offers neither. One maintainer writes most of the code, and the project's only sponsor [ended its sponsorship](https://github.com/telepresenceio/telepresence/discussions/4334) in October 2026. Help comes from GitHub and the CNCF Slack. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Telepresence has no database or queue isolation. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## vCluster Alternatives for PR Environments (Signadot vs vCluster) Source: https://www.signadot.com/comparison/vcluster/ Summary: vCluster gives each tenant, team or PR its own virtual Kubernetes cluster, so a cluster per PR holds a full copy of the app; Signadot gives each change an ephemeral environment that copies only what changed and shares the rest of one cluster. Seven key differences, when each fits (vCluster for multi-tenancy and cluster-level changes), using both, vCluster pricing, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs vCluster ### PR environments without a vCluster for each one vCluster gives each tenant its own virtual Kubernetes cluster, so a cluster per PR still runs a full copy of your app. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs vCluster: seven key differences They work at different levels. vCluster gives a team, tenant or CI run its own virtual cluster, with its own API server, and apps deploy to it unchanged. That suits multi-tenancy and testing changes to the cluster itself, but each PR environment holds a full copy of the app. Signadot gives each change an ephemeral environment that copies only what changed and shares the rest of one cluster. The [namespace vs cluster vs vCluster comparison](https://www.signadot.com/blog/namespace-based-environments-for-testing-pros-and-cons/) covers the isolation choices, and the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares the platforms that create PR environments. Feature Signadot vCluster Foundation One model, the ephemeral environment, for local dev, PRs and agents. Copies only what changed. Virtual clusters, each with its own API server. Apps deploy unchanged. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. Isolates agent workloads. No MCP server or skills for testing changes. PR environments Ready in seconds. Hundreds share one cluster. A virtual cluster per PR, holding a full copy of the app. Databases, queues and more Any database or queue. Resource plugins add anything else. A copy in each virtual cluster, deployed by you. No data tooling. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. Syncs Istio and Gateway API resources to the host. Istio needs Enterprise. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. None. You run tests from CI. Cluster-level isolation Environments share one cluster's API server and CRDs. Its own API server, CRDs and RBAC in every virtual cluster. #### When to choose vCluster The fit when teams or tenants need a Kubernetes cluster of their own. - Each team or tenant needs its own API server, CRDs and RBAC - You test operators, CRDs or other changes to the cluster itself - You run a platform with many tenants on shared hardware #### When to choose Signadot The choice when every PR, developer and coding agent needs an environment, and a cluster for each is too slow or too costly. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Signadot runs your tests too. vCluster leaves them to your CI. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions What are the best vCluster alternatives? The best vCluster alternative depends on why you use vCluster. For PR and developer environments, Signadot gives each change an ephemeral environment that copies only what changed and shares the rest of one cluster, so you don't need a virtual cluster per pull request. For multi-tenant clusters, the alternatives are plain namespaces with quotas and RBAC, or separate clusters. vCluster vs Signadot: what's the difference? vCluster virtualizes the cluster: each virtual cluster has its own API server, so a team or a PR can deploy a full copy of the app unchanged. Signadot virtualizes the environment: each change gets an ephemeral environment that copies only what changed and shares the rest of one cluster, through request routing, message routing or configuration. vCluster isolates tenants. Signadot isolates changes. Can I use vCluster and Signadot together? Yes, because they work at different levels. vCluster can give a team or tenant a virtual cluster of its own. Signadot gives each change inside a cluster an ephemeral environment that shares the rest, so a team does not need a virtual cluster for every pull request, developer or coding agent. Is a vCluster per PR expensive? A vCluster per PR costs more as your app grows. Each virtual cluster runs its own control plane plus a full copy of your app and its data, and only Enterprise plans can put idle ones to sleep on their own. Signadot environments hold only what changed, so hundreds share one cluster. [Yellow.ai](https://www.signadot.com/case-studies/how-yellow-ai-enabled-parallel-feature-development-using-signadot/) looked at a vCluster per developer and found it "ended up costing quite a lot". How much does vCluster cost? vCluster is open source and free. vCluster Platform has a [free tier](https://www.vcluster.com/pricing) for up to 64 CPUs, and Enterprise plans, priced on request, add features such as auto sleep, SSO and the Istio integration. Signadot has a free Starter plan and usage-based paid plans. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. A virtual cluster holds whatever copies of them you deploy into it. Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Qovery Alternatives for Preview Environments (Signadot vs Qovery) Source: https://www.signadot.com/comparison/qovery/ Summary: Qovery deploys apps to Kubernetes on your own cloud and clones a whole blueprint environment for each PR; Signadot gives local development, PRs and coding agents one model, the ephemeral environment, which copies only what changed and shares the rest, with databases, message queues and Temporal. How Qovery previews work, Qovery pricing, seven key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Qovery ### The Qovery alternative that copies only what changed Qovery clones a whole environment for every pull request. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Qovery: seven key differences Qovery is a platform for deploying apps to Kubernetes on your own cloud, and its previews clone a whole environment for each pull request. Signadot runs in the cluster you already have and gives each change an ephemeral environment that copies only what changed, with databases, queues and tests built in. For the other approaches, see the [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/) and the [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/). Feature Signadot Qovery Foundation One model, the ephemeral environment, for local dev, PRs and agents. Copies only what changed. A platform that deploys apps to your cloud. Previews clone a whole environment. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. An MCP server and skills that drive Qovery. Each agent workspace is its own environment. PR environments Ready in seconds. Hundreds share one cluster. A clone of a blueprint environment per PR, in about three minutes. Local development Run the service you are changing on your laptop, inside an ephemeral environment. Port-forwarding only. Cluster traffic never reaches your laptop. Databases, queues and more Any database or queue. Resource plugins add anything else. Cloned per preview without their data, or shared with no isolation. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. None in its docs. Its Gateway API ingress handles incoming traffic only. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. None. You run tests from CI against the preview URL. #### When to choose Qovery A fit when you want one platform to deploy and run your apps on your own cloud. - You want a managed way to run apps on AWS, GCP, Azure or Scaleway - Your app is small enough to clone for every PR - You want previews and production deploys from one tool #### When to choose Signadot The choice when cloning the whole environment for every PR, developer or agent is too slow or too costly. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Signadot runs your tests too. Qovery leaves them to your CI. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions What are the best Qovery alternatives? The best Qovery alternative depends on what you use Qovery for. For preview and ephemeral environments, Signadot copies only what changed and shares the rest, so hundreds of environments run on one cluster, with local development and coding agents on the same model. For deploying and running apps on your own cloud, teams also compare Northflank and Porter. Qovery vs Signadot: what's the difference? Qovery deploys your apps to Kubernetes on your own cloud account, and gives each pull request a clone of a whole environment. Signadot runs in the cluster you already have and gives each change an ephemeral environment that copies only what changed and shares the rest. Qovery is a deployment platform with previews. Signadot is built for testing changes, from local development to PRs and coding agents. How do Qovery preview environments work? Each pull request gets a clone of a blueprint environment in its own namespace, with its apps, databases and configuration, in about three minutes by Qovery's figure. You can leave services out or share a database, but nothing routes a preview's requests to the services it leaves out. A database clone copies its configuration, not its data, so Lifecycle Jobs seed it. How much does Qovery cost? Qovery's [plans](https://www.qovery.com/pricing) start at $299 a month billed yearly, for three users, one cluster and 3,000 deployment minutes, with Growth, Business and Enterprise above it. There is no free plan, only a 14-day trial. Previews are included, but each one uses deployment minutes and runs on your own cloud bill. Signadot has a free Starter plan and usage-based paid plans. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Qovery clones databases and queues into each preview, without their data. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Release (ReleaseHub) Alternative (Signadot vs Release) Source: https://www.signadot.com/comparison/release/ Summary: Release copies the whole stack for every PR and runs the clusters; Signadot gives local development, PRs and coding agents one model, the ephemeral environment, which copies only what changed and shares the rest, with databases, message queues and Temporal. Six key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Release ### The Release (ReleaseHub) alternative that copies only what changed Release copies your whole stack for every pull request. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Release: six key differences Both give each pull request an environment to test in, but they are built differently. Release copies the whole environment, every service and database, and runs the clusters for you. Signadot copies only what a change touches and shares the rest, so environments stay fast and cheap as your app grows. The [complete guide to preview environments](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) compares Release with the other preview platforms. Feature Signadot Release Foundation One model, the ephemeral environment, for local dev, PRs and agents. Copies only what changed. Copy the whole environment for every PR. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. An API for agents, and a full-stack copy for each one. PR environments Ready in seconds. Hundreds share one cluster. A full stack per PR. Cost grows with every service. Databases, queues and more Any database or queue. Resource plugins add anything else. A database copy per environment. Queues run inside each copy. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. None in its docs. Routing is NGINX ingress by host and path. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. None. You run your own tests against it from CI. #### When to choose Release A fit when your app is small enough to copy in full. - Your app has a handful of services - You want every PR to get a standalone copy, with nothing shared - You want a vendor to run the clusters in your cloud account #### When to choose Signadot The choice when copying the whole stack for every PR is too slow or too costly. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Signadot runs your tests too. Release does not. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Release vs Signadot: what's the difference? They take opposite approaches. Release copies the whole environment for each PR and runs the clusters for you. Signadot copies only what changed and plugs it into one shared environment, so hundreds of environments run on one cluster. Full copies suit small apps. Copying only what changed suits apps with many services. Is Release the same as ReleaseHub? Yes. ReleaseHub is the earlier name of Release. releasehub.com now redirects to [release.com](https://release.com/). Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Release copies or branches a database for each environment, and runs queues inside each copy. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Does Signadot see my code or data? No, not by default. Only resource metadata, such as names and status, leaves your cluster: never source code, images or application data. Optional features send more, encrypted at rest. Signadot is SOC 2 Type II. See the [security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations). Does Signadot require a service mesh? No. Without a mesh, or with Linkerd, a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar routes each request to the right environment. If you run [Istio](https://www.signadot.com/docs/guides/request-routing/istio), Signadot updates the routing rules in your VirtualServices and adds no sidecar of its own, in sidecar or ambient mode. It also works with the [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api). The routing key travels in the OpenTelemetry `baggage` header your tracing libraries already pass along. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Pact vs Signadot Smart Tests: How They Work Together Source: https://www.signadot.com/comparison/pact/ Summary: Pact checks consumer-driven API contracts in unit tests with nothing deployed; Signadot gives each change an ephemeral environment with real services, where Smart Tests compare the changed service with the baseline and your own tests check behavior, data and side effects. Complementary, not a replacement: six key differences, when to use each, using both, Pact ownership and pricing, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Pact ### Pact checks the contract. Signadot runs the change with real services. Pact catches a broken API contract in your unit tests, with nothing deployed. Signadot gives each change an ephemeral environment with real services, where Smart Tests and your own tests check what contracts leave out. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Pact: six key differences They answer different questions, so most teams that use one can use both. Pact checks, with nothing deployed, that a provider still gives each consumer what it expects. Signadot gives each change an ephemeral environment with real services, where Smart Tests compare the changed service with the baseline and your own tests check behavior, data and side effects. For the category, see the [complete guide to contract testing](https://www.signadot.com/contract-testing/). Feature Signadot Pact Foundation Ephemeral environments with real services. Tests and agents run in them. Contract tests in your unit test suite. No environment needed. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. An MCP server and agent skills that write and review Pact tests. What it checks Smart Tests compare a changed service with the baseline, on real dependencies. That a provider still meets each consumer's expectations. Databases, queues and more Real databases and queues in every environment, through resource plugins and message routing. Message contracts, not side effects. Database writes are left to other tests. Protocols Smart Tests cover REST with JSON. Jobs run tests for anything else. HTTP and messages, plus gRPC and GraphQL through plugins. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. Contract tests only, by design. Functional tests run elsewhere. #### When to choose Pact The fit for checking that services keep their promises to each other. - You own both sides of an API and want breaks caught in unit tests - You want can-i-deploy to gate every release - You need contracts for messages, gRPC or GraphQL #### When to add Signadot The fit for testing what contracts leave out, with real services. - Behavior, data and side effects across services - An ephemeral environment for every PR and coding agent, local or in the cloud - Smart Tests on every environment, plus your own suites as Jobs - Plans that coding agents build and run - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Pact runs in your unit tests. Signadot runs the tests that need real services, in an ephemeral environment. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Is Signadot an alternative to Pact? No. Signadot works beside Pact rather than replacing it. Pact checks that a provider still gives each consumer what it expects, in your unit tests, with nothing deployed. Signadot gives each change an ephemeral environment with real services, where Smart Tests and your own tests check behavior, data and side effects, which Pact leaves to other tests. Pact vs Signadot Smart Tests: what's the difference? Pact is consumer-driven: each consumer's tests record what it expects from a provider, and the provider replays those requests against its own code. Smart Tests are provider-side: they send the same requests to the changed service and to the baseline in a live environment, and flag the differences that matter. Pact knows what each consumer needs. Smart Tests see how the service behaves with its real dependencies. Can I use Pact and Signadot together? Yes. Keep Pact in each service's build and gate releases with can-i-deploy. Then give each pull request an ephemeral environment in Signadot, where [Smart Tests](https://www.signadot.com/docs/concepts/smart-tests-and-jobs) run when the environment is created and Jobs run your integration and end-to-end tests against real services. Pact catches a broken contract early. Signadot catches what only shows up with real services, data and queues. Do Smart Tests need contracts or mocks? No. You write the requests in a short Starlark script, and Signadot sends them to the changed service and to the baseline in your cluster. A model scores each difference between the responses, so you don't write an assertion for every field, and you add [checks](https://www.signadot.com/docs/reference/smart-tests/checks) where you want one. Smart Tests cover REST APIs with JSON. For other protocols, run your own tests as Jobs. Who owns Pact, and what does it cost? Pact is free and open source, maintained under the Pact Foundation with SmartBear as its sponsor. SmartBear bought PactFlow, the hosted Pact Broker, in 2022 and is moving it into Swagger Contract Testing. [PactFlow](https://pactflow.io/pricing/) has a free Starter plan and paid Team and Enterprise plans. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Pact checks the shape of each message, not what happens when a service handles it. Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Garden.io Alternatives (Signadot vs Garden) Source: https://www.signadot.com/comparison/garden/ Summary: Garden deploys your stack into a namespace per developer, PR or CI run with build and test caching; Incredibuild acquired it in 2024 and it has shipped only security patches since February 2026. Signadot gives local development, PRs and coding agents one model, the ephemeral environment, which copies only what changed and shares the rest. Whether Garden is still maintained, the alternatives, using both, seven key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Garden ### The Garden.io alternative that copies only what changed Garden deploys your stack into a namespace for every developer and PR, and has shipped no new features since February 2026. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Garden: seven key differences Garden automates building, deploying and testing your stack, with a namespace for each developer, PR or CI run and caching that skips what has not changed. Signadot copies only what changed into an ephemeral environment and shares the rest, for local development, PRs and coding agents. For a deeper look at the two models, see [Signadot vs Garden.io](https://www.signadot.com/articles/signadot-vs-garden-io/). Feature Signadot Garden Foundation One model, the ephemeral environment, for local dev, PRs and agents. Copies only what changed. A namespace per developer, PR or CI run, with the whole stack deployed by default. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. None. No MCP server or agent skills. PR environments Ready in seconds. Hundreds share one cluster. A namespace per PR. No routing to shared services. Databases, queues and more Any database or queue. Resource plugins add anything else. Deployed into each namespace, or shared with no isolation. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. None in its docs. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. Test actions with cached results. No agent test workflows. Project status Sold and supported by Signadot. Acquired by Incredibuild. Security patches only since February 2026. #### When to choose Garden A fit when you already run Garden and your stack is small enough to copy into every namespace. - Every service already has its garden.yml - Cached builds and test results matter most - You can plan for a project that ships only security patches #### When to choose Signadot The choice when copying the stack for every developer, PR or agent is too slow or too costly. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Garden runs test actions and caches their results. Signadot runs your test suites in each ephemeral environment, and adds Plans that agents build. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Is Garden still maintained? Garden gets security patches but no new features. Incredibuild acquired Garden in November 2024. Its last feature release, 0.14.20, shipped in February 2026, and the releases since are security patches to the older 0.13 line. garden.io now redirects to the docs, and no roadmap or maintenance plan has been announced. What are the best Garden alternatives? The best Garden alternative depends on what you used Garden for. For environments per developer, pull request and coding agent, Signadot copies only what changed and shares the rest of one cluster, with databases, queues and tests built in. For a build and deploy loop, Tilt is the closest tool, and Google will archive Skaffold in January 2027. For cached build and test pipelines in CI, teams look at tools such as Dagger. Garden vs Signadot: what's the difference? Garden deploys your stack into a namespace for each developer, PR or CI run, and caches builds and test results so it skips what has not changed. Signadot copies only what changed into an ephemeral environment and shares the rest of one cluster, so environments stay small as the stack grows. Can I use Garden and Signadot together? Yes. Garden can keep building and publishing your images, and an exec action can call the [Signadot CLI](https://www.signadot.com/docs/guides/integrate-ci) to create an ephemeral environment with the images you changed. Everything else comes from your shared baseline, so each PR no longer needs a namespace with the whole stack. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Garden deploys a database into each namespace, or shares one with no isolation. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Tilt Alternatives (Signadot vs Tilt) Source: https://www.signadot.com/comparison/tilt/ Summary: Tilt (owned by Docker) runs a build and live-update loop, usually against a local cluster each developer runs; Signadot gives local development, PRs and coding agents one model, the ephemeral environment, in one shared cluster. Whether Tilt is still maintained, the alternatives, Tilt and MCP, using both, seven key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Tilt ### The Tilt alternative that scales past your laptop Tilt rebuilds and live-updates your services on a cluster of your own, usually on your laptop. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Tilt: seven key differences Tilt automates the loop from a code change to a running service, on a cluster each developer runs, usually a local one. Signadot gives each change its own ephemeral environment in one shared cluster, for local development, PRs and coding agents, so nobody has to run the whole app. For the approaches side by side, see the [guide to local development on Kubernetes](https://www.signadot.com/guide-to-local-development-kubernetes/). Feature Signadot Tilt Foundation One model, the ephemeral environment, for local dev, PRs and coding agents. A build and live-update loop, usually against a local cluster. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. No official MCP server or agent skills. Community MCP servers only. Environments An ephemeral environment per change. Hundreds share one cluster. None per PR. Each developer runs services on a cluster or namespace of their own. Databases, queues and more Any database or queue. Resource plugins add anything else. Deployed like any other service. No data isolation. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. None in its docs. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. Tests run as local commands. tilt ci checks that services start. Project status Sold and supported by Signadot. Owned by Docker. Monthly upkeep releases. No paid support. #### When to choose Tilt A fit when each developer can run the services they need on a cluster of their own. - Your app fits on a laptop, or on a cluster per developer - You want live updates and a dashboard for that loop - A free tool with community support is enough #### When to choose Signadot The choice when the app no longer fits on a laptop, or PRs and coding agents need environments too. - Dozens or hundreds of services - Local dev and PR environments on one model - An ephemeral environment for every PR and coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Tilt runs your test commands beside your services. Signadot runs your test suites in each ephemeral environment, and adds Plans that agents build. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Is Tilt still maintained? Yes. Tilt is open source and owned by Docker, which acquired it in 2022. It still ships releases about once a month, mostly upkeep such as new Kubernetes and Go versions plus small features, with most of the work from one maintainer. Tilt Cloud, its hosted team features, shut down in 2022, and there is no paid plan or support. What are the best Tilt alternatives? The best Tilt alternative depends on what you need. To give each developer, pull request and coding agent an environment to test in without running the app on a laptop, Signadot copies only what changed and shares the rest of one cluster, with databases, queues and tests built in. To keep a build and live-update loop on a cluster of your own, Skaffold is the closest tool, but Google will archive it in January 2027. For apps that run in Docker Compose, Compose watch mode syncs files the same way. Tilt vs Signadot: what's the difference? Tilt watches your code, rebuilds or live-updates your services and deploys them to a cluster, usually a local one, so each developer runs the services they need. Signadot gives each change an ephemeral environment in a shared cluster that copies only what changed and shares the rest: one model for local development, PRs and coding agents. Can I use Tilt and Signadot together? Yes. Tilt can keep running the loop for the service you are changing, while Signadot gives every pull request and coding agent an ephemeral environment with real services, data and tests. For local work against real services, [`signadot local connect`](https://www.signadot.com/docs/tutorials/quickstart/local-development) runs that service on your laptop inside an ephemeral environment, with nothing to rebuild for each edit. Does Tilt have an MCP server? Not an official one. A few community projects wrap Tilt in an MCP server. Signadot ships its own MCP server and agent skills, and gives each coding agent an ephemeral environment to test in. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Tilt deploys a database like any other service, with no isolation. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Skaffold Alternatives (Signadot vs Skaffold) Source: https://www.signadot.com/comparison/skaffold/ Summary: Skaffold builds and redeploys images to a cluster and creates no environments, and Google will archive its repository on January 29, 2027; Signadot gives local development, PRs and coding agents one model, the ephemeral environment, which copies only what changed and shares the rest. Whether Skaffold is still maintained, the alternatives, using both, seven key differences, when to choose each, coding agents, testing with Jobs and Plans, service mesh (Istio, Gateway API, Linkerd) and deployment. Signadot vs Skaffold ### The Skaffold alternative for local dev, PRs and coding agents Google will archive the Skaffold repository in January 2027. Signadot gives local development, PRs and coding agents one model: ephemeral environments that copy only what changed and share the rest. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) One platform · every layer Agent-native validationPlans · MCP server · Agent skills Testing runtimeSmart Tests · Jobs Isolated environmentsSandboxes · Request routing · Resource plugins Your Kubernetes cluster Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - #### Signadot vs Skaffold: seven key differences They solve different problems. Skaffold builds your images and redeploys them to a cluster as you edit, and creates no environments. Signadot gives each change its own ephemeral environment, for local development, PRs and coding agents, so you test against real services without deploying everything yourself. For the approaches side by side, see the [guide to local development on Kubernetes](https://www.signadot.com/guide-to-local-development-kubernetes/). Feature Signadot Skaffold Foundation One model, the ephemeral environment, for local dev, PRs and coding agents. A build and deploy loop run from your laptop or CI. It creates no environments. Coding agents Agents on laptops or in the cloud each get an ephemeral environment for Jobs and Plans. None. No MCP server or agent skills. Environments An ephemeral environment per change. Hundreds share one cluster. None. It deploys your config to the cluster and namespace you pick. Databases, queues and more Any database or queue. Resource plugins add anything else. None. Whatever it deploys is yours to isolate. Service mesh and Gateway API Native on Istio, with no extra sidecar. Gateway API and Linkerd too. None in its docs. Testing: Jobs and Plans Jobs run your tests. Plans let agents build test workflows. Image tests before deploy, verification tests after. No agent test workflows. Project status Sold and supported by Signadot. Repository to be archived on January 29, 2027. No paid support. #### When to choose Skaffold A fit while one developer needs a free build and deploy loop, and can plan for the archive. - You build and deploy to a cluster or namespace of your own - You use Cloud Deploy, which renders and deploys with a Skaffold config - You can move off it before January 2027 #### When to choose Signadot The choice when developers, PRs and coding agents need environments, not just a deploy loop. - Local dev and PR environments on one model - An ephemeral environment for every PR and coding agent, local or in the cloud - Any database or queue, plus resource plugins - Built-in testing with Jobs and Plans - A free Starter plan to get going #### The bottleneck has shifted from writing code to proving it works. Agents write code faster than teams can check it. Signadot gives every change its own environment with real services and data, so your tests and agents can show it works. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) The Signadot platform #### Four pillars of the Signadot platform Signadot spins up an ephemeral environment for every code change, from a developer's laptop to thousands of coding agents, so your tests and agents can check it against your real services, databases and message queues. agent-01feature-auth validated agent-02bugfix-cart running agent-03refactor-db validated agent-04pr-1234 testing Built for Agents ##### Run thousands of agents in parallel. Each agent gets its own lightweight ephemeral environment, cheap and fast to spin up, so agents run more validations on their own before human review. Local devinner loop PR validationouter loop Ephemeral environmentthe same for both Unified Architecture ##### Inner and outer loop, one platform. The same ephemeral environment covers local development and PR validation. One platform. One mental model. Test runtime Smart TestsJobsPlans Ephemeral environments \+ ∞ Comprehensive Coverage ##### Smart Tests, Jobs and Plans, built in. Jobs run your existing tests. Plans let agents build test workflows. All of it runs in your cluster. Ephemeral environmentpr-1234 Per ephemeral environment Databasebranch or partition Message queuerouting key Cloud resourceresource plugin Deep Isolation ##### Data, queues and cloud resources. Share a database or branch it per ephemeral environment. Queues stay shared, with messages routed by key. Resource plugins create cloud resources outside Kubernetes. [Learn more about the Signadot platform](https://www.signadot.com/docs) Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering Coding agents #### Built for agentic development Most code is now written by coding agents, on laptops and in the cloud. Each one gets its own ephemeral environment to test in, the same kind a developer uses. - An MCP server and agent skills that work across the coding agents your teams use - Background agents running in cloud sandboxes get their own ephemeral environment too (see the [Modal example](https://github.com/signadot/examples/tree/main/modal-sandboxes-tutorial)) - Jobs and Plans that run in each agent's environment [Set up Signadot for coding agents](https://www.signadot.com/docs/integrations/coding-agents) Testing #### Run the tests too, with Jobs and Plans Skaffold tests your images before it deploys them. Signadot runs your test suites in each ephemeral environment, and adds Plans that agents build. - Jobs run the test suites you already have - Plans let agents build automated test workflows - Your platform team picks the Plans every change must pass [See Plans in action](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation) Cloud native stack #### Works with your service mesh Signadot routes through the mesh you already run. On Istio it adds no sidecar of its own: it updates your VirtualService rules and Istio does the routing. - [Istio](https://www.signadot.com/docs/guides/request-routing/istio), natively, in sidecar or ambient mode - [Gateway API](https://www.signadot.com/docs/guides/request-routing/gateway-api): routes beside your HTTPRoutes and GRPCRoutes - Linkerd or no mesh: a lightweight [DevMesh](https://www.signadot.com/docs/guides/request-routing/devmesh) sidecar - OpenTelemetry baggage carries the routing key [How request routing works](https://www.signadot.com/docs/guides/request-routing) Deployment and security #### Runs in your cluster Most of Signadot runs in your cluster. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). - Only resource metadata leaves your cluster by default - Never your code, app data or production PII - SOC 2 Type II [Read the security docs](https://www.signadot.com/docs/guides/admin/security/data-integrations) #### Built for Developers and Agents [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) Works with your coding agents *Image: Claude Code logo**Image: Cursor logo**Image: OpenAI logo* Integrate with CI tools you already use *Image: jenkins logo**Image: GitLab logo**Image: GitHub logo**Image: Bitbucket logo* Support all test frameworks *Image: Cypress logo**Image: Selenium logo**Image: Cucumber logo**Image: k6 logo**Image: Postman logo**Image: Playwright logo* > Signadot is making it easier for our developers to build and test microservices using deployments that are incredibly similar to production. This has made **both human and AI-written code easier to validate and test**. *Image: Naomi Klein, Lead Infrastructure Engineer at Laurel* Naomi KleinLead Infrastructure Engineer at Laurel #### Frequently asked questions Is Skaffold still maintained? Skaffold gets dependency and security updates until Google archives it. [Skaffold's docs](https://skaffold.dev/docs/) say the GitHub repository will be archived on January 29, 2027, and Skaffold leaves the gcloud CLI after January 15, 2027. Existing releases stay available. Its 2026 releases are mostly dependency and security updates, and Google has named no successor. What are the best Skaffold alternatives? The best Skaffold alternative depends on what you used Skaffold for. To give each developer, pull request and coding agent an environment to test in, Signadot copies only what changed and shares the rest, so hundreds of environments run on one cluster, with databases, queues and tests built in. To keep a build and deploy loop on your laptop, Tilt is the closest tool. For rendering manifests in CI, Helm or Kustomize can do what `skaffold render` does. Skaffold vs Signadot: what's the difference? Skaffold is a command line tool that builds your images, deploys them to the cluster and namespace you choose, and redeploys as you edit. It creates no environments and shares nothing. Signadot gives each change an ephemeral environment in your cluster that copies only what changed and shares the rest: one model for local development, PRs and coding agents. Can Signadot replace Skaffold for local development? Yes, for developing against real services. Run the service you are changing on your laptop with [`signadot local connect`](https://www.signadot.com/docs/tutorials/quickstart/local-development), or let your coding agent do it. Requests for your change reach it and everything else comes from your shared baseline, so there is no image to rebuild and redeploy for each edit. Can I use Skaffold and Signadot together? Yes, while you still run Skaffold. Build only the images you changed with `skaffold build`, which can write their tags to a file, then create an ephemeral environment with those images from your [CI pipeline](https://www.signadot.com/docs/guides/integrate-ci). Signadot shares everything else from your baseline. Does Signadot work with databases and message queues? Yes, with any database or queue. [Resource plugins](https://www.signadot.com/docs/concepts/resources) create a database for each environment, using tools you already use. [Queues](https://www.signadot.com/docs/guides/set-up-message-queue-isolation) stay shared, and each message reaches the right environment. The same works for [Temporal](https://www.signadot.com/docs/tutorials/testing-temporal-workers). The [complete guide to ephemeral environments](https://www.signadot.com/guide-to-ephemeral-environments-kubernetes/#how-ephemeral-environments-handle-databases-message-queues-and-workflow-systems) compares the approaches. Skaffold has no database or queue isolation. Is Signadot self-hosted? Signadot is hybrid. Most of it runs in your cluster, with a small cloud control plane that only sees resource metadata by default. Need fully self-hosted? [Talk to us](https://www.signadot.com/schedule-a-call/). Can coding agents use Signadot? Yes. Coding agents such as Claude Code and Cursor connect through Signadot's [MCP server](https://www.signadot.com/docs/integrations/mcp) and [agent skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills). Each agent gets its own ephemeral environment, whether it runs on a developer's laptop or as a background agent in the cloud, and runs Jobs and Plans against it before anyone reviews the change. #### Validate code as fast as agents write it. Cut environment costs by 90%. Explore how Signadot helps engineering teams validate code at the speed agents generate it, with cost-efficient parallel validation on your existing Kubernetes infrastructure. [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Blog & Articles ## Blog Source: https://www.signadot.com/blog/ Summary: Engineering blog covering microservices testing, Kubernetes development, agentic workflows, and DevOps best practices. ### Signadot Blog [ *Image: Watch Claude Code Validate a Microservices Change in a Closed Loop* Featured Agents #### Watch Claude Code Validate a Microservices Change in a Closed Loop Anirudh Ramanathan · June 4, 2026 Two Claude Code sessions, the same microservices task. One ships a change that looks correct. The other uses signadot-validate to catch a cross-service break before calling the work done. Read more ](https://www.signadot.com/blog/claude-code-signadot-validate-demo/) - [ *Image: Code Generation Isn't the Bottleneck Anymore: What Platform Leaders Told Us Over Dinner* Agents ### Code Generation Isn't the Bottleneck Anymore: What Platform Leaders Told Us Over Dinner Platform, security and AI leaders agreed on one thing at our recent executive dinner: agents made code cheap, and everything after the code is now the constraint. Arjun Iyer · October 22, 2026 ](https://www.signadot.com/blog/exec-dinner-agentic-coding-throughput/) - [ *Image: Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk* AgentsEphemeral Environments ### Verification Is the New Bottleneck: Takeaways From DZone's Cloud-Native Foundations Trend Talk Drift, late staging failures and agent pull request volume are moving cloud-native delivery's bottleneck to verification. Key themes from DZone's Cloud-Native Foundations Trend Talk. Tristan Willis · October 21, 2026 ](https://www.signadot.com/blog/dzone-trend-talk-cloud-native-foundations/) - [ *Image: Independent Deployability Has Two Halves. Most Teams Built One.* ### Independent Deployability Has Two Halves. Most Teams Built One. Teams with mature deploy tooling still ship microservices in batches, because nothing lets them trust one change on its own. Coding agents are widening that gap. Arjun Iyer · October 19, 2026 ](https://www.signadot.com/blog/independent-deployability-validation-gap/) - [ *Image: The Signadot GitHub Action Is Live* Product Announcements ### The Signadot GitHub Action Is Live The Signadot Sandbox GitHub Action gives every pull request its own Sandbox: created on the first push, updated on each push after, and deleted when the PR closes. Joe Horsnell · October 15, 2026 ](https://www.signadot.com/blog/signadot-github-action-is-live/) - [ *Image: Signadot Is Now Available on AWS Marketplace* Product Announcements ### Signadot Is Now Available on AWS Marketplace Self-serve the Business tier or secure an Enterprise plan via private offer to streamline procurement and billing through your existing AWS account. Signadot · October 13, 2026 ](https://www.signadot.com/blog/signadot-on-aws-marketplace/) - [ *Image: What a Green Check on a Pull Request Should Promise* Agents ### What a Green Check on a Pull Request Should Promise Most merge gates stop at build, unit tests and contracts. Coding agents clear those easily, so the check that matters most, behavior against the real system, has to run before merge. Arjun Iyer · October 12, 2026 ](https://www.signadot.com/blog/what-a-green-check-should-promise/) - [ *Image: Coding Agents Run Their Own Code Now. How Much of the System Does That Reach?* Agents ### Coding Agents Run Their Own Code Now. How Much of the System Does That Reach? Greptile, Cursor, Codex and Devin now execute the code an agent writes. In a microservices system, a private runtime with mocks still misses the bugs that live between services. Arjun Iyer · October 5, 2026 ](https://www.signadot.com/blog/runtime-verification-coding-agents-reach/) - [ *Image: Async Coding Agents Need a Verifier That Isn't a Person* Agents ### Async Coding Agents Need a Verifier That Isn't a Person When agents run from events and schedules, nobody is watching to check their work. In a cloud-native system, that check has to happen in a real runtime before the PR opens. Arjun Iyer · September 28, 2026 ](https://www.signadot.com/blog/async-coding-agents-need-a-verifier/) - [ *Image: Coding Agents Need a Runtime to Check Their Work Against* Agents ### Coding Agents Need a Runtime to Check Their Work Against Coding agents self-correct only when something tells them whether a change worked. In a cloud-native system, that signal comes from a running environment, not from local checks. Arjun Iyer · September 21, 2026 ](https://www.signadot.com/blog/coding-agents-runtime-feedback-gap/) - [ *Image: Agent Loops Don't Have a Token Problem. They Have a Feedback Problem.* Agents ### Agent Loops Don't Have a Token Problem. They Have a Feedback Problem. Loop-based development means agents run longer, in parallel, and without anyone watching each attempt. What that costs is decided by the verification surface under the agent, not the token price. Arjun Iyer · September 14, 2026 ](https://www.signadot.com/blog/agent-loops-feedback-problem/) - [ *Image: LegalOn Technologies' Agent-Ready Golden Path on GKE* Case StudiesEphemeral EnvironmentsAgents ### LegalOn Technologies' Agent-Ready Golden Path on GKE How a Tokyo-based legal AI company adapted its internal GKE platform for AI agents, connecting local development to the cluster and validating every PR with Signadot preview environments before merge. Signadot · September 2, 2026 ](https://www.signadot.com/blog/legalon-technologies-agent-ready-golden-path-on-gke/) - [ *Image: OpenSpec + Signadot: Integration Tests Inside the Agent Loop* AgentsIntegration/E2E Testing ### OpenSpec + Signadot: Integration Tests Inside the Agent Loop Austin Xu wired Signadot sandboxes into his AI coding agent workflow and moved integration testing inside the agent's loop. A high-level look at his journey, the integration, and the results. Arjun Iyer · August 12, 2026 ](https://www.signadot.com/blog/openspec-signadot-integration-tests-inside-the-agent-loop/) No matching posts found. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Articles Source: https://www.signadot.com/articles/ Summary: In-depth technical articles on testing strategies, platform engineering, and AI-assisted development. ### Signadot Articles [ *Image: Shift-Left Testing for Microservices on Kubernetes* Featured #### Shift-Left Testing for Microservices on Kubernetes Arjun Iyer · September 30, 2026 How to move integration tests from shared staging to the pull request: what the environment behind that gate needs, two ways to build one on Kubernetes, and the order to add checks. Read more ](https://www.signadot.com/articles/shift-left-testing/) - [ *Image: Shift-Left Testing for Microservices on Kubernetes* ### Shift-Left Testing for Microservices on Kubernetes How to move integration tests from shared staging to the pull request: what the environment behind that gate needs, two ways to build one on Kubernetes, and the order to add checks. Arjun Iyer · September 30, 2026 ](https://www.signadot.com/articles/shift-left-testing/) - [ *Image: What Is Environment Parity? How to Make Dev Environments Match Production* Ephemeral Environments ### What Is Environment Parity? How to Make Dev Environments Match Production What environment parity means, the four kinds of drift behind bugs that show up in staging but not locally, and how to get a production-like test environment without cloning production. Arjun Iyer · September 10, 2026 ](https://www.signadot.com/articles/environment-parity/) - [ *Image: Integration tests pass with mocks but staging still breaks: how to catch it earlier* Integration/E2E Testing ### Integration tests pass with mocks but staging still breaks: how to catch it earlier Mocks check your assumptions about a dependency, not the dependency. The failures they miss, three additions that catch them before merge, and how stack size picks the environment. Arjun Iyer · August 12, 2026 ](https://www.signadot.com/articles/integration-tests-pass-but-staging-breaks/) - [ *Image: Signadot vs Garden.io: Which Kubernetes Environment Model Fits Your Team?* ### Signadot vs Garden.io: Which Kubernetes Environment Model Fits Your Team? How Garden's full copy of the stack per environment and Signadot's Sandboxes on a shared cluster each work, where each strains, and which fits your stack size and number of environments. Arjun Iyer · August 6, 2026 ](https://www.signadot.com/articles/signadot-vs-garden-io/) - [ *Image: Preview Environments for QA and Stakeholder Review* ### Preview Environments for QA and Stakeholder Review How a preview environment per pull request moves QA, product and design review ahead of the merge, what changes for the QA role, and the access, data and teardown the workflow needs. Arjun Iyer · August 5, 2026 ](https://www.signadot.com/articles/preview-environments-for-qa-and-stakeholder-review/) - [ *Image: ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains* ### ArgoCD Preview Environments: How the ApplicationSet Pattern Works and Where It Strains How Argo CD's ApplicationSet pull request generator builds a preview per PR, where that duplication strains as stacks and PR volume grow, and when shared-cluster request routing fits better. Arjun Iyer · July 24, 2026 ](https://www.signadot.com/articles/argocd-preview-environments/) - [ *Image: Per-PR Preview URLs, Explained* ### Per-PR Preview URLs, Explained Why a preview URL on every pull request is nearly free for a frontend and hard for a backend, the two ways to route a PR to its own preview, and how access, data and cleanup work. Arjun Iyer · July 21, 2026 ](https://www.signadot.com/articles/per-pr-preview-urls/) - [ *Image: Preview Environments: The Complete Guide* ### Preview Environments: The Complete Guide What a preview environment is, how it relates to staging and ephemeral environments, the four ways to build one, how the platforms compare, and what coding agents change about them. Arjun Iyer · July 31, 2025 ](https://www.signadot.com/articles/comprehensive-guide-to-preview-environments/) - [ *Image: Best Microservices Testing Tools for Kubernetes in 2026* ### Best Microservices Testing Tools for Kubernetes in 2026 The microservices testing tools for Kubernetes in 2026, in two layers: test automation such as Testkube, Pact and k6, and environment platforms such as Signadot and Release. Arjun Iyer · July 17, 2025 ](https://www.signadot.com/articles/best-microservices-testing-solution-for-kubernetes-in-2025/) - [ *Image: What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control* ### What Ephemeral Environments Cost, and How Teams Keep the Bill Under Control What drives the cost of ephemeral environments, a full copy per pull request versus only what changed, the hidden costs, the controls that work, and what Brex, Arkestro and Yellow.ai report. Arjun Iyer · July 17, 2025 ](https://www.signadot.com/articles/reducing-costs-boosting-productivity-kubernetes-ephemeral-environments/) - [ *Image: AI Powered Contract Testing for Microservices Excellence* Contract Testing ### AI Powered Contract Testing for Microservices Excellence Why hand-written contracts fall behind fast-moving microservices, and how AI-powered contract testing infers them from real API traffic and compares each change against a baseline. Nick Andrews · July 3, 2025 ](https://www.signadot.com/articles/ai-powered-contract-testing-for-microservices-excellence/) - [ *Image: Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster* Ephemeral Environments ### Isolation Models for Ephemeral Environments: Full Copy vs Shared Cluster The two ways ephemeral environments keep changes apart, a full copy of the stack or one shared cluster that deploys only the changed services, compared on cost, spin-up time and scale. Nick Andrews · June 30, 2025 ](https://www.signadot.com/articles/isolation-models-for-ephemeral-environments-infra-level-vs-request-level/) - [ *Image: Stop Breaking Your Microservices with SmartTests: AI Powered Contract Testing* Contract Testing ### Stop Breaking Your Microservices with SmartTests: AI Powered Contract Testing How small API contract changes break microservices, the practices and tools that catch them, and how SmartTests compares real traffic against a baseline to flag breaking changes before merge. Nick Andrews · June 18, 2025 ](https://www.signadot.com/articles/stop-breaking-your-microservices-with-smarttests-ai-powered-contract-testing/) - [ *Image: Signadot SmartTests vs Pact Contract Testing* Contract Testing ### Signadot SmartTests vs Pact Contract Testing Pact's consumer-defined contracts checked against mocks, next to Signadot SmartTests' AI-assisted checks against real services in Kubernetes, compared on setup effort and maintenance. Nick Andrews · June 13, 2025 ](https://www.signadot.com/articles/signadot-smarttests-vs-pact-contract-testing/) - [ *Image: The Ultimate Guide to Lightweight Kubernetes Environments* Tutorials/Guides ### The Ultimate Guide to Lightweight Kubernetes Environments Local clusters like K3s, Minikube, Kind and MicroK8s compared for developing on Kubernetes, and why testing a change against the rest of the system needs ephemeral environments on a shared cluster. Arjun Iyer · May 1, 2025 ](https://www.signadot.com/articles/the-ultimate-guide-to-lightweight-kubernetes-environments/) - [ *Image: Why Signadot Outperforms Other K8s Testing Solutions* ### Why Signadot Outperforms Other K8s Testing Solutions How Signadot Sandboxes deploy only the changed services into a shared Kubernetes cluster, and what that means for test speed, cost, SmartTests, pull request testing and security. Arjun Iyer · April 30, 2025 ](https://www.signadot.com/articles/why-signadot-outperforms-other-k8s-testing-solutions/) No matching articles found. Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Resources Source: https://www.signadot.com/resources/ Summary: Whitepapers, guides, and downloadable resources on microservices testing. ### Resources Library Documents Videos Webinars [ *Image: Breaking the Staging Bottleneck: Scalable Microservices Testing for Fintech* Whitepaper #### Breaking the Staging Bottleneck: Scalable Microservices Testing for Fintech This whitepaper reveals how traditional staging breaks under modern FinTech complexity and introduces a faster, safer alternative: sandbox testing. Learn how leaders like Brex and DoorDash are scaling developer velocity and slashing costs with Signadot. Read more ](https://www.signadot.com/resources/breaking-the-staging-bottleneck-scalable-microservices-testing-for-fintech/)[ *Image: Cloud-Native Foundations 2026 Trend Report* Trend Report #### Cloud-Native Foundations 2026 Trend Report How teams run Kubernetes and platform engineering at scale, what slows software delivery down, where AI fits in operations, and how Bitso scaled delivery with coding agents. In partnership with DZone. Read more ](https://www.signadot.com/resources/dzone-2026-cloud-native-foundations-trend-report/) - [ *Image: Creating a Sandbox in your K8s Cluster with a Local Workload* ### Creating a Sandbox in your K8s Cluster with a Local Workload 6 min · August 28, 2023 ](https://www.signadot.com/videos/creating-a-sandbox-in-your-k8s-cluster-with-a-local-workload/) - [ *Image: Does context propagation cover and work with protocols like HTTP and GRPC?* ### Does context propagation cover and work with protocols like HTTP and GRPC? 1 min · August 28, 2023 ](https://www.signadot.com/videos/does-context-propagation-cover-and-work-with-protocols-like-http-and-grpc/) - [ *Image: Feature Previews using Sandboxes* ### Feature Previews using Sandboxes 10 min · August 28, 2023 ](https://www.signadot.com/videos/feature-previews-using-sandboxes/) - [ *Image: How do Sandboxes work* ### How do Sandboxes work 5 min · August 28, 2023 ](https://www.signadot.com/videos/how-do-sandboxes-work/) - [ *Image: How do you realize slice-based environments with asynchronous message queues?* ### How do you realize slice-based environments with asynchronous message queues? 4 min · August 28, 2023 ](https://www.signadot.com/videos/how-do-you-realize-slice-based-environments-with-asynchronous-message-queues/) - [ *Image: How to implement slice-based environments in Kubernetes* ### How to implement slice-based environments in Kubernetes 16 min · August 28, 2023 ](https://www.signadot.com/videos/how-to-implement-slice-based-environments-in-kubernetes/) - [ *Image: DZone Trend Talk: Cloud-Native Foundations* ### DZone Trend Talk: Cloud-Native Foundations Arjun Iyer joins TraceLink and GlobalLogic on DZone's cloud-native panel: multicluster drift, integration testing, per-PR environments and why verification is... Arjun Iyer · October 8, 2026 ](https://www.signadot.com/webinar-and-podcast/dzone-trend-talk-cloud-native-foundations/) - [ *Image: Rapid microservices development with Signadot* ### Rapid microservices development with Signadot While developing microservices locally is possible, running and testing them in a production-like Kubernetes environment is complex. A typical development... Peter Jausovec · October 31, 2024 ](https://www.signadot.com/webinar-and-podcast/rapid-microservices-development-with-signadot/) - [ *Image: Scaling developer testing for microservices in Kubernetes* ### Scaling developer testing for microservices in Kubernetes Join Anirudh Ramanathan, CTO at Signadot, in our latest webinar for a groundbreaking approach to microservice testing. Learn how to scale tests in Kubernetes... Anirudh Ramanathan · March 1, 2024 ](https://www.signadot.com/webinar-and-podcast/scaling-developer-testing-for-microservices-in-kubernetes/) - [ *Image: Rapid microservices development with Kubernetes and Signadot* ### Rapid microservices development with Kubernetes and Signadot Join Peter Jausovec as he delves into his journey with microservices development in Kubernetes, highlighting the pivotal role of Signadot. From the innovative... Peter Jausovec · October 23, 2023 ](https://www.signadot.com/webinar-and-podcast/rapid-microservices-development-with-kubernetes-and-signadot/) - [ *Image: Leveraging a Service Mesh to Scale Developer Environments : Signadot and CNCF* ### Leveraging a Service Mesh to Scale Developer Environments : Signadot and CNCF Anirudh shows how a service mesh like Istio routes requests so services under test can share one baseline environment, cutting the cost and spin-up time of dev... Anirudh Ramanathan · September 21, 2023 ](https://www.signadot.com/webinar-and-podcast/leveraging-a-service-mesh-to-scale-developer-environments-signadot-and-cncf/) - [ *Image: DevOps Toolkit Webinar: Mastering Local and Ephemeral Development with Kubernetes* ### DevOps Toolkit Webinar: Mastering Local and Ephemeral Development with Kubernetes See Viktor taking Signadot for a spin for local development and PR previews of microservices! Viktor Farcic · September 4, 2023 ](https://www.signadot.com/webinar-and-podcast/devops-toolkit-mastering-local-and-ephemeral-development-with-kubernetes/) - [ *Image: With Codefresh: Is the developer experience around microservices lagging?* ### With Codefresh: Is the developer experience around microservices lagging? Explore the challenges of using feature flagging for testing in shared staging environments and discover how tools like Signadot's ephemeral "Sandboxes" can... Nica Mellifera · August 3, 2023 ](https://www.signadot.com/webinar-and-podcast/with-codefresh-is-the-developer-experience-around-microservices-lagging/) - [ *Image: How Sandboxes Within Your Kubernetes Clusters Can Help Improve Developer Testing* ### How Sandboxes Within Your Kubernetes Clusters Can Help Improve Developer Testing In this episode, CEO and Co-founder at Signadot, Arjun Iyer joins DevOps Paradox hosts Darin Pope and Viktor Farcic to speak about how sandboxes within your... Arjun Iyer · June 21, 2023 ](https://www.signadot.com/webinar-and-podcast/simplify-microservice-development-with-signadot-devops-paradox/) - [ *Image: DevSec For Scale Podcast - Shift-Left Testing For Microservices* ### DevSec For Scale Podcast - Shift-Left Testing For Microservices In this engaging episode of the DevSec for Scale Podcast, we discuss how testing needs to shift left, particularly as development environments evolve with the... Arjun Iyer · September 28, 2022 ](https://www.signadot.com/webinar-and-podcast/devsec-for-scale-podcast-shift-left-testing-for-microservices/) Stay in the loop Get the latest updates from Signadot Validate code as fast as agents write it. [Sign up](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- # Company ## About Us Source: https://www.signadot.com/about-us/ Summary: Signadot's mission, team, and company information. ### Empowering developers with the freedom to be creative when building software at scale Developing complex applications does not need to be complex. We believe that the process of developing software must become significantly simpler than it is today for scaling teams and the impact that they have. Our team has first-hand experience and empathy from having experienced the challenges of developing software at scale. We founded Signadot to equip developers with superpowers, strike down complexity, and enable creative and collaborative problem solving for building the cloud-native applications of the future. #### Our mission & values ##### Teamwork We genuinely care about and respect our team members. We realize we win as a team and have fun along the way! ##### Think Big We love to think big and outside the box. We foster an environment where ideas are freely shared and discussed. ##### Bias towards action We are biased towards trying things and iterating fast. As a young startup we seek to accelerate learning. #### Globally connected, locally empowered We're proud of our fully distributed team that spans continents, time zones, and cultures. At Signadot, we don't just adapt to remote work, we thrive in it. We've deliberately built a borderless company that seeks out exceptional talent wherever they call home. Our collaborative culture bridges physical distances through open communication, psychological safety, and genuine human connection. We've found that great ideas don't require shared office space, just shared purpose and mutual respect. Every team member brings their unique perspective and local context, creating a richer problem-solving environment as we tackle complex challenges for developers worldwide. ##### Americas United States · Colombia · Uruguay ##### Europe France · Spain ##### Asia Pacific Philippines #### Our investors Supported by the best in the industry *Image: Redpoint Ventures logo* *Image: Y Combinator logo* - *Image: Jason Warner* Jason WarnerFormer CTOGitHub - *Image: Adam Gross* Adam GrossFormer CEOHeroku - *Image: Timothy Chen* Timothy ChenManaging PartnerEssence VC - *Image: Chris Golda* Chris GoldaFounding PartnerRogue Capital - *Image: Allison Pickens* Allison PickensGeneral PartnerThe New Normal Fund - *Image: John Kodumal* John KodumalCTO / Co-founderLaunchDarkly - *Image: Joseph Abebe* Joseph AbebeFormer Head of FinanceSlack - *Image: Martin Gontovnikas* Martin GontovnikasCo-founderHyperGrowth Partners - *Image: Sebastien Pahl* Sebastien PahlCo-founderDocker #### Media and press - [ DevOps.com ### Signadot Unveils Kubernetes-Native Developer Platform to Scale Agentic Development February 19, 2026 ](https://devops.com/signadot-unveils-kubernetes-native-developer-platform-to-scale-agentic-development/) - [ Techstrong TV ### Why Code Validation Is the Next Frontier March 31, 2026 ](https://techstrong.tv/videos/interviews/why-code-validation-is-the-next-frontier) - [ TechCrunch ### Signadot promises developers faster feedback loops February 23, 2022 ](https://techcrunch.com/2022/02/23/signadot-promises-developers-faster-feedback-loops/) - [ PR Newswire ### Y Combinator Alum Signadot raises $4M led by Redpoint Ventures to Reimagine Microservices Testing on Kubernetes February 23, 2022 ](https://www.prnewswire.com/news-releases/y-combinator-alum-signadot-raises-4m-led-by-redpoint-ventures-to-reimagine-microservices-testing-on-kubernetes-301488164.html) - [ Tomasz Tunguz ### The Next Key Innovation in the Shift-Left Movement February 25, 2022 ](https://tomtunguz.com/investing-in-signadot/) #### Take Signadot for a whirl Learn more about how to scale pre-merge testing with microservices [Start for free](https://www.signadot.com/signup/) [Book a demo](https://www.signadot.com/schedule-a-call/) --- ## Schedule a Call Source: https://www.signadot.com/schedule-a-call/ Summary: Book a demo or consultation with the Signadot team. ### Turn AI-generated code into shipped software See how engineering teams keep up with agent-speed development, without the babysitting, the production incidents, or the runaway infrastructure cost. Reclaim developer time Agents validate their own changes and keep iterating on their own, so your engineers stop babysitting coding agents. Ship with fewer incidents Shift-left validation that scales for the agentic era catches breaking changes before they reach production. Cut infrastructure cost Isolate only what each change touches instead of duplicating full environments for every test. *Image: Brex logo* $2M saved annually > Signadot for the preview environments use case fit what we were trying to do better than anything else. It was a more mature solution than the other stuff that we were looking at. And the return on the investment was obvious... just in infrastructure costs, it saves us about $2 million annually. *Image: Phil Burrows, Head of Platform Engineering at Brex* Phil BurrowsHead of Platform Engineering Not ready for a call? You can [start free in minutes](https://www.signadot.com/signup/) instead. Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering #### Frequently asked questions How is Signadot different from other solutions? Traditional ephemeral environments duplicate the entire infrastructure stack for every change. Signadot duplicates only what a change needs, usually the services it touched, and wires those copies into the shared environment, so each change gets what behaves like an entire environment of its own. You can run many of them in a single Kubernetes cluster, they spin up in seconds, and they cost a fraction of full copies. What capabilities does Signadot offer? Signadot provides ephemeral environments and the automated tests that run in them: \- Ephemeral environments (Sandboxes in the CLI and API): lightweight, isolated environments that duplicate only what a change needs, usually just the service you changed, on a shared baseline cluster \- Local Testing: develop and test microservices on your local machine while connecting to real dependencies in your Kubernetes cluster \- Preview Environments: preview every code change from web and mobile frontends \- Jobs: run your existing test suites against each ephemeral environment for every PR \- SmartTests: AI-powered contract testing that detects API breaking changes automatically \- Plans: declarative test workflows that agents and pipelines can run end to end What are Sandboxes? Sandboxes are Signadot's ephemeral environments: lightweight, isolated environments that let you test changes without interfering with other developers or the base environment. Unlike traditional approaches that duplicate entire infrastructure, Sandboxes duplicate only the components you're modifying, plus a database or queue when a change needs its own, and request and message routing wire those copies into everything else. Where do ephemeral environments run? Ephemeral environments run inside your existing Kubernetes clusters. Signadot installs as an operator in your Kubernetes environment, so all workloads, data, and testing happen within your infrastructure. The Signadot control plane orchestrates the creation and management of these environments, but your code and data never leave your environment. How much effort is it to get started? Getting started with Signadot typically takes less than 15 mins. The minimal setup involves just two steps: 1\. Installing the Signadot operator in your Kubernetes cluster. 2\. Using our CLI to create ephemeral environments and run tests. How does Signadot integrate with CI/CD systems? Signadot integrates seamlessly with all major CI/CD systems through our CLI, which can be used as a step in your pipeline. We have examples of integrations for GitHub Actions, GitLab CI, Jenkins, and BitBucket in our [documentation](https://www.signadot.com/docs/guides/integrate-ci). --- ## Sign Up Source: https://www.signadot.com/signup/ Summary: Create a free Signadot account. ### Validate code as fast as agents write it Sign up free and watch your first change run against real microservices in minutes. - Free, no credit card - No cluster of your own - ~5 minutes Local dev with your coding agent Claude Code, Cursor, and Codex plug in through our MCP server and CLI. A sandbox for every pull request Ephemeral, production-like, spun up automatically per PR. First sandbox in ~5 minutes On our playground cluster. [See the quickstarts](https://www.signadot.com/docs/overview). *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer [Read the story](https://www.signadot.com/case-studies/how-bitso-scaled-delivery-with-coding-agents/) Trusted by engineering teams worldwide - *Image: Brex logo* - *Image: Miro logo* - *Image: Wealthsimple logo* - *Image: Bitso logo* - *Image: Qlik logo* - *Image: SoFi logo* - *Image: Earnest logo* - *Image: Care.com logo* - *Image: InfoTrack logo* - *Image: Quizlet logo* - *Image: OnBoard logo* - *Image: Calendly logo* - *Image: Lightspeed logo* - *Image: Storyblocks logo* - *Image: OutSystems logo* - *Image: Luxury Presence logo* - *Image: Posh AI logo* - *Image: Juniper Square logo* - *Image: Leadr logo* - *Image: Nox Health logo* - *Image: Favor Delivery logo* - *Image: Laurel logo* - - - - - - - - - - - - - - - - - - - - - - Testimonials #### Hear from our customers *Image: Brex logo* $2M saved annually > On the margin, with the Signadot approach, **99.8%** of the isolated environment's infrastructure costs look wasteful. That percentage looks like an exaggeration, but it's really not. *Image: Connor Braa, Software Engineering Manager at Brex* Connor BraaSoftware Engineering Manager *Image: Bitso logo* 83% lower change failure rate > We basically stopped creating full preview environments and replaced our custom solution with Signadot. Instead of isolating the full environment, the strategy using routing keys is much lighter, and we are able to provide an isolated environment, even with isolated databases, per PR quite fast. *Image: Marcus Tavares, Staff Software Engineer at Bitso* Marcus TavaresStaff Software Engineer *Image: Wealthsimple logo* No more fighting over staging > Creating a sandbox is extremely fast, it works every time, lets me test what I'm building quickly, and move on. This is awesome. *Image: Tyler Marien, Senior Software Developer at Wealthsimple* Tyler MarienSenior Software Developer *Image: Miro logo* 10x integration capacity > Our engineers were spending more time waiting for an environment than shipping. Now they test against our real services in parallel, which made a whole tier of fixed test infrastructure unnecessary. Retiring environments we had carried for years is projected to save us **several million dollars annually**. *Image: Jony Jeyaratnam, Head of Engineering at Miro* Jony JeyaratnamHead of Engineering #### Frequently asked questions Is Signadot free to start? Yes. The free tier is open to every developer, with no credit card required, so you can try ephemeral environments and run your tests in them before you ever talk to us. Do I need my own Kubernetes cluster to try it? No. You can start on our hosted playground cluster in a few minutes. When you are ready, connect your own cluster by installing the Signadot operator. How do I connect my coding agent? Point Claude Code, Cursor, or Codex at our MCP server, or use the Signadot CLI. The agent can then spin up an ephemeral environment, run its change against real dependencies, and read the results on its own. How quickly can I see it working? About five minutes. Follow either quickstart to create your first ephemeral environment and watch your change run against real microservices. Where does my code run, and is it secure? Ephemeral environments run inside your own Kubernetes cluster. The Signadot operator installs in your environment, so your code and data never leave your infrastructure. --- # Documentation and External Resources These pages are part of the Signadot index but are served by the docs site or GitHub; fetch them directly: - [MCP Server](https://www.signadot.com/docs/integrations/mcp): Signadot's Model Context Protocol server, bundled with the Signadot CLI. Exposes tools for creating sandboxes and route groups and running validations, so MCP-compatible agents can test code changes against real dependencies. - [MCP Setup: Claude Code](https://www.signadot.com/docs/integrations/mcp/claude-code): Configure the Signadot MCP server in Claude Code. - [MCP Setup: Cursor](https://www.signadot.com/docs/integrations/mcp/cursor): Configure the Signadot MCP server in Cursor. - [MCP Setup: VS Code](https://www.signadot.com/docs/integrations/mcp/vscode): Configure the Signadot MCP server in VS Code. - [MCP Setup: Other Clients](https://www.signadot.com/docs/integrations/mcp/other-clients): Configure the Signadot MCP server in any MCP-compatible client. - [Coding Agents Overview](https://www.signadot.com/docs/integrations/coding-agents): How coding agents use Signadot to provision environments and validate their own changes. - [Agent Skills](https://www.signadot.com/docs/integrations/coding-agents/agent-skills): Installable skills (signadot-validate, signadot-plan) that teach coding agents Signadot validation workflows. Install with `npx skills add signadot/agent-skills`. - [Tutorial: Closed-Loop Validation with Claude Code](https://www.signadot.com/docs/tutorials/closed-loop-microservices-validation-claude-code): End-to-end tutorial where Claude Code implements a change and validates it in a Signadot sandbox. - [GitHub: closed-loop-comparison](https://github.com/signadot/examples/tree/main/closed-loop-comparison): Harness that runs Claude Code in Docker Sandboxes with and without Signadot agent skills on the same prompts, for side-by-side comparisons on HotROD or Google's microservices-demo. - [Tutorial: Closed-Loop Verification with Cursor](https://www.signadot.com/docs/tutorials/closed-loop-verification-cursor): End-to-end tutorial where Cursor validates changes against real services in a sandbox. - [Tutorial: Autonomous Validation with Codex](https://www.signadot.com/docs/tutorials/autonomous-closed-loop-codex): Autonomous closed-loop validation of code changes with OpenAI Codex. - [GitHub: signadot](https://github.com/signadot): Signadot's open source repositories, including the CLI, agent skills, and validation actions. - [GitHub: agent-skills](https://github.com/signadot/agent-skills): Source repository for the Signadot agent skills. - [GitHub: actions](https://github.com/signadot/actions): Community-extensible catalog of validation actions used by Signadot Plans. - [Docs llms.txt](https://www.signadot.com/docs/llms.txt): Index of the full technical documentation in llms.txt format, with markdown exports of every docs page. - [Docs Overview](https://www.signadot.com/docs/overview): Getting started guide and platform overview. - [Installation](https://www.signadot.com/docs/getting-started/installation): Install the Signadot Operator in your cluster and the Signadot CLI on your machine. - [Quickstart: First Sandbox](https://www.signadot.com/docs/tutorials/quickstart/first-sandbox): Create your first sandbox and test a change against real dependencies. - [Quickstart: Plan-Based Validation](https://www.signadot.com/docs/tutorials/quickstart/plan-based-validation): Run your first plan-based validation workflow. - [Sandbox Concepts](https://www.signadot.com/docs/concepts/sandbox): How sandboxes work. Lightweight isolated environments that virtualize your Kubernetes cluster by routing traffic to only the changed services. - [Smart Tests and Jobs Concepts](https://www.signadot.com/docs/concepts/smart-tests-and-jobs): How Smart Tests and Jobs execute test workloads against sandboxes, including AI-assisted diff analysis. - [Set Up Local Testing](https://www.signadot.com/docs/guides/use-cases/set-up-local-testing): Guide for connecting local development environments to your Kubernetes cluster via sandboxes. - [Set Up PR Sandboxes](https://www.signadot.com/docs/guides/set-up-pr-sandboxes): Guide for automating preview environment creation on every pull request. - [Run Automated Tests](https://www.signadot.com/docs/guides/use-cases/run-automated-tests): Guide for executing test frameworks (Playwright, Cypress, K6) against isolated sandboxes. - [Tutorial: Test Temporal Workers in Kubernetes with Sandboxes](https://www.signadot.com/docs/tutorials/testing-temporal-workers): Route Temporal workflow tasks to a sandboxed worker on a shared task queue, with Python, TypeScript, Go, and Java workers. - [Tutorial: Testing Kafka-Based Microservices in Kubernetes](https://www.signadot.com/docs/tutorials/testing-kafka-microservices): The three isolation strategies for testing Kafka-based microservices (cluster, topic, and message-level), how message isolation works with routing keys and consumer groups, the hard patterns like CDC and batch jobs, and a hands-on tutorial with sandboxes. - [Tutorial: Test Google Pub/Sub Flows End to End](https://www.signadot.com/docs/tutorials/testing-google-pub-sub-flows): Give each sandboxed subscriber its own Pub/Sub subscription and use a routing key in each message so it processes only its own messages on a shared topic. - [Tutorial: Testing Amazon SQS Microservices in Kubernetes](https://www.signadot.com/docs/tutorials/testing-sqs-microservices): Test SQS consumers on one shared queue: each consumer reads the routing key from the message's baggage attribute and releases messages that are not its own with ChangeMessageVisibility 0, with SNS fan-out and a queue per sandbox as options. - [Tutorial: Testing RabbitMQ Microservices in Kubernetes](https://www.signadot.com/docs/tutorials/testing-rabbitmq-microservices): Run baseline and sandboxed RabbitMQ consumers side by side on one broker, with a queue per sandbox and a routing key in the message headers. - [Tutorial: Testing Database Schema Changes in a Sandbox](https://www.signadot.com/docs/tutorials/testing-database-schema-changes): Give a schema change its own temporary MariaDB database through a resource plugin, test it with curl or the browser extension, and delete it with the sandbox. - [Tutorial: Load Testing Pull Requests with Grafana k6](https://www.signadot.com/docs/tutorials/k6-load-testing): Run k6 inside Kubernetes as Signadot jobs against a pull request's sandbox and the baseline, fail the run on p95 latency and error-rate thresholds, and scale across runner pods. - [Tutorial: Testing Apollo Federation Schema Changes](https://www.signadot.com/docs/tutorials/apollo-graphql-schema-testing): Test a federated GraphQL schema change per pull request with a GraphOS variant, a forked Apollo Router and the changed subgraph in a sandbox. - [Tutorial: Testing WunderGraph Federated GraphQL Schemas](https://www.signadot.com/docs/tutorials/wundergraph-schema-testing): Serve a pull request's subgraph schema from one shared Cosmo router through a feature flag named after the sandbox's routing key. - [Tutorial: Full-Stack Vercel Preview Environments](https://www.signadot.com/docs/tutorials/vercel-preview-environments): Wire each Vercel preview deployment to a backend sandbox built from the same pull request with GitHub Actions, so reviewers test frontend and backend changes together. - [Tutorial: How to Test a Feature That Spans Multiple Pull Requests](https://www.signadot.com/docs/tutorials/multi-pr-feature-testing): Combine the sandboxes of several pull requests into one preview with a route group, and automate it from GitHub Actions. - [Tutorial: Preview Backend Changes in a Mobile App](https://www.signadot.com/docs/tutorials/mobile-app-previews): Point an iOS app at a sandboxed backend with a routing header and compare the sandbox with the baseline. - [CI Integration](https://www.signadot.com/docs/guides/integrate-ci): Integration guides for GitHub Actions, GitLab CI, Jenkins, and BitBucket pipelines. - [CLI Reference](https://www.signadot.com/docs/reference/cli/overview): Command reference for the Signadot CLI. - [REST API Reference](https://www.signadot.com/docs/api): Full REST API specification for the Signadot control plane. - [Release Notes](https://www.signadot.com/docs/release-notes): Changelog of Signadot platform, operator, and CLI releases. - [AWS Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-d4g65y3cpqvvw): Subscribe to the Business plan self-serve, or accept an Enterprise private offer, and pay on your AWS bill.