Integration tests pass with mocks but staging still breaks: how to catch it earlier

When integration tests pass with mocks but staging still breaks, the gap is dependency fidelity, not test coverage. A mocked test verifies your assumptions about a dependency. Staging is the first place those assumptions meet the dependency itself, and the failures that surface there are exactly the ones a mock cannot represent.

Adding more mocked tests cannot close that gap. A mock encodes the dependency’s behavior on the day the mock was written, and nothing updates it when the real service changes. More mocked tests raise the coverage of your assumptions, not their accuracy.

This article covers why mocked integration tests miss what they miss, the escalation ladder of what to add, and how early each rung can realistically run.

Why do mocked integration tests miss staging failures?

A mock can only fail on what its author anticipated, and the failures that reach staging are the unanticipated ones. They cluster into four classes that live outside a mock’s reach:

• Mock drift. The mock matches the dependency as it was when the test was written. The dependency’s team ships changes on their own schedule, and no build step fails when the mock falls behind. Comparing the changed service’s responses against the stable version’s on the same requests is one way to surface it, an approach covered in shadow testing for microservice integration.
• Contract versus behavior. A mock can match the schema exactly and still return values the real service never produces: an empty list where production always has entries, a status the service stopped emitting a year ago.
• The network is absent. Timeouts, retries, serialization errors, and header propagation never execute in a mocked test, and each of those is a production failure class of its own.
• Deployed-only configuration. Connection strings, feature flags, resource limits, and auth config exist only in the deployed environment, so no in-process test can exercise them.

Integration failures are the failures between processes: data inconsistencies, latency behavior, fault tolerance. Unit tests cannot verify inter-service communication no matter how many of them exist, which is why a green pipeline says little about what happens when services meet.

The second-order cost shows up in how teams respond. When the automated suite stops predicting staging behavior, verification moves to manual checks against the staging environment, and the suite degrades into background noise. A suite with a stable count of failing tests stops being a signal at all: teams have run for months watching that the number of failures stayed constant rather than investigating them.

Shared staging then makes the feedback worse. Multiple teams deploy versions into the same environment at once, so test conditions change under your feet, and a failure could belong to any change that landed that day.

What should I add to catch integration bugs before staging?

The additions form a ladder. Each rung catches a failure class the one below it cannot, and each needs a higher-fidelity environment and more moving parts to run, so the question is how much fidelity your failure pattern actually demands. Simplest first:

1. Contract tests. A consumer and provider agree on the shape of their interaction, and each side verifies against the recorded contract without deploying the other. Pact is the most widely used implementation. This catches signature and schema drift in under a minute of pipeline time. It does not catch behavior, and it carries a standing cost: both sides have to write and maintain the contracts continuously, and sustaining that discipline across many teams is where the approach usually breaks down. The Pact comparison covers where the contract boundary ends.
2. Integration tests against real dependencies. Give the changed service an environment where its dependencies are real running services rather than mocks, then point the tests you already have at it. This rung catches behavior, routing, configuration, and serialization failures, the classes mocks miss. Something has to supply that environment, whether as a full copy of the stack or as an isolated slice of a shared cluster, and the next section compares those options.
3. Pre-merge end-to-end tests scoped to the changed service. Run the two or three user journeys the change touches, not the full suite. This catches cross-journey regressions and demands the most from both the environment and the suite, so scope is what keeps it viable.

How early can this realistically move?

All three rungs can run pre-merge. Contract tests already do so trivially, because they verify recorded contracts without deploying anything. The open question is rungs two and three, which need a per-change environment with real dependencies, available quickly enough to sit inside a PR pipeline. They are also the rungs where pre-merge matters most, because post-merge feedback lands in the same place staging failures already land.

Two architectures supply that. A full environment per PR deploys a complete copy of the stack for every open pull request. Routing on a shared cluster deploys only the changed service next to a single shared set of stable dependencies and steers tagged requests to it.

	Full environment per PR	Routing on a shared cluster
Cost per PR	One full copy of the stack	Only the changed service
Isolation	Complete, including data	Per-request, shared stable dependencies
Spin-up time	Minutes	Seconds
Prerequisite	Deployable manifests for every service	Header propagation across every hop

Release and Bunnyshell provide the full-environment pattern as a managed workflow. It is simpler to reason about, since each PR gets a fully independent copy, and below roughly ten services it is the right call. Teams that build the same pattern in-house tend to run ten or fifteen standing test environments and absorb a second cost the table does not show: keeping every copy in parity with production configuration, which is ongoing operational work rather than a one-time setup.

The crossover comes from arithmetic. Cost scales with the number of services multiplied by the number of open PRs, so a 40-service stack with 30 open PRs is provisioning 1,200 service instances to test 30 changes. Routing on a shared cluster flips the multiplier, because each PR deploys one service and shares the rest.

The shared-cluster pattern is what Signadot provides. A Sandbox is a lightweight ephemeral environment holding the changed service, deployed into the cluster where the stable versions of its dependencies already run. Requests carrying the Sandbox’s routing key reach the changed service, every other call resolves to the shared stable versions, and provisioning takes seconds because nothing else is duplicated. The prerequisite is header propagation: every service forwards the routing header on each hop, and teams already propagating trace context have most of that in place.

Shared staging gated by feature flags comes up in this comparison often, and it does not belong in it. Flags gate exposure after the code has merged, so nothing on the ladder moves pre-merge, and the flags add a cleanup obligation of their own. Whatever their value for release control, they are not a testing architecture.

How to choose

Two decisions come out of this article, and they are worth separating.

The first is which rungs to add, and the failure classes you actually see decide it. Shape mismatches between services point to contract tests, the cheapest rung and the one with no infrastructure requirement. Failures that only appear when services actually talk, wrong values, broken routing, misread configuration, point to the real-dependency rung, because no amount of contract or unit coverage reaches them. Recurring breakage in flows that span several services justifies the scoped end-to-end rung on the journeys that keep breaking.

The second is which architecture supplies the environment for those rungs. Stack size is the main input: below roughly ten services, a full environment per PR is simpler to reason about and the duplication cost stays tolerable. Above that, the services-times-PRs arithmetic takes over, and routing on a shared cluster is the option that keeps cost and provisioning time flat as the stack grows. Header propagation is the gate. If your services already forward trace context, the shared-cluster pattern is close to free to adopt, and if they do not, that plumbing is the price of entry.

The wider testing lifecycle around these stages is covered in Microservices Testing Environments on Kubernetes.

Frequently asked questions

My integration tests pass with mocks but staging still breaks. What am I missing?

The mocks encode assumptions about your dependencies, and staging is where the assumptions meet reality. The failure classes that get through are mock drift, behavior a schema-correct mock never produces, network effects, and deployed-only configuration. Adding a stage that tests the changed service against real dependencies is what closes the gap.

Should I delete my mocked tests?

No. Mocked tests are the right tool for logic inside a service, and they stay fast and deterministic. Stop asking them fidelity questions they cannot answer, and add a real-dependency stage for those.

Do I need a full copy of my stack to test against real dependencies?

No. A full copy per PR is one architecture, and below roughly ten services it is a reasonable one. Past that size the arithmetic turns: cost grows with the number of services multiplied by open PRs, and spin-up time tracks the slowest service in the stack. Routing on a shared cluster deploys only the changed service and resolves everything else to a single shared set of stable dependencies, which keeps cost and provisioning time flat as the stack grows. That is why it is the architecture that holds up for larger systems.

How do I know when mock drift is happening?

Indirectly: staging or production failures in interactions your suite says are covered. Directly: contract tests, which fail when a provider's actual interface no longer matches what consumers recorded. Comparing the changed service's responses against the stable version's on the same requests also surfaces drift. Mock drift itself produces no signal, which is the problem.