Coding Agents Need a Runtime to Check Their Work Against
Coding agents self-correct only when something tells them whether a change worked. In a cloud-native system, that signal comes from a running environment, not from local checks.
Every coding agent reaches the same moment. The code compiles, the unit tests pass, and the agent declares the task done. Whether it is done depends on something the agent rarely sees: how the change behaves inside the running system. That question is the hard part of verifying AI-generated code on Kubernetes.
Addy Osmani’s writing on agent harness engineering makes an argument worth taking seriously. An agent is a model plus the scaffolding around it: prompts, tools, context policies, execution sandboxes, subagents, hooks and feedback loops. That scaffolding often moves results more than a model upgrade does. Addy points to Viv Trivedy’s team, which moved a coding agent from the top 30 to the top 5 on Terminal Bench by changing only the harness.
The argument holds. What it leaves open is what happens when the agent is changing code in a cloud-native system, because one piece of that scaffolding stops working there.
Feedback is the piece that decides the outcome
The parts around an agent do different jobs. Prompts shape what it attempts. Tools like MCP servers and CLIs define what it can act on. Skills guide how it approaches a change. Policies and execution sandboxes fence in its actions.
Each of those influences behavior. None of them answers the question that ends the task: did the change work? That answer comes from feedback, and feedback is what turns an agent from an open-loop code generator into something that can check its own output, fix it and finish without a person in the middle.
The pattern that works on local software
For software that runs on one machine, the feedback recipe is well understood. Keep success quiet and make failure loud. A passing typecheck says nothing, while a failing one drops the error straight into the agent’s context so it can fix and retry.
Tests and linters follow the same shape. Hooks at lifecycle points like pre-commit and post-edit turn “the agent was asked to do X” into “the system enforces X.”
Anthropic’s work on long-running harnesses shows how far the pattern goes. A planner, a generator and an evaluator worked together, and the evaluator used Playwright to click through the running application the way a user would and grade it against criteria. Over multi-hour runs, the setup produced working full-stack applications, and the evaluator’s feedback carried the result.
Look at what made that possible. The whole application ran locally, so the question of whether it worked could be answered by starting the dev server and exercising it. Most production software does not offer that.
Distributed systems cut the feedback wire
A change to one microservice takes effect inside a web of real dependencies. It touches the databases it queries, the queues it publishes to, the auth and routing layers in front of it, and the services on either side. The signal that says the change works is the system’s behavior at runtime, and nothing on the agent’s machine contains that system.
A green unit test run covers a small slice of the verification surface. Integration, async flows, auth and tenancy, contracts and performance all sit outside it.
Teams reach for three workarounds, and each one cuts the loop in a different place.
- Mocks and stubs keep the loop fast, but they return whatever the developer or the agent wrote into them. The failures that matter in distributed systems come from real services interacting with real state, which a mock cannot reproduce.
- CI after the PR opens catches real failures, but the verdict arrives after the agent has finished. The working context is gone, a person is back in the loop, and a cycle of seconds has become hours.
- Full per-branch preview environments produce real signal at a price that stops scaling. Previews for human PR volume already strain budgets, and agent branch volume multiplies that.
So the standard setup walks the agent to “I believe this works” and leaves it there. A person takes over verification, which was the job the automation was meant to remove.
What an environment needs to serve an agent
The feedback has to come from a running instance of the system that the agent can reach from inside its loop. A mock of that system does not count, and neither does a pipeline that reports back after the session ends. Four properties decide whether an environment can be an agent’s feedback source rather than a person’s manual test bed.
- Fast and cheap. A full staging clone fails on both counts. The workable shape is a lightweight ephemeral environment that shares stable infrastructure and deploys only what the change touches. It should be ready in seconds, with a marginal cost tied to the services changed.
- Real. Requests route through the actual services, data stores and queues, with the agent’s build swapped in for the service it changed. The agent debugs the conditions production runs under.
- Programmable. The agent creates the environment, exercises it, reads the results and tears it down through a CLI, API or MCP server. Any step that needs a person breaks the loop before it starts.
- Isolated without duplication. Hundreds of agents verify at once on one shared cluster without stepping on each other. That takes an isolation model that forks only the changed services instead of copying infrastructure.
None of this is exotic. Platform teams have assembled these properties for human developers for years. The change for agentic development is wiring them into the loop the agent drives, instead of parking them beside it for people to consult at PR time.
What changes when the loop closes
With runtime feedback inside the loop, the agent can verify past its own process boundary. It can run integration paths across the real call graph. It can check contract behavior against live consumers and producers instead of generated stubs, and run against the actual database to catch the migration that breaks an unrelated query.
Engineers run those checks by hand today, after the agent finishes. Moving them inside the loop changes what a PR is. The diff arrives with evidence: traces showing the new code ran, integration results showing the calls between services still succeed, and metrics showing nothing else regressed.
Review then shifts from “does this work” to “should this ship,” which is a better use of a senior engineer’s attention. Closing the loop for coding agents in cloud-native systems walks through that shift with a demo.
The missing piece of the scaffold
The scaffolding argument is right, and feedback is the part that decides whether the rest of it pays off. The pieces that shape an agent’s first attempt have matured fast, and local feedback through types, tests and hooks is a solved problem. What teams building distributed systems still lack is feedback from a real running system, at the speed and concurrency agents work at.
That gap is what Signadot closes. Its lightweight ephemeral environments run on a shared Kubernetes cluster, deploy only the changed services and route the rest of each request to the shared stable versions. Agents get a programmable runtime to verify against inside the loop rather than after it. Signadot for agentic development shows how teams set that loop up.
