Software Factories: The Complete Guide for Cloud-Native Systems

A software factory is a system that turns intent into running software through many automated loops at once. Coding agents write the changes, harnesses constrain them, checks verify them and pipelines ship them, while engineers decide what gets built and own the result.

The idea is nearly sixty years old. In 2026 it names something specific: the agentic software factory, a delivery system where agents produce most of the code, so the hard question is no longer how fast the factory can write software. It is how quickly and how reliably the factory can tell whether that software works. That question is hardest on cloud-native systems, where what a change breaks lives between services.

Engineering leaders keep running into the same pattern. Faros AI’s Q2 2026 report, drawn from 22,000 developers, compared each organization’s lowest and highest periods of AI adoption. Epics completed rose 66.2% and incidents per pull request rose 242.7%, while deployments per week fell (Faros AI). More work goes in. Less comes out, and more of what comes out breaks.

A control engineer would recognize that curve. A software factory is a feedback loop: intent is the setpoint, agents are the controller, and verification is the sensor. Most organizations spent the past year turning up the gain, with more agents running with more autonomy, and left the sensor as it was.

A loop with high gain and a slow, biased sensor behaves the way that data does. It overshoots, it oscillates, and it spends more effort to deliver less. Agents made the controller cheap. Nobody made the sensor faster.

This guide defines the software factory and its parts, traces where the idea came from, and explains the dark factory and the levels of autonomy on the way to it.

It then uses that loop to show why so many factories are unstable, what a good sensor looks like when your product is dozens or hundreds of services, and how to decide how much autonomy your factory can safely run. For the hands-on mechanics of letting an agent test its own change, see the guide to verifying AI-generated code against real Kubernetes dependencies.

What is a software factory?

A software factory is the whole system that turns a queue of intent into deployed changes, with as little human handling per change as the organization can trust. It is not one tool. It is a set of loops, each with a job, connected so that work flows from a specification to production and signals flow back.

Addy Osmani gave the definition most people now work from: “A software factory is many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans owning the whole thing from above” (Osmani). The published designs, from StrongDM’s to OpenAI’s, share the same parts:

• Intent. Specifications, the center of spec-driven development, plus tickets, scenarios and acceptance criteria that say what done means.
• Queue. The backlog the loops draw from, increasingly written or split up by agents themselves.
• Harness. The sandbox, tools, context, rules and memory that constrain what an agent can do.
• Generation. The coding agents themselves: Claude Code, Codex, Cursor, GitHub Copilot and their peers.
• Verification. Static analysis, tests, scenario runs and runtime checks against real dependencies.
• Review gate. The point where a person, or a policy, decides a change may merge.
• Integration and deploy. CI, merge queues and progressive delivery.
• Production feedback. Telemetry, incidents and user signals that flow back into the queue. OpenAI automates this loop too: The Pragmatic Engineer describes a “Perf Factory” that monitors production and starts Codex agents to fix performance problems (The Pragmatic Engineer).

A software factory is not a CI/CD pipeline with a new name: a pipeline moves and checks code someone already wrote, and a factory also produces it. It is not one developer working with an agent in an editor, which is a tool rather than a system. And it is not necessarily lights out. Most factories running today keep people at the review gate, and most should.

You will also see it called an AI software factory or an agentic software factory, and the process it runs called the agentic software development lifecycle (agentic SDLC) or AI SDLC. At the far end of autonomy it becomes a dark factory.

Software factory vs CI/CD pipeline

A CI/CD pipeline is one stage inside a software factory. The pipeline builds, tests and ships a change. The factory adds everything upstream of it, from intent to generation, and the feedback that runs from production back into the queue.

The difference shows up when agents arrive. A pipeline sized for the rate at which people write changes starts receiving them at the rate agents write them. Anthropic reported that its CI job volume increased 25 times in six months (Anthropic).

Software factory vs harness engineering

Harness engineering builds the controls around one agent loop: the context, tools, rules and checks that keep a single agent on track. A software factory runs many of those loops against one shared queue and one shared production system.

That raises questions a single harness never meets: how changes from different loops interact, what verifies them, and how fast. Birgitta Böckeler’s description of a harness as guides and sensors is the right vocabulary at both levels, and the rest of this guide leans on it (Böckeler).

Software factory history: from 1968 to DoD software factories

The phrase is older than most of the people now using it. It has meant four different things, and each version put verification in a different place.

Era	Who	What "factory" meant	Where verification sat
1968 to the 1980s	Bob Bemer's proposal, then Hitachi (1969), SDC (1975) and Toshiba (1977)	Standard process, reusable designs and code, tools and measurement, so programming could be managed like production	Quality and process control at every stage
2004	Microsoft architects Jack Greenfield and Keith Short	Product lines built from patterns, models, frameworks and tools that generate much of an application	In the models and frameworks, checked by people
2017 onward	The U.S. Air Force's Kessel Run, then the DoD's Platform One	A DevSecOps organization: "a system of people, process, and technology built to continuously deliver valuable software that warfighters love"	Pipelines, automated tests, hardened containers, continuous authority to operate
2026	Dan Shapiro, StrongDM, Addy Osmani, Dex Horthy, OpenAI	Many agent loops fed by a queue of intent and drained through gates into production	Contested: review, tests, behavioral twins, runtime checks

The thread through all four is the production step. Bemer’s 1968 proposal already called for measures and controls for quality, and the Japanese factories that followed put process control at the center, as Michael Cusumano documented at MIT (Cusumano). Agents changed the production step itself. What still resists is the one every version of the factory depended on: knowing whether what came off the line is right.

The most recent pre-agent version left a warning. Bryon Kroger, a co-founder of Kessel Run, wrote that many defense software factories slid into “innovation theater” and were “measuring the wrong things or worse, not measuring at all” (Kroger). An agentic factory that counts pull requests and tokens is set up to repeat that.

What is a dark factory? The five levels of AI coding

A dark factory, or dark software factory, is a software factory in which no person writes or reviews the code. Specifications go in and working software comes out. The name comes from lights-out manufacturing. Shapiro points to FANUC, whose robot plant near Mount Fuji was already building about 50 robots a shift and running unattended for up to 30 days in 2003 (Business 2.0).

Dan Shapiro’s five levels, published in January 2026 and amplified by Simon Willison, borrow the levels of driving automation (Shapiro). Like the driving scale, they count from zero:

• Level 0. Nothing reaches the disk without your approval. AI is a search engine or an occasional autocomplete.
• Level 1. You hand discrete tasks, such as a unit test or a docstring, to an AI intern.
• Level 2. You pair with the AI like a junior colleague. Shapiro puts most self-described AI-native developers here.
• Level 3. The agent writes the code and you manage it, which means reviewing diffs all day. Shapiro says almost everyone tops out here.
• Level 4. You write the spec, review the plan, leave, and come back to see whether the tests pass.
• Level 5. The dark factory: “a black box that turns specs into software.”

Read the levels by what verifies the code at each one and the sticking point is plain. At level 3 the person is the sensor, reading every diff, and a person saturates. At level 4 the tests become the sensor, so whether level 4 is safe depends on what those tests run against. The levels describe how much the agent does. Underneath, they are a ladder of sensors.

The best-known dark factory is StrongDM’s. In February 2026 its CTO, Justin McCarthy, described how a team of three engineers, formed in July 2025, builds software under two rules: “Code must not be written by humans” and “Code must not be reviewed by humans” (StrongDM).

People write specifications and scenarios. Agents write the code and check it against what StrongDM calls a Digital Twin Universe: behavioral clones of the third-party services its product depends on, including Okta, Jira, Slack and Google Docs, Drive and Sheets. The company’s own summary is that “validation replaces code review” (StrongDM blog).

The rules got the attention. The world they built to test in is what made the rules safe. A twin universe that agents can run thousands of scenarios against is a sensor fast enough and faithful enough to stand in for a person reading the diff. Plenty of teams copied the rules. Far fewer built the sensor.

The counterexample came from Dex Horthy of HumanLayer. At the AI Engineer World’s Fair in June 2026, he described taking HumanLayer’s own product fully lights off in July 2025, and the failure anyone who runs that way for months eventually meets: the site goes down, the agent cannot fix it, and the answer is buried in code nobody has read in three months (Horthy).

His diagnosis is about loop speed. Checking that code runs and its tests pass is quick. The cost of bad architecture shows up over months and years, which makes maintainability far harder to verify. His answer was to turn the lights back on: put code review back, and make it cheaper by settling design and plans before the agent writes code.

Between those two sits the factory most enterprises will run for years: lit, with people at the gate, and dark in specific corners. A class of change such as dependency upgrades or new tests can run lights out once the sensor for that class is good enough. The useful question is never whether the whole factory goes dark. It is which corners have a sensor you trust.

A software factory is a control system

Every software factory is a feedback loop, whether anyone designed it as one or not. Naming its four parts turns the debate about whether people should still read code into a question you can measure: is the loop stable?

Intent is the setpoint: specifications, tickets, acceptance scenarios and SLOs. Coding agents and their harnesses are the controller, and how much they change without a person, autonomy multiplied by concurrency, is the gain. Your running services, data stores and queues are the system under control. Whatever reports how that system now behaves, from tests and environments to review and production telemetry, is the sensor.

Open loop vs closed loop

Strictly, no software factory runs open loop. A toaster does: it heats for a set time and never checks the bread. A factory always closes the loop somewhere, because production reports eventually. The useful question is which loop closes against the real system, and whether it closes before the controller acts again.

Most factories close the agent’s loop against unit tests and mocks, which are a model of the system rather than the system. The loop around the real system closes only after merge, when CI, staging or production finally reports. By then the controller has moved on to the next change, and so have the forty agents beside it.

None of this vocabulary is new. Böckeler describes a harness as guides and sensors. The Thoughtworks Technology Radar put feedback sensors for coding agents in its Trial ring in April 2026. Marc Brooker of AWS has described agents as the move from open-loop generation to feedback loops (Brooker).

Osmani calls the limit on autonomy back pressure, and Andrej Karpathy told Y Combinator’s AI Startup School in 2025 that we have to keep AI on the leash and make the loop between AI generation and human verification as fast as possible (Karpathy). What the metaphor has lacked is the engineering that comes with it. A sensor has measurable properties, and the stability of a loop depends on them.

How an AI software factory changes the SDLC

A software factory changes the SDLC in two places. It adds the generation step, so every stage after it receives changes at the rate agents write them, and it turns verification from a stage people wait at into a service agents call from inside their own loop. Everything else was already there.

Software delivery has always been a set of nested loops, each closed by a different sensor at a different speed. The inner loop is edit, build and run, closed by a compiler and unit tests in seconds. The pull request loop is closed by CI and review in minutes to hours.

The release loop is closed by staging, QA and canaries in hours to days. The production loop is closed by telemetry and incidents over days to weeks. The intent loop, where product and architecture decisions get corrected, runs over weeks to quarters.

People hid how slow some of those sensors were, because people were slow too.

A developer who spends two days on a change hardly notices that staging takes four hours to report, because the report arrives long before the next change is ready. An agent that writes the same change in four minutes notices every one of those hours. It either waits, which throws away the speed you bought it for, or it keeps going on what it can see, which is unit tests and mocks.

In most factories the agent’s loop is now the fastest loop in the building, while the sensor that reports on real services still sits in the pull request loop or the release loop. The fastest controller you have is wired to one of your slowest sensors.

The slowest loop matters too. Horthy’s point about architecture is a point about loop speed: no test fires when a design starts to rot, so the only sensor is people looking at the system over months.

That gives people their place in a software factory. They belong on the intent, writing specifications and scenarios, and on the slowest loops, where judgment is the only sensor there is. They do not belong as the per-change sensor for a controller that runs a hundred times faster than they read.

What the data shows: gain without sensing

The industry data is what you would expect from a loop whose gain went up while its sensor stayed put. DORA’s 2025 research found that AI adoption “is associated with an increase in both software delivery throughput and software delivery instability,” (DORA), and the report describes AI’s main role as an amplifier of an organization’s existing strengths and weaknesses (DORA 2025).

Throughput and instability rising together is the signature of a high-gain loop with a weak sensor.

Measured	Initial adoption, Q2 2026 report	Deepening adoption, Q3 2026 report	What it says about the loop
Incidents per pull request	Up 242.7%	Up 14.5%	Failures reaching production, then settling per change
Pull requests merged with no review	Up 31.3%	Up 76.3%	The human sensor saturated, then bypassed
Time in QA	Not reported	Up 300.6%	Readings queuing behind the sensor
Median time in review	Up 441.5%	Still elevated, no figure given	Reviewers saturated
Pull request size	Up 51%	Up 71.8%	Each change touches more of the system
Code churn, lines deleted per line added	Up 861%	Fell from its peak, no figure given	Changes corrected, then corrected again
Deployments per week	Down 11.7%	Reversed direction, no figure given	Effort that never reached the output
Monthly incidents	Up 57.9%	Up 125.4%	More failures in total, even as each change got safer
Epics completed	Up 66.2%	Not reported	The controller's output went up

Both columns come from Faros AI, which sells engineering analytics, so the figures are correlations across its customers rather than a controlled study. The Q2 2026 report compared each organization’s lowest and highest periods of AI adoption (Faros AI). The Q3 2026 update looked at teams deepening adoption they already had (Faros AI).

Read side by side, the second column is not a loop settling. Incidents per change came down while pull requests merged with no review rose 76.3% and time in QA rose 300.6%. Less sensing happens per change, and what remains waits in a longer queue. That is a sensor being bypassed and backed up, not improved.

The same pattern shows up from the other direction. CircleCI’s 2026 State of Software Delivery found average throughput up 59%, with the top 5% of teams nearly doubling theirs and the bottom 25% seeing no improvement at all, while roughly 30% of merge attempts failed (CircleCI). Same agents, different sensors.

At Bitso, more than 250 engineers and their coding agents work across over 200 microservices. After the company replaced its broken shared staging with per-change ephemeral environments, deployments rose from 664 to 1,400 a week while change failure rate fell from 5.2% to 0.9% (Bitso case study). Gain went up and the loop got more stable, because the sensor scaled with it.

The three properties of a good sensor

A sensor is judged on three properties, and you can measure each one this quarter. Latency is how long it takes to report. Fidelity is how closely its reading matches the real system. Bandwidth is how many readings it can take at once. A factory that is weak on any one of them closes the loop too late, or against the wrong thing, for every change that property misses.

Latency: how long until the factory knows

Delay is the classic way to make a stable loop unstable. The longer a signal takes to arrive, the more actions are taken before it does, and each of them is based on a state that no longer exists. Past a certain delay a loop cannot settle at all: every correction arrives late and overshoots, and the swings get wider instead of smaller.

A factory with slow feedback behaves the same way. Every action taken between a change and its signal is taken on stale information. The agent retries on top of a change nobody has verified, other agents build on top of that, and when the signal finally arrives it describes a system that no longer exists. A sensor that takes a day to report is not a sensor. It is a postmortem.

The number to track is time to signal: the minutes from a change being ready to a verified result against real dependencies. For an agent, the target is inside its own loop, measured in minutes, not after the pull request in hours.

Fidelity: whether the reading matches the real system

A biased sensor is worse than a slow one, because the loop trusts it. A thermostat mounted next to the oven will hold the kitchen steady at the wrong temperature. Unit tests and mocks are a sensor mounted next to the code. They report what a change does against a model of its dependencies, and the model is only as current as whoever last wrote it.

Agents make the bias worse, because they optimize for whatever the sensor rewards. METR found frontier models modifying tests and scoring code to raise their scores (METR), and Anthropic’s system card for Claude 3.7 Sonnet reported the model occasionally special-casing tests to make them pass (Anthropic).

StrongDM met the same thing in practice: its agents rewrote code to pass narrow tests, so the team moved its checks into scenarios kept outside the codebase, like a holdout set in model training.

Developers see the milder version every day. In the 2025 Stack Overflow Developer Survey, 66% named “AI solutions that are almost right, but not quite” as their top frustration (Stack Overflow). In a distributed system, almost right is the change that passes its unit tests and fails at a service boundary.

The number to track is escape rate: of all the failures you find, the share found after merge, in CI on the main branch, in staging or in production. A falling escape rate means the pre-merge sensor is getting more faithful.

Bandwidth: how many changes can be measured at once

A shared staging environment is a sensor that can take one reading at a time. When a few people merged a few changes a day, that was enough. With fifty agents, readings queue, or worse, they mix. Several changes land in the same environment, a test fails, and nobody can say which change caused it. A sensor that cannot attribute an error to an input cannot drive a correction, so the loop cannot close on that change.

Agents raise demand on this property fastest. Every attempt in a build, test and fix loop wants its own reading, and loops run in parallel by design. The number to track is concurrent verification capacity: how many changes can be checked against real dependencies at the same time, without waiting for each other or interfering with each other.

In most setups the three properties trade against each other. Mocks are fast and parallel but unfaithful. Shared staging is faithful but slow and single file. A full copy of the stack per change is faithful and parallel, but its cost grows with every agent you add. Getting all three at once is the design problem at the center of a software factory, and the way to stop trading them, covered below, is to duplicate only what each change needs and share the rest.

Why cloud-native systems make the sensor hard

In a distributed system, the behavior that breaks lives between services, so a faithful sensor has to include the services around a change and not only the change itself. That makes runtime verification, checking how a change behaves among the real services around it, the hardest part of a cloud-native software factory.

Look at what tends to fail when an AI-generated change reaches staging. A field renamed in one service that another team’s service still reads. A retry that is harmless alone and multiplies load across a real call chain. A consumer that assumes ordering on a Kafka topic that does not guarantee it. A configuration or network policy that exists only in the cluster. Each change is correct at its own boundary and wrong in combination.

Pull requests grew 51% larger with AI use in Faros’s first report and 71.8% in its second, so each change touches more of the system than it used to. CodeRabbit’s study of 470 open source pull requests found AI-co-authored ones carried about 1.7 times as many issues as human-only ones (CodeRabbit).

StrongDM’s design concedes the point. Its agents do not test against mocks. StrongDM built behavioral clones of the outside services its product touches, because the agents needed a world to run in.

That works for third-party SaaS APIs with stable public behavior. For your own 150 internal services, a clone is a second implementation that drifts from the first, while the real thing is already running in your staging cluster. The guide to environment parity covers how that drift happens and how to catch it.

Ways to build a software factory’s sensor

There are seven common ways to build the sensor, and each makes a different trade between latency, fidelity and bandwidth. Most factories use several, at different loops.

Unit tests and mocks report in seconds and run in parallel without limit, but fidelity stops at the service boundary and agents learn to satisfy them. The real cost of writing mocks adds up quickly.

Contract tests, with tools such as Pact and Specmatic, catch interface drift between teams and miss any behavior the contract never described, as the complete guide to contract testing explains.

Service virtualization and behavioral twins, from WireMock and Hoverfly to agent-built clones like StrongDM’s, are fast and cheap, with fidelity bounded by how well the twin models the real thing. They suit third-party APIs you do not control.

Agent sandboxes, such as the GitHub Actions environment behind Copilot’s coding agent or Codex’s cloud containers, run agent code safely but usually hold only the repository, so they are a place to run a sensor rather than a sensor. The Kubernetes sandbox explainer separates the two meanings of the word.

Shared staging is faithful when someone maintains it, hours behind a queue, and able to take one reading at a time, as the complete guide to staging environments covers.

Ephemeral environments that duplicate only what a change needs give each change an environment of its own without copying the whole stack, and you choose how much each one duplicates.

At one end, everything is duplicated: a namespace, virtual cluster or whole cluster per pull request, faithful and fully isolated, but minutes to start and more expensive with every concurrent agent. At the other end, only the components that changed are duplicated, and everything else is the stable shared version already running. In between, you duplicate the changed services plus the few dependencies the change needs to own, such as a database branch for a schema change.

Whatever the level, three things make it work. Some parts of the environment are duplicated. The duplicated parts take part in the application’s flows: request routing sends synchronous calls to the change’s own version, message routing does the same for asynchronous consumers on Kafka or a queue, where there is no request to route, and some components only need a configuration change to point at the right copy. And the result is verified end to end, against everything else that is real.

The less you duplicate, the faster an environment starts and the more of them run at once, so hundreds can share one cluster. The price is plumbing: every hop and message has to carry the change’s routing key, and each stateful dependency needs its own isolation set up. Lyft and Uber built in-house versions of the lightest end, and isolation models for ephemeral environments compares the two extremes.

Production signals, from canaries and feature flags to telemetry, give the most faithful reading there is, after a change has reached users. They close the production loop and cannot stand in for a check before merge.

Sensor	Time to signal	Fidelity	Changes at once	Best fit
Unit tests and mocks	Seconds	Low, stops at the service boundary	Unlimited	Logic inside one service, on every attempt
Contract tests	Seconds to minutes	Medium for interfaces, none for behavior	Unlimited	Interface drift between teams
Twins and service virtualization	Seconds	Bounded by the twin	High	Third-party APIs you do not control
Agent sandboxes	Seconds to minutes	Low to medium, dependencies usually absent	High	Running agent code safely
Shared staging	Hours, behind a queue	High when maintained	One	Release checks and demos
Ephemeral environment, whole stack duplicated	Minutes	High	Limited by cost	Small stacks, or changes that need the whole system to themselves
Ephemeral environment, changed parts plus their data	Seconds to minutes	High, with its own data	High	Schema and data changes
Ephemeral environment, only what changed	Seconds	High, real dependencies	Hundreds or more	Many services and many agents, on every change
Production signals	After deploy	Highest	Live traffic	The production loop

Signadot, a verification platform for cloud-native applications, gives every change an ephemeral environment without copying the stack. It installs as a Kubernetes operator on the cluster you already run. Each change duplicates only what it needs, from the modified services alone up to those services with their own databases or queues, and Signadot wires those copies into everything else, so to an agent or a developer it behaves like an entire environment of its own.

Requests reach the change’s version through the mesh you already run: Istio natively, the Gateway API, or a DevMesh sidecar with Linkerd or no mesh. Messages stay on the shared broker and carry the change’s routing key, so each consumer version takes only its own, with guides for Kafka, RabbitMQ, NATS, SQS and SNS, and Google Pub/Sub. A database can stay shared with each environment’s data kept apart, or a resource plugin can give an environment its own temporary database or a branch from Neon or Xata.

Existing test suites, such as Playwright, Cypress or k6, run inside those environments as Jobs, and Plans package the checks a change must pass so an agent can run them by name. Agents create and test against these environments through the CLI, the MCP server and the signadot-validate skill, which is what puts the sensor inside the agent’s own loop.

When the product itself is an agent

A factory that builds agents needs a sensor that is statistical and realistic at once. Agent output varies from run to run, so one passing run proves little. Agent tools call real services, so a mock of those services hides the failures that matter.

LangChain’s State of Agent Engineering survey of 1,340 practitioners found 57.3% running agents in production, with quality the biggest barrier, cited by about a third. Of those surveyed, 52.4% ran offline evaluations on test sets, and 37.3% ran online evaluations (LangChain).

Offline evals grade what the model says. They do not exercise the services its tools call. Each change to an agent-based product needs many runs against realistic dependencies, which multiplies the bandwidth the sensor needs. The guide to testing MCP servers and how Miro tests its agentic features show what that takes in practice.

What AI coding agents need from a software factory’s sensor

Coding agents are both the source of the factory’s new volume and the heaviest users of its sensor, so the sensor has to work on their terms. Six requirements follow from that.

1. It has to keep up with agent volume. CircleCI’s average throughput gain of 59% understates what one team feels, since the top 5% of teams nearly doubled theirs. Test infrastructure sized for the pull requests a team of people opens in a day receives a multiple of that once agents open them too.
2. It has to be callable from inside the loop. An agent in a build, test and fix loop cannot file a ticket for a test environment and wait. Claude Code, Codex, Cursor and GitHub Copilot need to create an environment, test against it and tear it down through a command line or an MCP tool, in the same session that wrote the change.
3. It has to run before review. Testing AI-generated code against real dependencies before a person opens the pull request means the change arrives with evidence, so the reviewer judges intent and design instead of re-testing behavior. A 441.5% rise in median time in review is what happens when the reviewer is the sensor.
4. It has to take one reading per change. Each change needs its own ephemeral environment. Batching agent changes into one environment to save money makes failures unattributable, and an agent that cannot tell which change failed retries blind.
5. It has to clean up after itself. A loop that runs all night can create hundreds of environments. Time-to-live limits and automatic teardown tie the bill to work in progress, not to forgotten experiments.
6. It has to report in a form an agent can act on. A specific failing request, response diff or log line lets the agent fix the cause. A red build with no detail sends it guessing, which is how loop costs multiply.

Watching Claude Code verify a microservices change shows what one closed loop looks like, start to finish.

How much autonomy can your software factory run?

As much as its sensor can verify, and no more. Osmani puts it as a rule: “Back pressure is the rule that you can only hand a loop as much autonomy as you can cheaply and reliably verify, and not one inch more.” You set autonomy by what you can measure, and you raise it only when the measurement improves.

The sensor you have	The autonomy it supports	Shapiro's level, roughly
Compiles and unit tests only	The agent proposes, and a person verifies and merges every change	0 to 2
Contract tests and CI, with real dependencies only in shared staging after merge	The agent opens pull requests, and a person reviews both the diff and the behavior	3
Per-change checks against real dependencies in minutes, inside the agent's loop	The agent iterates until the change is verified, and a person reviews the intent and the evidence	4
All of the above, plus held-out scenarios, behavioral diffs, production feedback and fast rollback	Defined classes of change merge without review	5, for those classes

The level a factory can safely run at is set by its sensor, not by its model. Three objections come up every time.

“Better models will fix this.” A better controller still needs a sensor. Models learn to satisfy whatever the checks reward, as the reward-hacking results above show, so a stronger model attached to a biased sensor reaches the wrong answer faster and more convincingly.

“Put human review back on every change.” Review is a sensor too, with latency in hours, a bandwidth of a few changes per reviewer per day, and fidelity limited to what a person can infer from a diff. Use people for the setpoint and the slow loops. Do not ask them to be the per-change sensor for a hundred agents.

“This is just a metaphor.” Then test it. If the model is right, improving time to signal, escape rate and concurrent verification capacity should lower churn and incidents without lowering throughput. If it is wrong, those numbers will improve and the instability will stay.

How to measure a software factory

Measure the sensor and the output, not the agents’ effort. Pull requests per engineer and tokens per engineer count how hard the agents are working.

StrongDM’s McCarthy offers a benchmark for the second: “If you haven’t spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement.” Token spend is a fine input to watch. As a measure of the factory, it rewards gain and ignores the sensor, which is why the Thoughtworks Radar puts coding throughput as a measure of productivity in its Caution ring.

Metric	What it tells you	Loop property
Time to signal per change, median and 90th percentile	How stale the agent’s information is	Sensor latency
Escape rate, the share of failures found after merge	Whether the pre-merge sensor matches reality	Sensor fidelity
Concurrent verification capacity, and time spent queued	Whether readings wait for each other or mix	Sensor bandwidth
Verified changes deployed per week	What the factory delivers	Output
Change failure rate	Whether the output holds up in production	Stability
Rework rate, code rewritten within weeks of merging	Whether the loop is oscillating	Stability
Review time per pull request	How much sensing still falls on people	Human load

Start with the first three. They are cheap to collect, and they tell you which property is the weak one.

Software factory best practices checklist

Twelve questions worth answering yes to. Each no is a place where the loop closes too late.

• Intent is explicit. Does every work item carry acceptance criteria or scenarios an automated check can test against?
• Signal arrives inside the loop. Does an agent get a result against real dependencies in minutes, before it opens a pull request?
• The sensor sees real dependencies. Do pre-merge checks exercise the services, data stores and queues around a change, not only mocks of them?
• One reading per change. Does each change get its own isolated environment rather than sharing a batch?
• Capacity matches concurrency. Can you verify as many changes at once as you run agents, without a queue?
• Evidence travels with the pull request. Does every agent pull request link to the checks it ran and their output?
• Reviewers judge intent. Has review moved from re-testing behavior to judging design, risk and intent?
• Autonomy follows the sensor. Does each class of change get only the autonomy its sensor supports, raised only when the sensor improves?
• Some scenarios are held out. Are some checks kept out of the agent’s view, so it cannot tune a change to them?
• Environments expire. Do per-change environments have time-to-live limits and automatic cleanup?
• Production feeds the queue. Do incidents and regressions turn into new scenarios and work items?
• The scorecard leads with the sensor. Do you report time to signal, escape rate and verification capacity next to throughput?

Where to go next

If your agents open more pull requests than you can test, start with why AI-generated code needs proof before review and then measure your time to signal on the last twenty changes. If shared staging is the queue every change waits in, the complete guide to ephemeral environments on Kubernetes compares the ways to give each change its own.

If you are deciding how far toward a dark factory to go, pick one class of change, measure its sensor on all three properties for a month, and raise its autonomy only where the numbers support it. Coding agent environments show how Signadot fits into that loop.

References

Sources cited in this guide, grouped by topic.

The software factory and the dark factory

• Addy Osmani, “Software Factories, Light and Dark,” July 20, 2026.
• Justin McCarthy, “Software Factories and the Agentic Moment,” StrongDM, February 6, 2026.
• StrongDM, “The StrongDM Software Factory: Building Software with AI,” February 2026.
• Dan Shapiro, “The Five Levels: from Spicy Autocomplete to the Dark Factory,” January 23, 2026, and Simon Willison’s commentary, January 28, 2026.
• Dex Horthy, “Harness Engineering Is Not Enough: Why Software Factories Fail,” AI Engineer World’s Fair, June 30, 2026.
• Gergely Orosz, “Inside OpenAI’s agentic software factory,” The Pragmatic Engineer, September 15, 2026.
• Birgitta Böckeler, “Harness engineering for coding agent users,” martinfowler.com, April 2, 2026.
• Thoughtworks Technology Radar Vol. 34, “Feedback sensors for coding agents” and “Coding throughput as a measure of productivity,” April 15, 2026.
• Marc Brooker, “What’s Easy Now? What’s Hard Now?,” May 18, 2026.
• Andrej Karpathy, “Software Is Changing (Again),” Y Combinator AI Startup School, June 2025.

History

• Michael A. Cusumano, “Factory Concepts and Practices in Software Development: An Historical Overview,” MIT Sloan School of Management working paper, 1989, and “Japan’s Software Factories: A Challenge to U.S. Management,” Oxford University Press, 1991.
• Jack Greenfield and Keith Short, “Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools,” Wiley, 2004.
• Greg Hadley, “Newest Air Force Software Factory Aims to Help AFRL, Attritable Aircraft Programs,” Air & Space Forces Magazine, January 20, 2022.
• U.S. Army Futures Command, “AFC announces ‘Software Factory’ in Austin,” army.mil, July 2020.
• Bryon Kroger, “Defense Software Factories Need a Major Reset,” Defense Opinion, October 13, 2025.
• Christopher Null and Brian Caulfield, “Fade to Black,” Business 2.0, June 1, 2003.

Delivery and quality data

• Faros AI, “The AI Engineering Report 2026: The AI Acceleration Whiplash,” April 2026.
• Faros AI, “The Speed Trap: 8 takeaways from Faros’s latest AI engineering research,” September 18, 2026.
• DORA, “Balancing AI tensions: Moving from AI adoption to effective SDLC use,” March 10, 2026, and the 2025 State of AI-assisted Software Development report.
• Sachin Malhotra, “Agentic coding is straining CI. Here’s how we scaled test impact analysis at Anthropic,” Anthropic, September 14, 2026.
• CircleCI, “2026 State of Software Delivery.”
• Stack Overflow, “2025 Developer Survey: AI.”
• David Loker, “State of AI vs Human Code Generation,” CodeRabbit, December 17, 2025.
• LangChain, “State of Agent Engineering,” survey conducted November to December 2025.
• Sydney Von Arx, Lawrence Chan and Beth Barnes, “Recent Frontier Models Are Reward Hacking,” METR, June 5, 2025.
• Anthropic, “Claude 3.7 Sonnet System Card,” February 2025.

Duplicating only what changed, in practice

• Lyft Engineering, “Scaling productivity on microservices at Lyft, part 3: extending our Envoy mesh with staging overrides.”
• Uber Engineering, “Simplifying developer testing through SLATE.”

Related reading

• Agent Loops Don't Have a Token Problem. They Have a Feedback Problem.
• Coding Agents Are Only as Good as the Signals You Feed Them
• Closing the Loop for Coding Agents in Cloud-Native Systems
• Agentic Coding Tools Are Accelerating Output, Not Velocity
• Ramp's Inspect shows closed-loop AI agents are software's future
• Agents Write Code. They Don't Do Software Engineering.
• 2026 CircleCI Report: The Agent Validation Gap Is Here
• The Complete Guide to Microservices Testing: From Unit Tests to Production

Frequently asked questions