The first time I watched an AI-generated test pass for the wrong reasons, it sat in a pull request for a payment-settlement module. The test had the right shape: it initialized a client, called a function, asserted a status code. The name even looked thoughtful—test_settlement_retry_on_failure. But the assertion checked for a 200 OK when the retry spec demanded a 202 Accepted. The mock was rigged to return 200 for every request, and the AI, trained on patterns, had dutifully produced a test that matched the mock instead of the specification. The pipeline went green. Two reviewers approved it. Three weeks later, a retry storm in production silently dropped transactions because nobody had actually verified that the retry logic worked. The test passed. That was the problem.
We have been here before. When IDEs rolled out code completion, the first wave of criticism warned that developers would accept suggestions without understanding them. When organizations outsourced QA to separate vendors, the complaint was that ownership of quality diffused across a contract boundary. In both cases, the technology or the organizational structure did not create the failure—it revealed an existing weakness in how we assign responsibility for correctness. AI-assisted generation lands in the same category, but the speed and scale at which it produces plausible, syntactically valid output makes the gap between something that compiles and something that is correct wider and harder to spot.
The core risk is not that AI writes bad code or bad tests. It is that AI output arrives with an implicit warranty of completeness that nobody signed. When a junior engineer submits a patch, a senior reviewer knows to interrogate the logic. When an AI generates the same patch—or a test, or a piece of documentation—the social contract shifts. “The AI wrote it” becomes a whispered excuse for skipping rigorous review. The green pipeline, the passing test, the coherent-sounding paragraph all conspire to lower the reviewer’s guard. And because the output is often almost right, the errors that survive are the most dangerous kind: subtle, systemic, invisible to the metrics we use to reassure ourselves that quality is under control.
The Empty Green Checkmark
Consider a test suite that verifies a user-permission model. A human engineer writing a test for an admin-role escalation would think about boundary conditions: what happens when the role changes mid-session, when the token cache is stale, when the escalation is attempted by a user who was an admin but has been demoted. An AI, given the function signature and a prompt like “write integration tests for role escalation,” will generate tests that exercise the happy path and a few obvious error cases—null input, missing headers—because those patterns dominate its training data. The tests will pass. Coverage percentages will rise. And the first time a demoted admin’s cached token is accepted, the system will grant access it should not, and nobody will have written a test for that condition because the AI did not think of it, nobody asked it to, and nobody reviewed the output with the skepticism the scenario demanded.
This is not a failure of AI. It is a failure of process that AI makes easy to ignore. For a skilled engineer, writing a test is an act of reasoning about the system. The test is a byproduct of that reasoning. When the reasoning is outsourced, the byproduct may still look correct, but the reasoning—the part that prevents production incidents—has not happened. The test becomes a hollow artifact. It passes and tells you nothing.
We have a vocabulary problem here. When a human writes a test that passes but is logically wrong, we call it a bad test. When an AI does it, we are tempted to call it an AI hallucination or a limitation of the model. That language absolves the human in the loop. A more precise description: the organization has produced untested output decorated with the ceremony of testing. The pipeline approved it. The coverage metric counted it. The system is no safer than before.
Diffusing Accountability to the Point of No One
Every quality system rests on a chain of responsibility. In a traditional team, the engineer who writes the code is responsible for its correctness. The reviewer is responsible for verifying that correctness. The QA engineer, if one exists, is responsible for designing scenarios the engineer and reviewer missed. The chain has flaws—review fatigue, time pressure, knowledge gaps—but the nodes are people with names and reputations. When a bug reaches production, the postmortem can trace the chain and ask where the reasoning broke down.
AI-generated output disrupts this chain in a specific way: it introduces a node that has no reputation, no career at stake, and no ability to explain its reasoning in a way that holds up under adversarial questioning. When a test written by an AI passes incorrectly, who is responsible? The engineer who prompted the AI? The reviewer who approved the PR? The organization that set the policy allowing AI-generated tests without additional review? In practice, the answer is often no one, because the organization has not defined a new contract of responsibility for AI-generated content. The output simply enters the codebase under the same governance as human-written code, but without the same scrutiny, because the green pipeline and the “AI-assisted” label create a collective assumption that someone else—the model, the tool, the vendor—has already done the checking.
This dynamic is not theoretical. The Authors Guild, in its AI Best Practices for Authors, argues explicitly that transparency and accountability are non-negotiable when AI is involved in creative and professional work. The Guild’s guidance focuses on writing, but the principle transfers directly to code: if you cannot explain who verified the logic behind AI-generated output, you have not verified it. The output is syntactically correct, but its relationship to the system’s actual requirements is unexamined.
The parallel to outsourced QA in the early 2000s is instructive. Companies that treated outsourced testers as a black box—send requirements in, receive test results out—learned that the quality of the tests depended entirely on the clarity of the requirements and the feedback loops between the testers and the engineers. Organizations that succeeded built tight integration, shared context, and clear ownership: the engineer who wrote the code remained responsible for the correctness of the tests, even if someone else executed them. AI-assisted generation demands the same clarity, but the feedback loop is even more fragile because the “someone else” is not a person with domain knowledge. It is a model with statistical knowledge of code patterns. The responsibility for bridging the gap between pattern and intent falls entirely on the human, and if the human does not accept that responsibility, the gap remains.
A Brief History of Tools That Made Quality Someone Else’s Problem
Code generation is not new. ORMs, scaffolding tools, and template engines have been producing syntactically valid output for decades. The difference with AI-generated code and tests is the scope of ambiguity. An ORM generates database access code from a schema; the schema is an explicit, machine-readable specification of intent. A scaffolding tool generates boilerplate from a few parameters; the parameters define the boundaries. An AI model generates code from a natural-language prompt that is inherently ambiguous, and then generates tests from the same ambiguous prompt, creating a closed loop of self-reinforcing assumptions. The test verifies the code, but both the code and the test were derived from the same underspecified intent, and neither was subjected to the kind of adversarial reasoning that a human engineer brings to the relationship between specification and implementation.
This pattern of mistaking surface plausibility for internal correctness is not unique to code. The same accountability gap appears in prose—a tool like the how Unsloppy AI Book Generator fits the writing workflow can produce fluent drafts, but unless a human editor verifies its internal logic against the intended narrative, the output is just plausible text—exactly the problem we see with generated tests. A test that passes sounds good. It fits the pattern. But it may be false to the system’s internal logic—the logic that lives in the minds of the engineers who designed it, the constraints debated in architecture meetings, the edge cases that emerged from previous incidents. AI models do not attend those meetings. They do not read the postmortems. They generate from patterns, and patterns do not encode the specific, contingent, hard-won knowledge that makes a system reliable.
What a Contract of Responsibility Looks Like
If you are in an organization where AI-assisted code or test generation is happening—and you almost certainly are—the single most important question to ask in your next team meeting is: Who is responsible for the correctness of AI-generated output, and how do they demonstrate that responsibility? The question sounds abstract, but it translates into concrete process changes.
First, every piece of AI-generated code or test must have a human owner whose name is attached to it in the version-control history. Not the name of the person who ran the prompt, but the name of the person who reviewed and asserted that the output is correct against the system’s requirements. This is not a new idea; it is the same principle that applies to code review. The difference is that the reviewer must approach AI-generated code with a higher degree of skepticism than human-written code, because the human writer’s reasoning process is not available for interrogation. The reviewer cannot ask, “Why did you mock this dependency this way?” and get an answer that reveals a misunderstanding. The reviewer must instead reconstruct the reasoning from the output alone, which is harder and requires more time.
Second, AI-generated tests must be verified against a specification that exists independently of the prompt. If the prompt is “write tests for the payment retry logic,” and no written specification of the retry logic exists, then the tests are verifying an assumption, not a requirement. The minimum bar is that the reviewer compares the AI-generated test assertions to the acceptance criteria in the ticket, the API contract, or the architectural decision record. If those artifacts do not exist, the review is performative.
Third, teams need to audit AI-generated test suites for logical coverage, not just line coverage. Logical coverage means: do the tests exercise the conditions that would cause the system to fail based on what we know about its failure modes? This is not a metric that can be automated easily. It requires a human—preferably the engineer most familiar with the system’s incident history—to read the test names and assertions and ask, “What is missing?” When the answer is “nothing,” and the tests were AI-generated, the team should be suspicious. The correct answer is almost never “nothing.”
These practices are not anti-AI. They are pro-quality. They acknowledge that AI changes the economics of producing code and tests, but not the economics of verifying correctness. Verification remains a human activity, and it becomes more valuable, not less, as the volume of generated output increases. The temptation to let the green pipeline substitute for verification is the modern version of the old mistake of letting a passing test suite substitute for understanding the system. The tooling is faster. The error is the same.
The Documentation Corollary
The same diffusion of accountability applies to AI-generated documentation, release notes, and even postmortem summaries. A paragraph that reads fluently and uses all the right technical terms can create the illusion of thoroughness. I have seen teams accept AI-generated API documentation without verifying that the documented behavior matches the implemented behavior, because the prose was convincing and the review deadline was tight. The documentation becomes a liability: it promises behavior the system does not deliver, and the first person to discover the gap is a user trying to integrate.
The principle is unchanged. AI-generated prose must be verified by a human who understands the system and is willing to stake their reputation on the accuracy of the output. If that human does not exist, or does not have the time, the output should not be published under the organization’s name. The alternative—publishing unverified AI-generated documentation—is a choice, and it is a choice to transfer the cost of discovery to the user.
The Authors Guild’s best practices, referenced earlier, emphasize that the author must retain creative control and disclose AI involvement. The technical parallel is that the engineer must retain verification control and disclose AI involvement in a way that signals to reviewers that extra scrutiny is required. The tool does not remove the obligation; it raises the stakes.
Resisting the Organizational Gravity
The organizational pressure to treat AI-generated output as “done” is immense. AI reduces cycle time. It lets teams produce pull requests faster. Coverage metrics trend upward. In sprint reviews and quarterly planning, these are the numbers that get attention. Pushing back—insisting that AI-generated tests need more review than human-written ones, not less—will be framed as obstruction. The vocabulary for pushing back effectively is precise and non-negotiable: We are not measuring the right thing. A passing test that does not verify the correct behavior is a liability. The coverage number is not quality. The green pipeline is not trust.
The argument that lands with engineering leaders is not about philosophy. It is about incident cost. Every AI-generated test that passes incorrectly is a production incident waiting to happen. The incident will not be attributed to the AI. It will be attributed to the team, the system, the on-call engineer who gets paged at 3 a.m. The postmortem will ask why the test suite did not catch the failure, and the answer will be uncomfortable: because we allowed a test to exist that looked correct but was not, we did not assign anyone to verify it, and we shipped anyway. That is not an AI problem. It is a quality-ownership problem with the same root cause as every quality-ownership problem in the history of software: someone assumed someone else was checking, and nobody was.
The fix is not to stop using AI. The fix is to define, in writing, in your team’s working agreements, who is responsible for the logical correctness of AI-generated output and what evidence of verification is required before that output can merge. The fix is to treat AI-generated code and tests as untrusted input until a human has signed off on them, the same way you would treat a pull request from a new contractor whose judgment you have not yet calibrated. The fix is to remember that the green checkmark is not a fact; it is a claim, and the claim is only as good as the reasoning behind it.
If you do not verify the logic, your users will. And unlike your CI pipeline, they will not give you a green checkmark when they find the gap.
For additional context, see Purdue OWL.
