open to work
Let's talk
← All articles

Who tests the code AI wrote? When a green test stops being proof

October 01, 2026

Many teams have built this loop without noticing: AI writes the code, AI writes the tests, CI turns green and production breaks. Each step looks reasonable on its own. The problem is the sum, because nobody inside the loop is independent from whoever wrote the code.

The 2026 numbers point to this tension. In an Applause survey run in August and released on September 30, more than 92% of respondents used AI in their testing process, versus 59.6% in the previous survey, and 29% reported an increase in the number or severity of defects. In another, by SmartBear, with 1,436 participants in the United States and the United Kingdom, 46% of teams had already shipped AI-generated code to production that later failed, and still 69% of that group had a lot or complete confidence that this kind of code behaves as intended. I treat these numbers as an indication, not a verdict, because they come from vendor surveys with their own samples and criteria. Even so, the combination of more speed, more defects and intact confidence describes a problem that anyone in QA recognizes.

Why a green test stops proving anything

When the test is generated from the code, it inherits the code's assumptions. If the implementation misunderstood the requirement, the test misunderstands it the same way, and the two agree with each other. Consider splitting a payment into installments, in cents:

// generated code
function parcelar(totalCentavos, n) {
  return Array(n).fill(Math.round(totalCentavos / n));
}

// test generated from the code
expect(parcelar(1000, 3)).toEqual([333, 333, 333]);

The test passes, but the installments add up to 999: one cent vanished. The test copied the code's behavior, not the requirement, which says the installments must add up to the total. A test derived from the requirement fails on its first line:

// test derived from the requirement
const p = parcelar(1000, 3);
expect(p.reduce((a, b) => a + b, 0)).toBe(1000);
expect(Math.max(...p) - Math.min(...p)).toBeLessThanOrEqual(1);

The question that separates the two tests is simple: where did the expected value come from? If it came from the code, the test only confirms that the code does what the code does. If it came from the requirement, it can disagree, and that ability to disagree is what gives a test its value. With a property-based testing library such as fast-check, those last two lines become a rule checked against thousands of generated inputs.

There is a second cheap check: ask whether the test can fail. Mutation testing tools, such as Stryker (JavaScript and TypeScript) and PIT (Java), change the code on purpose and check whether any test notices. In the example above, swapping Math.round for Math.floor changes nothing for the generated test, and a suite that does not notice that swap does not protect what it seems to protect. Line coverage says the code ran, while mutation says whether the test would notice a mistake.

What changes for QA

If generating code and tests became cheap, QA work shifts to the more expensive question: can we trust the process that produced this software? That is more than finding bugs at the end of the line. It means measuring confidence with evidence instead of declaring it. A few measures help: the mutation score on critical modules, the flaky-test rate, the defects that escaped to production, and the time the team takes to update automation after a change. In a Leapwork survey in partnership with SD Times, with 302 respondents between December 2025 and January 2026, 56% pointed to test maintenance as what slows their teams down the most, behind only test creation, cited by 71%, which shows that automation alone does not solve it.

In practice, three habits change the outcome considerably. The first is writing the expected value from the requirement before asking AI for the code. The second is reviewing every generated test with the question about where the expected value came from. The third is running mutation on the paths that cannot fail, such as value calculation, authorization and third-party integration. And since no suite covers everything, it is worth adding observability and gradual rollout in production, so that the mistake that slipped through shows up small and is reverted fast.

This work requires reading code and writing code, which is why it works best when QA and development stop being opposite sides. AI's speed is a fact. Trust in what it delivers is still something the team has to build, test by test, with evidence.

Let's talk