Augmented QA. What AI changes in software testing.

The suite nobody has time to maintain.

Across the engineering organisations we work with in Paris, Dubai, Singapore and Bali, the pattern is consistent. There is an automated suite built over four years, it takes fifty minutes to run, a third of it is failing or quarantined, everybody knows which failures to ignore, and the release decision is taken by two people who read the output and use their own judgement. Nobody has time to fix it because every sprint brings new features that need covering.

Adding a model that generates two hundred more test cases into that situation makes it worse. The interesting question is narrower and more useful: which parts of the testing craft consume time without requiring judgement, and can a model do those well enough to give the team its judgement time back.

What AI-augmented QA means.

AI-augmented QA is the use of language and code models inside an existing testing process to draft, repair, classify and prepare, with a human reviewing anything that becomes a verdict. It sits alongside a separate discipline, testing AI systems themselves, which follows different rules because the system under test is probabilistic. The two should not be confused, and the principle that keeps them apart is the one we set out in AI does the work, humans do the validation. This article is about using AI to test ordinary deterministic software better.

The definition contains its own limit. A model is good at producing plausible text and code fast. It has no access to the intent behind a feature, no knowledge of which customer segment tolerates which failure, and no stake in the release. Everything that depends on those three things stays with a person.

Where AI earns its place.

• Test design from requirements. Give a model a user story with acceptance criteria and ask it to enumerate scenarios, including the boundary and negative cases a tired analyst skips at four o'clock on a Friday. The output is a draft, and the value is in the coverage of possibilities rather than in the wording. The quality of the output tracks the quality of the acceptance criteria almost perfectly.
• Automation maintenance. This is the strongest use and the least discussed. A front-end refactor breaks forty selectors, and repairing them by hand is a week of work that teaches nobody anything. A model reading the diff and the failing traces proposes the repairs, a tester reviews and merges. Teams that apply AI here first see the effect within a sprint, because they are attacking the backlog that was blocking everything else.
• Defect triage. Is this a duplicate of something raised eleven months ago in different words. Which component does it belong to. What severity did similar defects receive. Which recent commit touches the code path in the stack trace. Models are genuinely good at this, and triage is where large organisations lose days per week to routing rather than to fixing.
• Test data. Producing realistic, varied, non-production data that respects referential integrity and privacy constraints is slow manual work with a clear specification, which makes it a good fit. Generated data also covers value distributions that a hand-built fixture never does, and that alone surfaces defects.
• Exploratory support. A model that reads the change log and the past six months of defects can propose where to look this week. The exploration stays human. The reading of history does not have to be.

Where it does not, and why the oracle problem matters.

An oracle is the thing that tells you what the correct output should be. A model asked to write tests for existing code reads that code and infers the expected behaviour from it, so it writes tests that pass. The bug that has been sitting in a rounding rule for two years gets encoded as the specification, and now it has a green test defending it.

A generated test encodes the behaviour the code already has, including the bug.

That is the boundary. Generation from requirements is useful, generation from implementation is a trap, and a model cannot tell you which of two plausible behaviours the business actually wanted. Three other areas stay with people. Risk judgement, because deciding what to test in a fixed amount of time before a release requires knowing which failures the organisation can absorb, and that is a business conversation. Acceptance and sign-off, because a verdict produced by a model that nobody reviewed is not a verdict, and no regulator, auditor or customer will treat it as one. And non-functional testing, where performance, resilience and security work depend on infrastructure knowledge, realistic load modelling and threat reasoning. Models help draft scenarios and read results there. They do not replace the engineering.

The trap: more tests than you can maintain.

Every team we have seen adopt generation makes the same move in the first quarter. Coverage becomes the target, the suite doubles, the run time doubles with it, flakiness rises because generated tests are more brittle than hand-written ones, and within two sprints people are ignoring failures again. The team has industrialised the exact problem it started with. Four rules keep this under control.

• Cap the suite. Fix a maximum run time for the pipeline and treat it as a hard constraint. Adding a test means justifying it or removing another.
• Review every generated test as code. Same pull request, same reviewer, same standard. A generated test that nobody read is technical debt with a green tick on it.
• Delete aggressively. Tests that have never failed in eighteen months, or that fail for reasons unrelated to the behaviour they claim to check, cost more than they return.
• Fix the quarantine before you generate. If a third of the suite is already unread, generation adds noise to noise. Repair first, then extend.

What the tester keeps.

The role gets more interesting and harder, which is not the same as easier. Time moves out of writing and repairing and into three things: designing what should be tested and why, reviewing output produced faster than it can be read carelessly, and holding the line on the risk conversation with product and engineering.

Two capability gaps show up immediately. The first is reviewing at volume, which is a learnable skill and rarely taught: knowing what to look for in a generated case, spotting a test that asserts nothing, recognising an assertion that mirrors the implementation. The second is evaluation literacy, which is less about clever prompt wording than about writing a specification precise enough to be checkable, and it is the same discipline we describe in the evaluation suite that ships. The testers who struggle are the ones whose value came from volume of manual execution. The ones who thrive are the ones who were already asking why a feature exists.

Measuring it.

Test counts and coverage percentages go up the moment you switch generation on, and they tell you nothing about whether testing got better. Five measures do.

• Cycle time to a trustworthy verdict. From code complete to a result the team acts on without a side conversation. This is the number the whole exercise is about.
• Escape rate. Defects found in production per release, segmented by severity. If this rises while test counts rise, the suite is testing the wrong things.
• Quarantine share. The proportion of the suite currently disabled, failing or ignored. Rising quarantine is the clearest early signal that generation is outrunning maintenance.
• Maintenance effort. Hours spent repairing tests per sprint, tracked before and after. This is where the first real gain appears.
• Review load. Time spent reviewing generated artefacts. A gain that simply moves work from writing to reviewing is not a gain, and this measure is the one teams forget to take.

Baseline all five before the first tool arrives. Retrofitting a baseline afterwards is guesswork, and it makes the eventual business case unarguable in the wrong direction.

A first quarter that works.

Start with maintenance, not generation. Point the model at the failing and quarantined tests, repair the suite, and get the pipeline back to a state where a red build means something. That alone changes the team's relationship with its own tooling, and it takes weeks rather than quarters.

Then add triage, because it is low risk and the time saving is immediate. Then design generation from acceptance criteria, on one squad, with mandatory review and a capped suite. Extend to a second squad only once the first has held its run time and its quarantine share for a full sprint. Run the whole thing inside your source control and your pipeline, with the same review gates as production code. Testing tooling that lives outside the engineering workflow decays quietly, and the decay is invisible until a release goes wrong.

Where we come in.

We run QA assessments and augmentation programmes: a read of your current suite and its quarantine, the maintenance repair pass, triage and generation set up inside your pipeline with review gates, the measurement baseline above, and the cohort training that makes your testers fast at reviewing rather than dependent on a tool. Our tech factory teams work inside your architecture and hand the method over.

The Consulting team in our Paris, Dubai, Singapore and Bali offices is reachable from the brief form below. We answer within one working day, with the partner who will sit on the file.