Point a coding agent at a repository and ask for test coverage and you will get a lot of it, quickly, in the right framework, with sensible names and reasonable structure. That is real and it is useful.
It is also the part of test automation that was never the expensive part, and treating it as the win is how teams end up with a much larger suite and no more confidence than they started with.
What AI agents for automation testing actually do
A test-generating agent reads your source, infers what the code does, and writes assertions that describe it. Some also run the suite, read failures and iterate until things pass.
Read that sentence again, because the whole limitation is inside it. The agent derives its expectations from the implementation. It is describing what the code does, not what the code should do.
For code that is correct, that distinction does not matter and the tests are useful regression protection. For code that is wrong, the agent will faithfully assert the bug, the test will pass forever, and you now have a green tick sitting on top of a defect.
The oracle problem is why this is not a model capability gap
Software testing has a name for the hard part. The oracle problem: to decide whether a program behaved correctly, you need an independent source of truth about what correct means.
The reason this matters for agents is that the information is genuinely not in the input. Intended behaviour lives in a requirement document, a regulation, a support conversation, a decision somebody made in a meeting, or a user's unstated expectation that the total should never go negative. None of that is in the repository.
So this is not a limitation that a stronger model removes. You can improve reasoning as much as you like and it will not conjure a fact that was never supplied. The fix is to give the agent the oracle, which means the specification has to exist and be precise, and that is human work that most teams have not done.
Where the cost of a test suite actually accumulates
Ask a team that has run a large suite for a few years where the effort went. Very few say writing the tests.
They say maintenance, flakiness, and deciding what a failure means. A suite of four thousand tests that reports ninety failures every morning is not a safety net, it is a queue. Somebody has to open each one and answer a question with no automated answer: real defect, intentional change that nobody updated the test for, or the suite being flaky again.
Get that call wrong twice in a fortnight and the team learns to ignore red, which leaves you worse off than having no suite at all, because now you are paying for one and still shipping blind.
Generation makes the numerator bigger and does nothing to the denominator. More tests, same triage capacity.
Where test automation agents genuinely earn their keep
Being clear about the limit makes the real wins easier to see. Agents are strong wherever the oracle already exists and the remaining work is mechanical:
- Generating cases from acceptance criteria that are already written down. If somebody has specified the behaviour, turning that into thorough cases is fast, complete and dull, which is the ideal thing to delegate.
- Porting a suite between frameworks or languages. The old suite is the oracle. The task is translation, which these tools do very well.
- Triage assistance. Clustering ninety failures into four probable causes saves real time, as long as a person still decides what each cluster means.
- Test data and fixtures. High volume, low judgement, and a common reason suites are thinner than the team would like.
- Filling gaps around code that is already trusted and stable. Regression protection for behaviour you have deliberately decided is correct.
Why code coverage stopped being a useful metric
Coverage was always a weak proxy. It measures whether a line executed during a test run, not whether anybody thought about what that line should do.
It survived as a metric for a practical reason: writing tests was expensive enough that high coverage implied somebody had spent real effort thinking about the code. The number was standing in for the effort.
Remove the expense and the proxy collapses. Ninety percent coverage produced by an agent in an afternoon carries almost none of the information that ninety percent produced by a team over two quarters used to carry.
If you adopt agents for testing, change what you measure at the same time. Escaped defects, mean time to diagnose a red build, and the proportion of failures correctly triaged on first inspection all still mean something. Coverage does not.
The skill this makes more valuable, not less
The scarce ability here is not writing test code, which is precisely what got automated. It is knowing what the system is supposed to do, and being able to state that precisely enough to argue with a machine about it.
That is specification skill plus domain knowledge, and it is difficult to hire for because it does not show up in a portfolio. It shows up when somebody looks at a fully green suite and says the uncomfortable thing: this all passes and I still do not believe it.
It is also difficult to assess with a CV or a framework question, which is the same problem we ran into building assessments for engineering skills generally. What worked was giving people a real artefact with real problems in it and watching where they looked first.
