Are We Automating the Wrong Part of Testing?
Why do we use AI agents to write automated tests when they could execute checks themselves? Exploring test automation without a separate layer of test code.
Lately, I've been seeing more and more LinkedIn posts about new tools and frameworks that use AI for test automation.
The people building these solutions demonstrate how quickly AI agents can generate automated tests, how good the resulting code is in their view, and how these tools can help make tests more reliable, increase coverage, and reduce development time.
It's genuinely impressive. I enjoy seeing the engineering community experiment so actively with AI, build new tools, and share what they've learned.
But every time I come across another solution like this, I find myself asking the same question: are we making the best use of AI agents?
Automating Automation
AI agents are already a form of automation. They can interact with software, run utilities, analyze results, and decide what to do next.
Yet in testing, we often use them to write code that will then perform automated checks.
That creates a curious setup: we're using automation to build more automation.

And it got me thinking: do we really need that intermediate layer of code?
Perhaps AI agents have given us an alternative to conventional code-based test automation that we haven't explored enough yet.
Do We Need the Middle Layer?
Let's take a simple example.
Suppose we need to check how a web service responds to a particular HTTP request.
When testing manually, we can use curl to send the request, inspect the response, and compare it with what we expect.
If we want to automate that check in the traditional way, we'll probably write a small test in Python, Java, or another programming language. It will send the request, receive the response, and perform the necessary assertions.
Of course, in real projects we usually use HTTP libraries rather than calling curl from our code. But that distinction doesn't matter much for this example.
The point is that code lets us repeat a sequence of actions automatically and check the outcome.
Now imagine that, instead of writing that test, we give an AI agent an instruction: send a request to a particular endpoint, check the response status, and verify that the returned data matches our expectations.
The agent can call the very same curl, inspect the result, and perform the check.
Why write Python code first if the agent can already perform the necessary actions?
This doesn't mean getting rid of programming altogether, of course. The agent itself, its tools, and the infrastructure used to interact with the system under test still rely on code.
But perhaps we no longer need to write a separate implementation for every test scenario.
Instead, we could use instructions that both a person and an AI agent can understand.
And I'm not suggesting that we replace Python tests with a new formal language or another framework that maps written steps to functions. That would risk introducing a new intermediate layer to replace the one we're trying to remove.
I'm thinking of a reasonably clear description of what needs to be checked, which tools are available, and what outcome we expect.
What About Self-Healing?
Traditional automated tests usually follow a predefined sequence of actions. If the result doesn't meet the specified conditions, the test fails.
And quite often, that's exactly what we want.
But sometimes tests break not because of a product defect, but because of a small change in the environment, output format, or the way we interact with the system.
For example, a utility might slightly change the format of its output while still providing all the information we need. A rigid assertion may fail even though the functionality we're trying to verify hasn't changed.
An AI agent could potentially work through that situation while executing the test.
Rather than stopping immediately, it could examine the result, compare it with the purpose of the test, and determine whether the expected behavior was actually violated.
To me, this seems like a more natural approach to self-healing automation.
There is an important limitation, though.
The agent must not decide on its own which requirements it can ignore.
If the exact response content—including spaces, quotation marks, or formatting—is part of the contract being tested, any deviation should be treated as a potential defect.
The way a check is performed can adapt, but the criteria for passing it must not. We also need a record of what the agent changed and why, so that adaptation doesn't conceal a bug.
Otherwise, we risk ending up with tests that adapt beautifully to changes but stop catching real problems.
Why Might This Matter Now?
Traditional code-based test automation is particularly effective for regression testing.
We implement a check once and then run it repeatedly, getting fast, reproducible results. When the functionality is reasonably stable, this approach can be extremely efficient.
But in projects with short release cycles, changes happen frequently, new features arrive quickly, and test automation has to keep up.
Building a good automated test still takes time.
An engineer needs to understand the functionality, define the expected behavior, work out how to interact with the system, and then implement the check in code.
Ideally, an automated test builds on a scenario that has already been verified manually. But even when an engineer goes straight to writing an automated test, they still need to understand how the feature under test is supposed to work.
And that's where I see an interesting opportunity.
If a person already knows which actions to take and what result to expect, why not let an AI agent execute the test directly, using that knowledge?
Perhaps we could shorten the time between defining a test scenario and having an AI agent execute it for the first time.
This doesn't mean traditional automated tests are getting worse or becoming irrelevant. But when things change frequently, the cost of building and maintaining tests may matter more than the speed of an individual run.
What's the Cost of This Approach?
Of course, agent-executed testing has some significant drawbacks.
The first is execution time and cost.
A well-written conventional test usually runs quickly and requires relatively little compute. An AI agent, on the other hand, needs to interpret instructions, call a model, interact with tools, and analyze the results.
So an agent-executed check may well be slower and more expensive than a traditional one.
The second drawback is nondeterminism.
Even modern language models can make mistakes, misinterpret instructions, or draw the wrong conclusions from the data they receive.
I suspect that clear instructions and carefully chosen models could reduce the likelihood of these errors. But for now, that's only a hypothesis that needs to be tested.
And we don't necessarily need the most powerful and expensive model for every check.
Perhaps smaller, relatively inexpensive models could execute limited scenarios reliably enough, given clear instructions and the right tools.
Ultimately, we need to compare more than the cost of a single run. We need to consider the entire test lifecycle: creation, execution, result analysis, adaptation to changes, and maintenance. Maybe more expensive execution would be offset by lower development and maintenance costs. Or maybe traditional tests would turn out to be much more efficient in some situations.
I don't have an answer to that yet.
From Test Scenario to Executable Instruction
There's another potential benefit I hadn't given much thought to before.
We usually keep the description of a test scenario separate from its implementation in code.
The first is intended for people. The second is intended for the automation system. Both representations need to be maintained, and they can drift apart as things change.
But what if the same description could be used by both a person and an agent?
For example, an engineer could perform a new check manually while working alongside an AI agent. Together, they could establish the sequence of actions, clarify the expected results, and save the resulting instruction.
Later, another engineer—or the agent itself—could execute the same scenario.
Of course, these instructions would still need to be maintained, reviewed, and refined. But perhaps we could avoid implementing the same check separately in code.
And this goes beyond how quickly we can write tests. It's an opportunity to bring test design, manual testing, and automated execution closer together.
Worth a Try
I'm not claiming that AI agents can replace traditional test automation. But it feels like we've reached an interesting point where it's worth reconsidering some familiar engineering decisions.
Until recently, writing code was one of the main ways to create flexible, repeatable automated checks. Now we have another possible executor: an AI agent that can understand instructions and interact with tools on its own.
Perhaps, instead of focusing solely on writing automated tests faster, we should explore how often we need to write them at all.
I plan to try this approach on a real project, compare it with traditional automation, and see how reliable, practical, and cost-effective it turns out to be.
For now, though, I definitely think it's worth a shot.