100% Test Automation Might Be Desirable, But Is It Practical?

By Jonny Steiner, Product Marketing Manager

If I got my hands on a time machine, or even time binoculars which are a magic way of seeing through time that I just made up, there are a lot of things I would do with them. For the purposes of this blog post, however, I would like to use my time binoculars to look into the future. Specifically, I would like to see if we ever do reach a point of 100% automation in continuous testing.

100% Automated Testing Might Be Impossible

It is something everyone wants. Automating 100% of their continuous testing is the dream of every QA and testing team member. However, in some ways, this is a real challenge. To start, some code is too complex to write automation tests for it. On the other end, some code is not important enough to need automation tests. The idea is that using a risk-based approach to test automation ensures that the most important parts of a given Sofware are focused upon.

Some experts advise letting go of the 100% automation goal. They say there is no time to automate completely in the constantly changing software development process. The focus, they say, should be on ROI and that which will make business leaders and stakeholders happy.

What If?

Let’s say automating 100% of your continuous testing would be possible. There are signs that some organizations are already trying it. With the growth of Codeless development as well as autonomous testing solutions, the foundation has been laid for a future in which an application is developed and then run through an entirely automated testing process. The only human interaction would be to restart each test run when a new build is ready or when a defect has been mitigated.

The benefit would be the ability to increase release velocity further while also eliminating bugs early in the process without human interaction. However, as we shall see, automated testing creates value but not always in the way that you expect.

The “Value” of Test Automation

The automated testing process goes as follows. Tests are run using a start place, a change, and an expected result. When a test run is complete, that means that the automation has run, but the feature under test is not done yet. Not until the checks are run. Value comes from the process of testing/fixing/retesting. This process results in clear instructions for how to correct errors.

It is important to understand that when automated checks are created, the initial test execution is complete, changes are implemented, and the tests are rerun, the tests are then detecting defects when the expected result is incorrect.

There is an old saying in the world of continuous testing: “One software change turns test automation into change detection.”

Automation As the Start of All Your Problems

Simply put, continuous testing can lead to more challenges. The minute a change is made to the application in development, the automated test suite is outdated. Test maintenance, which is often overlooked, is also essential. A problem that some organizations suffer from is a lack of oversight in the maintenance of existing tests and the deletion of outdated ones, which leads to testing debt.

Test coverage is also important in automated testing. Every scenario needs to be covered, or a test that ‘passed’ might not be as accurate as you think. Sure, you might have passed 100% of your tests, but what about the scenarios you missed? They will not be included.

Then there is the issue of failed tests. These all need to be investigated. What makes things even more challenging is that sometimes tests fail for environmental reasons and not technical ones. In a world where every minute saved is precious, ‘false negative’ tests are.

There are also test failures to consider, as each one needs to be investigated. Some tests are ‘false negatives’ – e.g., a test that times out for some reason due to an environmental issue. This costs time rather than saving it.

The final challenge with test automation is that the tests do not think for themselves. They follow the path that has been set for them and what they were coded for. This is not how humans interact with applications. We get distracted, diverted, and sometimes click on the wrong things. An automated test covers only pre-selected behavior by the user.

How Much Automation Is Enough?

From business leaders and executives to developers and QA, people constantly come up with ideas to test and explore when using or navigating their applications. You might only try it once to see if it works. What you are not going to do, and no one should, is develop a test for all these little scenarios.

We are not quite at the level of feeding requirements to our testing tools and waiting to see the expected results. However, we are at the point where testers want to be able to run their tests with a click and get results. If we talk about automating our testing to 100%, this would mean that a passed execution would mean the software can go straight to production without any further analysis.

The good news is that I just described automated regression testing. This excludes performance and security testing but means that an app can be pushed to production once the test has been performed in isolation and that test passed.

Living With The Truth

So maybe 100% test automation is not possible. It’s a good idea but not the most practical. There will always be code that is too hard to write automated tests for, which might be hard to read, but sometimes a piece of code is not important enough for automation. The risk-based approach I am advocating for here focuses on the most important pieces of software.

Perhaps it is better to think in terms of ROI. Think about the level of coverage that your business really needs. Take the temperature of your executives and leaders to see what their appetite for risk is relative to the software being developed.

Remember that all bugs are not the same, and as a result, not all testing needs to be automated.