Test automation is an extremely large and complex topic, wouldn’t you agree?
There are so many important pieces of information that can improve your test automation.
Hence, I decided to document all the automation patterns and anti-patterns to help you. I will update and expand this post regularly.
Tests Should Be Atomic
Your automated test should form a single irreducible unit. This means that your tests should be extremely focused and test only a single thing. A single automated test should not do something like end-to-end automation.
A good rule of thumb that I use on my teams is:
Automated acceptance test should not run longer than 1 minute on your local resources
If your test runs longer than 1 minute, then you are possibly doing something wrong. The advantages of writing atomic tests are numerous…
First, writing small tests allows you to fail fast and fail early. This implies that you will get extremely fast and focused feedback. If you want to check the state of a feature, it will take you no longer than 1 minute.
Second, writing small tests reduces flakiness because it decreases the amount of possible breaking points in that test. Flakiness is less of a problem with unit or integration tests. But it is a large problem with acceptance UI automation.
Here’s an example:
- Open UltimateQA.com home page
- Assert that the page opened
- Assert that each section on the page exists
- Open Blog page
- Search for an article
- Assert that the article exists
For UI automation, every single step is a chance for something to go wrong. A locator may have changed, the interaction mechanism may have changed, your synchronization strategy may be broken, and so on. Therefore, the more steps that you add, the more likely your test is to break and convey false positives.
Third benefit of writing small tests is that if it fails, it will not block other functionality from being tested. For example, the test I mentioned above. If it fails on Step 3, then you might never get to check if the Blog page works or that the Search functionality works. Assuming that you don’t have other tests to check this functionality. As a result of a large test, you will reduce your test coverage.
Finally, another great benefit of writing small tests is that they will run quicker when parallelized…
If all of your tests run in one minute max and you have one hundred tests, then throwing one hundred VMs at that suite means that you can have feedback in one minute. However, if you have even a single test that runs in ten minutes, then applying the same number of VMs will return you results in ten minutes. That’s a 10X decrease in speed to feedback. Fast feedback loops are critical in test automation.
The total automated suite feedback time will be as slow as your slowest test
By the way, I have seen automated tests that take 30 – 90 minutes to execute. These tests are extremely annoying to run because they take so long. Even worse, I’ve never seen such a test produce valuable feedback in my entire career. Only false positives.
How to break up giant end-to-end UI tests?
One of the most common problems I see with clients is that they have extremely long running UI tests that break the Atomic Tests Pattern.
So the question is, how can you break up your giant UI test? Here is a simple scenario:
- Open Amazon.com
- Search for an item
- Add item to cart
The first problem is that many automation engineers assume that you must do an entire end-to-end flow for this automated test.
Basically, you must complete step 1 before step 2 and so on…
The correct approach is to be able to inject data to populate the state of the application prior to any UI interactions.
You can inject data via web requests or via updating your database directly. The former method is preferred.
If you can inject data between the seams of the application, then you can isolate each step and test it on it’s own
- Use an API to send a web request that will generate a user
- Use an API that will generate an item in your Amazon cart
- Now you can pull up the UI to the cart page and checkout
- Clean up all test data after
This is the best practices approach. You tested the checkout process without using the UI for steps one through three.
Using an API is extremely fast… A web request can execute in like 100 ms.
This means that steps 1,2,4 can take less than one second to execute. The only other step you will need to do is to finish the checkout process.
It gets better:
Using an API is much more robust than using a UI for test steps. As a result, you will drastically decrease test flakiness in your UI automation.
What if you don’t have the capability to inject data for testing?
I know that the world isn’t perfect. I’ve worked with clients that are not able to follow the prescribed process above. So they ask what they can do.
You have two options:
First, you should work with the developers to make your application more testable. Not being able to easily test your application is a sign of poor development.
Work with your team to provide you seams where you can inject and manipulate test data and application state. I have done it myself and I have seen it done with my clients.
It’s doable and most developers are happy to help.
Here’s the kicker, and you won’t want to hear it:
If you can’t work with the developers, then just don’t automate such a complicated scenario. I know that your manager asked you to automate it…
However, we are the automation engineers. We are the professionals. We must decide what to automate and not to automate based on our understanding of application requirements.
If your manager is non-technical, they should not be telling you how to do your job. You don’t see managers telling developers how to code. Why is it okay for managers to tell an automation engineer what to automate?
The answer is it’s not okay!
You must be the expert and decide on the correct approach to do your job.
Using complicated data store such as Excel
One of the most common questions from my students and clients is how to use Excel for test data management in test automation.
Don’t use Excel to manage your automation test data
I understand the rationale behind using Excel for your test data. I’ve been doing test automation for a really long time and I know about Keyword Driven Frameworks and trying to let manual testers create automated tests in Excel. My friends…
It just doesn’t work… I wasn’t able to make it work myself and I’ve never seen anyone else make it work successfully.
Let’s talk about the reasons why using Excel is an anti-pattern:
- The logic to read and manage Excel adds extra overhead to your test automation that isn’t necessary. You will need to write 100s of lines of code just to manage an Excel object and read data based on column headers and row locations. It’s not easy and is prone to error. I’ve done it many years ago. All of this will eat into your automation time and provide no value to the business that employs you.
- You will be required to manage an external solution component for your test automation. This means that you can never simply pull the code and have everything work. You will need to have a license for Excel. You will need to download and install it. And you will need to do this for all of your automation environments. Usually local, Dev, Test, and Prod. This means that you need to manage this Excel instance in all of these environments. This is simply another waste of your time.
So what are the solutions:
- The best solution is if you have an API that you can use to read test data. This is a robust and lightweight solution
- If you don’t have an API, you can talk directly to the Database. This takes much less code and it’s much easier to manage than working with and external Excel object.
- If you must use some data source, use a .csv or .json file. CSZ and JSON files are extremely lightweight, easy to read, and can be directly inserted into your automation code. This means that you will be able to simply download the code and have everything work without needing to perform other installations.
Trying to use UI automation to replace manual testing
Automated testing CANNOT replace manual testing
I have not read or seen any automation luminary who claims that automation can replace manual testing. Not right now at least… Our tools have a long way to go.
However, I know and have worked with managers and teams whose goal with test automation is exactly what cannot be done.
And so these individuals pursue a goal that is impossible… Obviously leading to failure.
Use of automation can drastically enhance the testing process. If used correctly, automation can reduce, not replace the manual testing resources required.
Why can’t automation replace manual testing?
First, it’s impossible to get 100% automated test coverage. It’s actually impossible to get 100% test coverage in general…
That’s why we still have bugs on all the apps in the world, right?
Anyways, if you can’t automate 100% of the application, that means that you will need some sort of manual testing to validate the uncovered portions.
Second, UI automation is too flaky, too hard to write, and too hard to maintain to get over 25% test coverage…
This is based on experience and is an approximation… I don’t have any hard data on this.
However, you actually don’t want higher coverage than 25%.
Here’s the kicker:
Using a combination of unit, integration, and UI automation, you might get close to 90% test coverage…
But that’s really hard. And this is just a side note.
Finally, there are some manual activities that cannot be automated technologically…
That’s for now and for at least a few years in my opinion.
Some examples include UX Testing and Exploratory Testing.
So again, if you are trying to use automation to replace manual testing, it will be a futile effort.
What is the solution?
Use the automation testing pyramid and don’t try to replace manual testing with UI automation.
Use UI automation and any automation to enhance the testing process.
A combination of manual testing and automated testing will create the best end user experience.