TA: Is your test automation suite Good, fast, cheap (pick two)?

I think every test automation engineer will come at a point where you need to look at improving/optimizing your test automation script so it will take less time to run. But doing so is always tricky. Your tests should run as reliable as possible, at some point speed also becomes a factor in the equation. So what can you do, what works and which trade-offs should you avoid?

Core reason for your TA suite

The most important thing to keep in mind is that you should never give in on the core reason why you implemented test automation. If you only have a set of tests that run locally and the results are for you to interpret anyway, speed might be more important and a test is allowed to fail once in a while. But when your test suite is connected to a CI cycle and the results are automatically interpreted, it is very important to have super reliable tests instead. Obviously you want to have fast and reliable tests all the time. But time can only be spend once and is limited in most projects. But keep in mind that spending hours of time to figure out how to reduce a test run from 2 hours to 1 hour and 58 minutes might also be a waste of time (and time is money).

Reduce the usage of fixed timeouts

When improving your test scripts, just start with the obvious. Try to prevent using fixed `sleep` timeouts in your script. It's better to wait until an element appears/disappears compared to adding a fixed sleep in your tests. This also makes it more reliable and faster if done correctly. And if you really need fixed timeouts, try to reduce the time you wait, but keep in mind that your local machine might run faster than the machine which will run the test automation script.

API

Avoid front-end tests that can be done on another level. But if you have to test on UI level, try to use API calls for certain precondition- and validation- steps. This topic I briefly covered in the previous post. For the precondition of a test it does not matter how you set/prepare for it, as long as your setup is in a state where it can execute the interaction steps, it's all fine. This should make your test run more reliable and also faster.

JavaScript click versus Selenium/Watir click

It can be that a menu of a previous test has been opened which covers the button you actually need to click for the current precondition of the test. In that case you can always try a JavaScript click instead to just trigger the click on the button even though it's covered. Personally I try to limit clicks with JavaScript to precondition steps only. The reason is that when using a JavaScript click, the test will not tell you when a button is covered by something else (which might be a real issue/bug). If the browser you want to add to your suite, has issues with clicking the element but has no issues with triggering the click with JavaScript, you should ask yourself if it's more useful to change your scripts or to test this browser manually. 

Use global variables

Especially on a mobile SPA website with no direct URL's to use, navigating can be time consuming. Think about the gmail app on your phone, if you selected a label and opened an e-mail, you can only change the label if you first exit the email again. Our SPA website has similar navigation, but with a lot of layers. So you can navigate to a page, then open a list, open a tab, open a breadcrumb, activate a sub-tab. What I noticed is that once I went a layer deeper, the elements that tell me on what page I am, are no longer available. Meaning to be sure I meet the precondition requirements I need to get all the way back to the top layer and start over again for each scenario and that is a waste of time. So I thought... What if I just store a string of each layer in a global variable as soon as I navigate a layer deeper. So first I use the UI and valid checks to go a level deeper and make sure I'm there, and when that check has been done I store the string in a global variable which I can re-use in the next scenario precondition. Here is an example:

def ensure_list_is_open(listname) unless $active_list == listname # Assume the correct list is open based on global var unless Self.is_list_title?(listname) Self.get_list(listname).fire_event("click") unless Self.is_list_title?(listname) error "Listname (#{listname}) is still not open" end end $active_list = listname # Change the global variable end end

In case a test failed, I reset all global variables in `hooks.rb`since I cannot guarantee that these are still valid:

After do |scenario| if scenario.failed? $active_list = nil browser.refresh ...

This solution saved me a lot of time in certain precondition step, but will also ensure me I'm not getting any false negatives after a previous test failed for whatever reason. Just keep in mind that this is actually making assumptions instead of performing a real check. And if you forget to update the global variable in one interaction, it can fail a test, so use with caution. Let me end this section with the following remark. It will always be better to ask the devs to add an element that you can check over assuming something. But if you have no other alternative and really need to speed up the process it might be worth giving it a try. On my personal laptop this only saved me some seconds on the whole suite. On the Azure machines which are running the test via selenium-grid, it saved me 5-10 minutes a one hour run. Obviously with a bit more than just this one, but it was still a nice speed gain.

Parallel sessions

Another great idea (if budget allows it) is to run your tests in parallel. Running your tests in parallel can easily reduce your test run time to a minimum but it comes at a cost. If you have the resources in-house it makes a lot of sense to use this before even thinking about some of the above mentioned solutions. The biggest downside of running your tests in parallel is the additional cost. Cost for writing your test in such a way they can run in parallel and cost for the machines and/or services you need to use. If you don't have the resources, knowledge or time to setup a selenium-grid, SauceLabs and BrowserStack are good services you could use.

Keep the following things in mind:

• All your tests are able to run independently
• All your tests can run at the same time as any other test (if not, think about a solution)
• i.e. We use a different user per browser that we run in parallel
• All your test data is unique per scenario (if not, think about a solution)
• i.e.  We use different assets per user

Only when your tests and test data is setup in such a way it can handle testing in parallel you can run it in parallel without running into the issue of having false negatives.

If your tests are depending on another, one failing test can fail many (this is already true in serie, but even more in parallel). If you run multiple browsers at the same time, you cannot use one user if, for example, you want to test a language setting. Because the moment you switch the language to Dutch for this user, all tests on the other browser will now fail since it was still expecting English results. The same counts for test data. Let say I do have multiple users, but on both browsers I'm not removing a specific asset from a specific activity, it's gone in browser A and that is fine, because I expected this. But browser B was still running a test where it was expecting this asset to be present. So think carefully how you set this up. It also depends what kind of content/data you need to deal with. Is it shared data, or is it user specific data.