PowerShell - Start your tests in parallel

Intro

Not long after I started to build the first set of tests for our single page web application, I noticed the time it took to run the tests started to increase quiet fast. I started to read about parallel testing, selenium-grid, which ruby gem I could use and that was all quiet interesting. Luckely I already wrote my tests to run independantly. For parallel testing you need to keep a couple more things in mind than just write independant tests and starting them, but I'll make a part 2 where I'll cover it more in depth. In this post I'll cover the basics for using PowerShell to trigger tests to run in parallel.

Setup

As mentioned in previous posts we make use of Visual Studio Online (VSO)as a build server and Azure VM's for our Selenium-grid setup. There are multiple ways to go at this, but we use the grid in the following setup. One hub, multiple nodes. The hub machine also has ruby installed and all needed gems. We use the lapis_lazuli gem for our test suite which uses cucumber. More info about that can be found on the testautomation.info website. Via the build blocks in VSO we push the code to the hub (zipped), then it's being extracted, some RDP sessions are being started (see previous post), and last but not least we start a PowerShell script to execute the cucumber command. At the end of the test execution we collect the test results so they can be published on VSO after everything is done. In this post we focus on the test execution and the lessons learned.

Test execution simple example

In the build server we have a block in which we execute the script that will start the cucumber tests which looks like this:
In the PowerShell script we can now start tests. for example like this:
Param(  [string]$testsLocationArgument ) Start-Process cucumber -WorkingDirectory $testsLocationArgument ` -LoadUserProfile -WindowStyle Normal -Wait;
This starts a new process with the cucumber command in the folder specified in the build block and it will wait for the process to be done before moving on to the next block.

Test execution parallel example

The first thing I did was to figure out how to increase the amount of browsers I could test on in parallel. To do so, you need to run the commands like this:
$par1 = $( # FF   Start-Process cucumber `     -ArgumentList "BROWSER=firefox -t @desktop" `     -WorkingDirectory $testsLocationArgument `     -LoadUserProfile -WindowStyle Normal -PassThru; # Chrome Start-Process cucumber `     -ArgumentList "BROWSER=chrome -t @desktop" `     -WorkingDirectory $testsLocationArgument `     -LoadUserProfile -WindowStyle Normal -PassThru; ) $par1 | Wait-Process
What you basically do here, is bundle two processes in a main-process. Each sub-process is started right after another (hence the -PassThru), but you wait for the main process to be finished, which is done when both sub-processes are finished. This way you can add as many variations/sessions as your setup allows, but still wait for everything to be finished before the build block is considered "done".

Lesson learned

These tests are running for quiet some time. The downside is that if you cancel the build in VSO, this process keeps on running. This means, it will keep on occupying the selenium-grid nodes and that is nto what you want when you cancel a build. You can solve this issue in a couple of ways.

Shutdown VM's 

The easiest is to just shutdown the VM's after each build/test run with a block in your build definition. But since starting up the VM and making sure there is a RDP connection takes time it might not be ideal in case you want to do multiple runs in a row.

PowerShell solutions

There are multiple ways you can go with this and it depends a bit on your setup. But think about a block in your build definition that will end the running test execution script. Since I start this via PowerShell you could just stop it with a simple script that just does `stop-process -name powershell`. But you could also choose to build in a check to see if the process is already running and close the running one at the start of the script. The first solution I have not tried yet but seems like the best way to go if your're not shutting down the VM's. For the second solution I have a sample script of a stopwatchtimer.
I got the idea from this stackoverflow question.

Selenium-grid and browsers

Depending on how you do all of the above, keep in mind that you also give the hub and nodes time to connect to another before the tests start. And if they start to cause trouble if you keep them running, you might consider to restart those as well before you start the test execution. Also if you keep the machine running you could consider checking for open browsers before you start and close those with a script as well. For my situation this was not needed since I choose to shutdown the VM's after each run.
Location: Wassenaar, Netherlands

Comments

Post a Comment

Popular posts from this blog

PowerShell - How to overcome Azure VM's fixed resolution limitation

Image
Intro This is the first in a series of PowerShell blogposts. I don't know a lot of PowerShell so I'm going to cover this part very short. What is PowerShell? PowerShell is a very powerful framework you can use to automate tasks. For more info, use Google or check the Wiki:  https://en.wikipedia.org/wiki/PowerShell For my current project we use Azure VM's that we use as selenium-grid (hub and nodes). And Visual Studio Online for kicking off builds. So I started to use a fair share of PowerShell in my test automation process. During the setup I came accross a couple of problems that I want to share with you all which I'll spread across a couple of posts. And we start with Azure VM's that have a fixed resolution of 1024 x 768 and how to overcome this issue/limitation. Azure VM's resolution limitation  Let's start with the problem we faced. Our website under test, requires the test machine to have at least a certain resolution ( Which is > 1024 ...
Read more

TA Basics: Website Test Automation on mobile devices via Appium server

Image
Recently I have been focussing on test automation with mobile devices. Specifically Android and iOS (no Windows phone because that platform is as good as dead at the moment). I first wanted to make a complete manual from setting up software and devices/simulators up to a small test script. But due to lack of time I'll focus on specific parts. If you have any questions reguarding the setup, feel free to contact me. So all that said, let's get to the point. The internet is full of blogposts, video's and examples of people testing mobile apps. However I don't test an app, I want to do some web test automation on mobile devices. It took me quiet some time to figure out what exactly I had to use to get it all to work as needed for my project. Setup Mac mini running OSX 10.13.4 High Sierra (Mac machine is needed for iOS simulation) Ruby version 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17] Appium desktop version 1.4.1 (with Appium server 1.7.2) lapis_laz...
Read more

TA: Who doesn't like proxies? Me!

Image
Are you on a company network that is using a proxy to connect to the internet? And do you want (or are you planning) to connect your tests to an external selenium-grid. Than stay tuned, before you get lost in a bunch of answers on the internet that don't resolve the issue you're facing. Source:  https://nl.wikipedia.org/wiki/Proxyserver I'm on a new project and looking into Robotframework. But then I bumped into an issue with the POC I had to make which might result in me convincing the company to switch to Ruby. Anyway long story short. They use a proxy and although the internet has all the answers, you do need to know what specific question to ask to get the correct result. And since proxies are not used that much anymore, it's hard to find the skilled people to help you out and push you in the right direction for a solution. The setup To access the internet, I need a configuration on my computer that will tell Windows to connect to a proxy server. I...
Read more