PowerShell - Start your tests in parallel

Intro

Not long after I started to build the first set of tests for our single page web application, I noticed the time it took to run the tests started to increase quiet fast. I started to read about parallel testing, selenium-grid, which ruby gem I could use and that was all quiet interesting. Luckely I already wrote my tests to run independantly. For parallel testing you need to keep a couple more things in mind than just write independant tests and starting them, but I'll make a part 2 where I'll cover it more in depth. In this post I'll cover the basics for using PowerShell to trigger tests to run in parallel.

Setup

As mentioned in previous posts we make use of Visual Studio Online (VSO)as a build server and Azure VM's for our Selenium-grid setup. There are multiple ways to go at this, but we use the grid in the following setup. One hub, multiple nodes. The hub machine also has ruby installed and all needed gems. We use the lapis_lazuli gem for our test suite which uses cucumber. More info about that can be found on the testautomation.info website. Via the build blocks in VSO we push the code to the hub (zipped), then it's being extracted, some RDP sessions are being started (see previous post), and last but not least we start a PowerShell script to execute the cucumber command. At the end of the test execution we collect the test results so they can be published on VSO after everything is done. In this post we focus on the test execution and the lessons learned.

Test execution simple example

In the build server we have a block in which we execute the script that will start the cucumber tests which looks like this:

In the PowerShell script we can now start tests. for example like this:

Param(  [string]$testsLocationArgument ) Start-Process cucumber -WorkingDirectory $testsLocationArgument ` -LoadUserProfile -WindowStyle Normal -Wait;

This starts a new process with the cucumber command in the folder specified in the build block and it will wait for the process to be done before moving on to the next block.

Test execution parallel example

The first thing I did was to figure out how to increase the amount of browsers I could test on in parallel. To do so, you need to run the commands like this:

$par1 = $( # FF   Start-Process cucumber `     -ArgumentList "BROWSER=firefox -t @desktop" `     -WorkingDirectory $testsLocationArgument `     -LoadUserProfile -WindowStyle Normal -PassThru; # Chrome Start-Process cucumber `     -ArgumentList "BROWSER=chrome -t @desktop" `     -WorkingDirectory $testsLocationArgument `     -LoadUserProfile -WindowStyle Normal -PassThru; ) $par1 | Wait-Process

What you basically do here, is bundle two processes in a main-process. Each sub-process is started right after another (hence the -PassThru), but you wait for the main process to be finished, which is done when both sub-processes are finished. This way you can add as many variations/sessions as your setup allows, but still wait for everything to be finished before the build block is considered "done".

Lesson learned

These tests are running for quiet some time. The downside is that if you cancel the build in VSO, this process keeps on running. This means, it will keep on occupying the selenium-grid nodes and that is nto what you want when you cancel a build. You can solve this issue in a couple of ways.

Shutdown VM's 

The easiest is to just shutdown the VM's after each build/test run with a block in your build definition. But since starting up the VM and making sure there is a RDP connection takes time it might not be ideal in case you want to do multiple runs in a row.

PowerShell solutions

There are multiple ways you can go with this and it depends a bit on your setup. But think about a block in your build definition that will end the running test execution script. Since I start this via PowerShell you could just stop it with a simple script that just does `stop-process -name powershell`. But you could also choose to build in a check to see if the process is already running and close the running one at the start of the script. The first solution I have not tried yet but seems like the best way to go if your're not shutting down the VM's. For the second solution I have a sample script of a stopwatchtimer.
I got the idea from this stackoverflow question.

Selenium-grid and browsers

Depending on how you do all of the above, keep in mind that you also give the hub and nodes time to connect to another before the tests start. And if they start to cause trouble if you keep them running, you might consider to restart those as well before you start the test execution. Also if you keep the machine running you could consider checking for open browsers before you start and close those with a script as well. For my situation this was not needed since I choose to shutdown the VM's after each run.