Test Automation on real devices versus emulator/simulator

May 31, 2018

Emulators and simulators look like a cheap alternative compared to testing on real devices. And that is true if you only look at the value of the device. But it's not per see true if you look at the quality of the test results. In this blogpost I'll show a real life example where testing on an iOS simulator failed to reveil a bug that was present on a real device.

Website design (desktop versus mobile)

Unlike the desktop (where there is enough space to show everything at once), the mobile view is limited and therefore part of the information is shown at the time. Our website is following Google's Material design as much as possible, so the mobile site works a bit like the Gmail application on your phone. The hamburger menu is visible at first, but when you open an item (like an e-mail in the Gmail app), the menu button transforms into a back button. So when you want to get back to the menu you need to press the back button first. When you do press the back button, the back button disappears and the hamburger menu button will be shown again. So far it seems straight forward and there is not much that can go wrong right?

Issue

We have regular menu items, which when you click on it a data grid is showing you results, you can click on the results to open them and navigate back and that all works. But there is also search functionality in our application which is using elastic search and shows results across all entities (contacts, tickets, activities, etc). What makes this part different is that it only shows results in the grid after you have performed a search. That is the only difference between the normal pages and the search page. When you perform a search, and the grid is shown and you open an item from that grid, the back button is shown (so far so good, same results as a regular page). When you navigate back the grid is shown again (still, so far so good), BUT, on the real device the back button stays visible and does not turn back into the hamburger menu, while on the iOS simulator this does work. On the Android emulators we see the same as on the real device. So why is this happening?

Cause

Let's start with the fact that there was a bug in this website that does not properly transform the back button into the hamburger menu in this scenario. But why is the iOS simulator not showing us this error where it's visible on the Android emulator and on real iOS/Android devices? The answer is simple. A simulator is not the real device. The same goes for an emulator. These things can mimic the real devices pretty well and in most cases it's sufficient enough. But it's not the real deal. It's another piece of software which can contain bugs or small differences compared to the real deal. And where there are differences in hardware/software, there are differences in issues you might encounter.

Solution

Never only test on emulators. or simulators. It's highly recommended to test your software or website also on a real device. I personally always use a healthy mix of devices, fast, slow, big, small, throw in an odd ball (within the limits of your support). If you need to choose, go for the real deal, because that is what is really representing the devices in real life, so you should run into the same issues that your users/customers would. But it also can't hurt to throw some emulators/simulators in the mix and see what happens. Remember, if you can't find an issue on the real device, it doesn't mean it doesn't exist. And if it exists, it might cause issues later, so better find it as soon as possible. Want to know how to setup an Android emulator or iOS simulator? Check out my previous blogpost about preparing your mac for (mobile) test automation.

Search This Blog

A Test Automation Blog