Mobile Device Beta Testing: Emulators or Actual Devices

When it comes to beta testing on mobile devices, many software quality assurance professionals have argued the merits of using emulators versus actual devices. Like most things, there isn’t any one single, simple answer, which shouldn’t come as a shock to anyone familiar with the world of software development. It is analogous to asking which is better, a fork or a spoon. A spoon isn’t much good for eating steak and a fork is worthless for eating soup. However, they both have their time and place, so to speak.

Emulators are cheap and convenient. What’s more, with some work, an automated testing solution can be crafted to take advantage of the emulator’s UI. There are no devices to store, keep track of, or maintain. There are no data plans to upkeep, no updates to download, and no hardware to replace in the event the device is damaged.

On the other hand, emulators tend to be slow to load. The newer emulators, such as the Intel emulator, do load faster, but initial setup can still eat up time, especially if multiple shutdown/reload cycles are needed to clear out a bug that caused the emulator to crash. Emulators themselves can also be buggy. So, that bug you are trying to pin down in the app may not actually be the app at all. In addition, all interactions with the device under testing must be done through the emulator. At first glance, that may seem pretty obvious, but it has some undesirable implications and raises some important testing questions.

For example, certain hardware functionality, such as the camera and the microphone, cannot be tested. Also, you can’t really test the installation and uninstallation of the app. Furthermore, how will the app operate in a low signal area or when the device is going into a dead zone?  Is there a seamless transition from 3 or 4g to Wi-Fi and vice versa?  Does the app recover gracefully from a sudden loss of power?  How will the app behave if the device orientation cannot be determined – will it go to a default orientation or will it just spin between portrait and landscape modes?  This kind of beta testing requires a physical component that emulators simply cannot currently mimic.  Finally, there is the problem with response latency.  There currently is no good way to acquire valid metrics for real time performance of an app on an emulator.

So does this mean emulators are worthless for mobile QA?  No, it does not.  Emulators most certainly have a place in testing.  But because of the limitations present, emulators are only really useful as a software developer tool, for alpha testing, and in the early stages of beta testing.  After that, it’s time to test on an actual, physical device.