Android and iOS Testing Strategies


Mobile Development, Solution Design

Date: 31/05/2016

We can cut through complexity.

With the number of Android and iOS apps available reaching more than 3.5m in 2015 getting your app installed is a major challenge. Yet how many times have you downloaded an app only to find it difficult to use or crash? So why spend all that effort building an app if it is not fit for purpose? We look at some key mobile app testing strategies to help.

Start at the Beginning

It may be an obvious thing to say, but your approach to testing any software development must start right back at the beginning before you have even fully worked out what your software will do. Why? Because a key element to any piece of software is how it is used. For consumer-facing apps, getting the right user experience is crucial. Make your app hard to use and nobody will install it. So, right at the beginning you must start to test every idea you have for your app — test it with real people by asking them what they think is important, what the app should include and how they might use it. This first real world test might not be what you immediately think of as testing (there's no JUnit in sight, after all), but it is an essential step in any software development. Make the software relevant and usable.

There is another important reason for thinking about testing from the start. When deciding which platform and versions to build your app for, you need to think about how you will test it. For example, a sensible choice for building an iOS app at the moment is to target iOS 7.1 and above, which covers in the region of 99% of all iOS devices in the world. The problem is that you need to think how you are going to test your app on phones running iOS 7.1 (making up nearly 5% of all iOS devices). The current version of Xcode only supports simulators running iOS 8.1 and above. So, do you happen to have an iPhone in your drawer which is running iOS 7.1? If you have, then you probably planned for this or you are quite lucky. Not many businesses have the foresight or money to put aside expensive hardware just in case they might need to test with it.

Design for Testing

With a clear idea of what your app will do and what user experience you want to deliver, now you can start on the technical stuff. While building your technical design, think carefully how you would test each element.

Thinking about testing during the design stage is especially important for mobile apps. If your app does anything that involves sensors, which it is likely to do, then how do you test the range of sensor values you might receive at arbitrary times? For example, both the Android and iOS simulators allow you to set a mock GPS location that will be injected into your app when requested. Connected to your development machine, you can also inject mock locations into real devices. However, this can be a tricky thing to set up, control and automate. Other sensors are much more difficult to mock, such as the camera, and these require changes to your application in order to inject mock data.

In other words, if you want to test all of the sensor inputs used by your app, you will need to think about how this testing mode in your app fits into the design. If you do not build this in from the start, you will either get to the point where you cannot mock sensor input, or you have to re-write sections of your app just to test it. And you do want to test it, don't you?

Unit and Integration Testing

Now we get to the bit of testing that is perhaps more familiar to developers — the long and tedious bit of writing unit tests for all of our classes and integration tests to make sure that all of our classes work together nicely and with the underlying platform.

The key to these tests is repeatability. In the past, repeatability came from writing a long test specification which detailed pre-conditions, test steps and expected outcomes. I've spent many days re-running tests on software manually back when automating tests was in its infancy. For mobile apps, tests can be easily automated so that regression testing and continuous integration mean that your software is tested at every possible step in its development lifecycle. Both Android Studio (plus Eclipse) and Xcode come with mobile app testing frameworks built-in. These allow you to write tests and run them any time you want.

Android uses JUnit for unit and integration testing. Just like any other unit test framework, JUnit allows you define a series of tests which can be run against a class, with appropriate set-up and tear-down methods. Assertion statements are used to check the validity of each step and their output is collated in a test run to show which tests passed and which failed. Android also provides instruments to mock the environment for your app so that you can mock system objects to control the test conditions and Espresso for user interface testing. This includes things like mocking system components, such as activities and services. Tests are then run on the simulator or connected device.

Another popular JUnit testing framework for Android is Robolectric. This does pretty much the same as Android's built-in framework but instead of running on the simulator, which can be excruciatingly slow, the tests are run directly on the development machine within a mocked Android environment. This is much faster and therefore helps with repeatability and continuous integration.

iOS also provides a comprehensive unit test framework. The XCTest framework does the same job as JUnit in Android, including allowing you to mock the environment. You can also use KIF for user interface testing. Xcode also has code coverage built in, and allows you to record user interface interactions for repeated testing.

The outcome of all of this is that you should aim to have a comprehensive set of tests which test every class and their interaction, within a mocked mobile environment. This will give you confidence that the nuts and bolts of your app are working, and give you a basis for continuously re-running your tests at every change. Here, you can use fully-fledged continuous integration services, such as Jenkins.

Moving into the Real World

Writing your code and unit tests will probably take most of your time and interest. But before you go running off to release your app to the app store, you better make sure it works in the real world.

The first thing to do is test your app on all available hardware and platform versions you can. For example, if you are building your app to run on iOS 7.1 and above, then you need to test on iOS 7.1, 8.1, 8.2, etc. Oh yes, and iOS 7.1 on iPhone 4S, iPhone 5, etc. The simulator will help to a degree, but obviously not using real world hardware. This is where services such as Amazon Device Farm come into their own. Device Farm currently has 125 iOS and 160 Android/Fire OS devices with various platform versions. More than I can fit in my drawer.

To test using Device Farm you will need to have written automated tests. Do you now see the importance of having written automated unit tests? Supported testing frameworks include Appium, JUnit (with Espresso and Robolectric) and XCTest. Fire off your tests to your selection of devices and wait for the results to come back (oh yes, and a bill for the time spent using Amazon's devices).

Back to the User

After all this you should be confident that your app works from the ground up and on all target devices. Of course, there are bound to be bugs — no software in life is bug free — but at least you have something that basically works. The next stage is therefore to release it to real people who really do have a knack of breaking things very easily.

Using Google Play's alpha and beta test tool, Apple's TestFlight or third-party tools such as HockeyApp, you can release your app to a limited number of people who can use and test it in the real world. This is a crucial step if you want your app to work well on official release — it allows you to get rid of those bugs that only real life will show up. It will also allow you to tweak the user interface or flow so that your app is used. You can even do this within specialist testing services which will allow you to release different versions of your app for A/B testing, such as via Optimizely.

Strategies at a Glance

Summarising the key strategies with example frameworks, we have:

Android iOS
  • Evaluate your app idea with potential end users: focus groups and surveys.
  • Consider your overall approach to testing and how this will change depending on the app functionality selected.
  • Consider what testing will need to be conducted using mocked environments and sensor readings.
  • Build into your design any suitable test modes or data injection methods needed to complete testing.
Device Testing
  • Test on all available hardware and platform version combinations via services such as Amazon Device Farm.
  • Testing will use your automated unit and integration tests, new tests or those developed using tools such as Appium.
User Testing Release your app to a limited set of users via Google Play, Apple's TestFlight or other services such as HockeyApp.
  • Do not just release and forget your app. Put effort into responding to user comments, fixing bugs and releasing enhancements.
  • Every time you make a change to your app, re-run all of the tests, perhaps using continuous integration environments such as Jenkins.

By including testing from the very first step in app development through to market testing in the real world, you should have an app which works well and does what you set out to achieve. Of course, this is still no guarantee that people will install it, but at least you know that when they do, it will work.

We can help

Digital processing is at the heart of innovation. It may be that you have a difficult or complex problem. Pervasive Intelligence have the expertise to cut through the complexity to give you a solution you can use in the real world.

If you would like to know more, or are interested in what support we can provide, take a look at what we can offer and get in touch. To receive similar insight, just follow us on .