Jumping Into Mobile Application Testing – A Continuously Moving Target

Adoption of mobile devices is at an all-time high and the demographics of the technology market are changing. If the user base changes, user acceptance testing must necessarily change. You can probably trust a Linux user running your command line app to steer pretty close to the intended workflow, but mobile devices are now in the hands of people from all walks of life and technology experience levels. According to a study done by mobile analytics company Flurry:

“Compared to recent technologies, smart device adoption is being adopted 10X faster than that of the 80s PC revolution, 2X faster than that of 90s Internet Boom and 3X faster than that of recent social network adoption. Five years into the smart device growth curve, expansion of this new technology is rapidly expanding beyond early adopter markets such as North America and Western Europe, creating a true worldwide addressable market.“ (http://blog.utest.com/).

And if mobile devices are in everyone’s pocket, it means that they are everywhere. This brings us to yet another configuration variable unique to the mobile market: carriers. The iPhone and/or iPad alone are available through more than 100 different carriers worldwide. Here’s a quick (approximate) region by-region breakdown:

  • North America: 11 carriers
  • Europe: 44 carriers
  • South and Latin America: 19 carriers
  • Asia: 35 carriers
  • Middle East: 12 carriers
  • Africa: 7 carriers

Each carrier presents a unique coverage footprint with varying signal strengths throughout to consider. How does your app perform with weak or overloaded signals? There is also the increasing divide between 3G and 4G coverage (http://c954852.r52.cf0.rackcdn.com/).

This is one of the key challenges of mobile application testing: to keep your test matrix from getting out of control under so many configuration variables. I expect that risk management (as usual) will continue to have a large role to play in this, but it is also worth mentioning that the quick adoption of mobile devices has the almost Newtonian side effect of causing quick, relentless obsolescence. Apple has already discontinued iOS 3 (released 4 years ago; compare that with Microsoft’s plans to discontinue support for the now 11-year old Windows XP platform in 2014 http://www.theregister.co.uk).

The same applies to several older iOS devices, including the first two iPhone models (with the first iPhone release being relatively recent, in January 2007 http://en.wikipedia.org/). Quick change, quick adoption and quick obsolescence means that there is relatively little risk in limiting your testing scope to the current versions, an observation which appears to be consistent with the configuration lists I’ve seen in real-world mobile test plans. More carriers, features and regions are also to be expected though.

To end on a fun note, I challenge you to complete this test on mobile apps (the last question is especially dedicated to testers):

  • True or False: “App” was the American Dialect Society’s 2010 Word of the Year.
  • True or False: As of 2012, the iOS operating system has greater market share than Android.
  • True or False: Angry Birds is the #1 downloaded app of all-time.
  • True or False: There are over 6k distinct Android devices.
  • True or False: More people spend time on the mobile web versus native apps.
  • True or False: The Android version of Siri is named Veronica.
  • True or False: Draw Something was developed by Rovio.
  • True or False: Apple Co-Founder Steve Wozniak owns (and loves) a Windows Phone.
  • True or False: Free apps have faster load times than paid apps.
  • True or False: Android apps crash with greater frequency than iOS.

Please visit below for the answers:


About Camilo Villamizar

Camilo Villamizar Results-driven IT analyst/developer with 10 years experience in all stages and areas of the software development process. Proven track record performing in large-scale, complex projects for global companies and government organizations. Camilo is currently a consultant with Accreon Inc. in Fredericton, New Brunswick. LinkedIn Profile
This entry was posted in  All, Other and tagged , . Bookmark the permalink.