Mobile Test Strategy

Manual Testing vs. Automation

There are very rare scenarios where a human eye does significantly better than an automated program while validating software for e.g. aesthetics of a website. That gap is also getting narrower by the day for e.g. AI (artificial intelligence) and ML (Machine learning) give a much better ROI in the age of speed and continuous delivery. Visual Test Automation by tools like Applitools is a great example. It is NOT to say that we should completely make manual testing function redundant in an organization, rather it is to say that a testing capability should include both – automation and manual with focus on automation In the context of Mobile testing, it is almost impossible for a human to manually test permutations and combinations across the dimensions of devices, SDK, API level, ecosystems. Hence a testing strategy should include (not limited to)

• Automate as early and often in the bottom-up order of Testing pyramid • Use BDD/TDD frameworks in the early stages when acceptance criteria is not clear and automation script cannot be run (use mock frameworks to mitigate the absence of a real system) • Build and Deployment of app should be self-serviced and automated without any exceptions • Collaborate with other roles in the team and in rare circumstances (during sprint demos), manually walking through workflows of the system is okay (otherwise the automation script should be run as demo)

Below infographic shows most of the open-source/free tools used successfully in most situations (Windows and Firefox ecosystems are not represented as their mobile market share is abysmal to date)

• Programming language – Start with known, tend towards most collaborative
• Inside-out / Outside-in (focus on end-user)
• Don’t boil the ocean – Pick one and pivot
• Interoperability - Cloud & on-premises infra
• Tend towards using Real Devices
• Appium, espresso & XCTest

Speed of Testing

High Speed of testing is a mandatory requirement in mobile space because a mobile app is successful in the market based on how fast it is delivered in the face of competition. There have been research studies pointing that mobile users have an attention span of minutes before they uninstall the app, hence releasing often and bug-free app is a make-or-break situation

Scaling

Think of Aws and Netflix. The services can respond by scaling the systems based on load, performance, user traffic and so on. While testing mobile apps, the testing capability should also be scalable across the 3 V’s (volume, variety and velocity) with a trade-off against cost. In the beginning of developing a project, it is advisable to rely on open-source and cloud services before investing heavily on paid solutions.

Cost

Cost is a constraint in every decision we make when resources are required. Resources can be human, machine, length of time etc. Hence investing in open-source solutions at the beginning is less risky because of no vendor lock-in. It is a trade-off decision because if we spend too much time on not-so-mature open source solutions, then we are losing on time dimension. All said and done, open-source solutions have come a long way and are mature in many ways, in fact much more than some paid solutions in some cases. The strategy here should be spend a week or two and scour the web for solutions and frameworks that many of them already built and use that as starting point.

Testing Pyramid

Testing pyramid helps in terms of having conversations around how much and what priority do different kinds of testing have in a project. A very good article is here