At Xamarin, I have a unique opportunity to speak with a lot of organizations at various levels of capability in mobile testing. One thing I’ve learned from these conversations is that mobile testing is still a relatively new thing compared to automation of workstation or web apps. Not everyone has yet adopted the mindset that mobile automated testing is important enough to actually get serious and do it.
Quality on mobile is especially hard
In the enterprise, we see a lot of organizations replatforming their line of business apps. What used to be done sitting at a desk is now done out in the field, at home, or even on an airplane. Apps that look and perform well (natively) on all form factors are the most likely to be adopted by users, versus apps that are slow, crashy, or buggy.
We know that mobile quality is difficult. Mobile development itself is difficult – that’s why Xamarin exists; we believe there is a better way to build, test, and monitor mobile apps. As a consequence of this perceived difficulty, I frequently hear from folks that while they may have a highly mature process for their web and desktop apps, they don’t feel confident in their mobile quality, don’t have their mobile apps building in CI, and in some cases it’s not even in a VCS system.
We have a responsibility and a desire to make mobile development & testing feel natural and easy. Many teams focus on feature delivery but not testing, in hopes of showing progress and getting the app out the door fast. Unfortunately, they are setting up to go to market with a poorly tested or entirely untested app – which causes weeks of last-minute refactoring and debugging, unhappy product owners, delays in the schedule and other problems.
Testing is not an afterthought. The earlier you implement proper testing, the more value you will get from it. It’s that simple.
There also is a lot of expectation placed on the success of mobile projects. For example, Gartner forecasts that less than 0.01% of consumer mobile projects will be considered a financial success – through 2018! by their developers. This can make a dev team feel like they are up against some pretty serious odds.
The last thing you want your team to worry about is your app crashing on one of the 19,000 unique Android configurations on the market, or suddenly not working on iOS 8, or looking terrible on screen sizes you haven’t considered.
Typical approaches to mobile testing
Based on the conversations I’ve had with mobile dev teams over the last six months or so, I rarely have heard that a team embraces automated UI testing from the beginning – but those teams are quite successful. In fact, most dev teams I speak with are frankly embarrassed even with their unit test coverage. (Why? Is your unit test coverage on other non-mobile platforms equally poor?) I attribute some of this to the fact that people are still figuring out how to write mobile apps, and testing feels like a whole other skillset – but it doesn’t need to be.
Mobile apps are insanely complex. You’ve got a (hopefully native) UI, calling REST services and populating data asynchronously, sitting on top of a set of device APIs on top of an operating system which may or may not be up to date, running on a hardware configuration that may be a vanilla iPhone 5c or something completely nonstandard. A UI test has the benefit of implicitly testing each of these layers of complexity as an integration unit.
So, while unit testing is very important for testing your business logic in isolation, having functional tests is where the real power lies in ensuring your app actually works.
When should I start writing automated UI tests?
The easy answer is “start now.”
If you haven’t started your project yet, the answer to this question is pretty simple. You want to fail fast. Don’t build 10 screens and then see what they look like on the top 10 marketshare devices. You are going to be really disappointed with the amount of work you now need to do. Now, imagine doing this same process 3 weeks before you release, when your UI is baked and your app is nearly complete – would you rather spend the final 3 weeks of your project tying up & polishing functionality, or completely refactoring things to support a core set of devices that you haven’t been testing on?
You should write your first UI test as soon as you complete your first screen. It does not matter if the UI will change later; great tools exist for refactoring code and you should be prepared to refactor & improve tests in the same way you do your codebase. If you’re using some object-based testing framework like Calabash or Xamarin.UITest, you will have the benefit of tests that evolve with your UI without too many changes. A functional test should be an acceptance criteria / deliverable / Definition of Done checklist item for every new piece of UI that you build or change.
“But that sounds like a lot of work! It’s going to slow me down!”
Yes, it is work. It will take time. When you plan the project, you should be adding a 20% buffer to your estimates for quality efforts anyway (expect to spend 8 hours of a 40 hour week writing unit and functional tests). The investment is well worth it – you and your management will always know the quality of your app at any given time in your cycle, instead of flying blind and getting bad news before your release, or looking bad when demoing to a product owner when something breaks.
Start creating functional tests as soon as you can.
Not all teams have the luxury of starting from day 1 – many teams hit some inflection point where some unbearable amount of pain is felt and they realize they need to start writing tests. It’s never too late to instill a culture of ensuring quality.
But where do I start?
Say you’re 6 months into a 12 month project and you only have unit tests – no problem, don’t despair! You should start with a simple happy path test. Have a developer create a functional test that exercises the basic scenario in your app that should never break if the user’s not doing anything weird, for example:
- I log in with a valid user account,
- Browse products and select the first one,
- Add it to the basket,
- Assert that I see the item in my basket,
- Confirm the order,
- Assert that I see a confirmation of my order.
This is a really basic test and should require about 10 lines of code using Xamarin.UITest. And the best part is that every time you do this, you have an automated, codified test that lives on and can test this part of your app forever at the click of a button. As you create more tests, you add more coverage and usefulness to the test suite. Now, scale that out that with the power of Xamarin Test Cloud, and run this test suite regularly across the 1000+ real devices in our cloud for quick feedback about what your app does on real devices in the field.
In conclusion: there is no investment in testing that doesn’t result in better app quality. Starting now is going to provide massive benefits over never starting.
If you’re taking your app to market fast in a Ferrari, Xamarin Test Cloud is your airbag system.
We’re here to help.
The team at Xamarin has years and years of expertise in the field of mobile testing. We are fanatical about mobile quality, and we want to help you get there too – in a way that feels fun and not like a chore. Get in touch with our sales team: firstname.lastname@example.org or check out our Xamarin Test Cloud Forums to get help from our automation engineers.