The Test Log: Xamarin Test Cloud’s Indispensable Debugging Tool

It’s a Log Way to the Top

(if you want a 5 star app)

It may not seem like much, but this feature may add quite a bit of insight to your UI Tests if you haven’t used it before.

Since launch, Xamarin Test Cloud has provided three types of very useful logs:

  • The Console Log, which is syslog relay on iOS or logcat on Android;
  • Test Failures, which appear when assertions in the test raise exceptions;
  • Stack Trace, to help pinpoint the source of errors and crashes in the actual codebase of the app.

All of that is useful and critical information to a tester. Until recently, there was no easy way to write your own custom information to a log unless you were using the app.Invoke “backdoor logging” hack.

Now, all you have to do is simply write whatever you want to stdout in your test script. In Xamarin.UITest, you would use Console.WriteLine(string). This appears in a new area called the Test Log, which is shown here in a single device view.

The Test Log button.
The Test Log button in a Xamarin Test Cloud report’s single device view.

Upon investigating the Test Log, we see this warm, fuzzy, familiar output (for NUnit users):

Our friendly console log. It's big, it's better, it's good!
Our friendly console log. It’s big, it’s better, it’s good!

This means that you can very easily write anything you like to the console, in any way you choose (NLog; whatever that crazy enterprise logging block from .NET 2 was; you name it!) The most basic approach is to call Console.WriteLine("message") from your test. As tests run across the devices, each device generates its own test log for that session.

The Test Log is free and easy to understand, so you should use it often.

How to use the Test Log

When you’re working with any remotely hosted resource, having as much log data available to you is absolutely critical. Your app running on the device can generate its own logs, which appear in Console Log tab, but what about logging from the execution of the test?  Why would we do that? Here are some common applications that you can use today.

  1. Better understand what’s happening on each device individually. If your test fails on a 3″ screen because the keyboard blocks the password field, you can know for sure with some nice logging. Just create a nice extension method called ToLogFormattedString for the Xamarin.UITest.AppResult class, and then use Console.WriteLine(app.Query().ToLogFormattedString()) in the test. This will dump out a pretty-printed control hierarchy for you to use. Since app.Query() returns only fully visible elements by default, you can use this technique to find out what elements can and cannot be “seen” by the automation framework (and by extension, your users) on each device.
  2. Generate your own diagnostics and metrics. If you’re trying to optimize tests, you can output execution timing to that log.
  3. Validate response data from service endpoints. Some E2E tests go all the way back to the database. An example use case would be creating a user: you’d use the UI to create a new user with a randomly generated, unique user ID created by the test code. You can automate the creation workflow, and at the end of the process, hit your REST API and try to pull a valid user with that same ID – you can then assert that the fields in the database match the ones entered by your test. Outputting JSON to the console log can help you understand what exactly is going over the wire.
  4. Sanity check your work. Because sometimes you need to make sure your code “got to there.”

The test log is just another way Xamarin Test Cloud gives you the information you need, when you need it, to get faster feedback and make better apps for the world to enjoy.

Add-in For Building Xamarin.UITest Page Objects Faster And More Accurately

This post is all about creating precise, stable automation suites as quickly as possible.

The explanation

In the last post (which was way too long ago), I left off with a couple of enhancements that I promised to cover next:

  • Remove the IApp dependency in pages
  • Knowing what page you’re on
  • Handling page transitions without extraneous waits
  • Generic loading of pages for cross-platform tests

I’m going to cover a simple evolution of what I described last time, but with the added bonus of a helpful add-in for Xamarin Studio. We won’t cover dependency injection or cross-platform just yet.

Much of my current role here at Xamarin involves the creation of custom Xamarin Test Cloud demos for prospective customers. I do a lot of custom demos, and my primary concern with the tests that I generate is not unlike what you’d want from your testers anyway. What I’m looking for in an approach is as follows:

  • The test is human-readable.
  • The test is easy to write and there is a reasonable pattern to follow.
  • The various methods in the test can be parameterized with different data.
  • Page objects can be composed together by someone with little coding experience and can be expected to “just work.”
  • All the difficult automation work is centralized and concerns are properly separated.

What I’m about to offer is certainly not perfect, and it’s not a prescription. It’s just what works for me every day in the context of Xamarin Test Cloud, and I’m happy to share it with you. You may want to extend it and add features, and that’s great!

Explaining The Approach

Our goal is to get to beautiful, fluent-looking tests that look like this:

[Test]
public void SuccessfullyPlaceAnOrder()
{
    app.LoginPage()
       .LogIn("testUser", "testPassword")
       .GoToHomeScreen()
       .ViewOrders()
       .SelectFirstOrder()
       .VerifyOrderDetails();
}

Notice how you can read this very clearly, and you can pass different arguments to methods that accept them. This type of approach works well in a single SDET type of environment, as well as where you have multiple people utilizing the page objects for one app to build out the test suite.

The key here is that all the page operations are centralized into that page’s class, so you never do the same thing in two different ways.

There are some more hidden thingies in here that I need to explain.

How the Fluent Syntax Works

Right off the bat, we have this weird line of code which certainly doesn’t exist in vanilla UITest: app.LoginPage()

How did our beloved app variable get knowledge about some login page? The secret is that we used an extension method.

The LoginPage() extension method actually instantiates and returns an instance of the LoginPage page object, also telling the new LoginPage to use the same app variable to perform all of its necessary operations. Let’s take a look at how this extension method works:

[Test]
public static LoginPage LoginPage (this IApp app)
{
    var page = new LoginPage (app);
    if (app.TraitExists (page.Trait))
        return page;
	
    throw new Exception ("The Login page was not found.");
}

You’re also probably going to notice this concept of a Trait. We’ll get into that later, but for now, just know that it is a strongly typed encapsulation of a query that every page has, and can be used for seamlessly transitioning between pages.

Once we have that LoginPage, we can start calling methods that exist on that object, for example, LogIn(string username, string password).

Pretty much all operations on the login page return the current instance of the login page, unless the operation results in a transition to another page. Compare the return types and return statements of these two methods:

public MainPage ValidLogIn (string username, string password)
{
	app.WaitForElement (usernameField);
	app.Screenshot ("On the Login page");

	app.EnterText (usernameField, username);
	app.Screenshot ("Enter Username");

	app.EnterText (passwordField, password);
	app.Screenshot ("Enter Password");

	app.Tap (loginButton);
	app.Screenshot ("Tap 'Log In'");

        app.WaitForNoElement("activity_indicator");

	return new MainPage(app); // transition to new page
}

public LoginPage InvalidLogIn (string username, string badPassword)
{
	app.WaitForElement (usernameField);
	app.Screenshot ("On the Login page");

	app.EnterText (usernameField, username);
	app.Screenshot ("Enter Username");

	app.EnterText (passwordField, badPassword);
	app.Screenshot ("Enter Bad Password");

	app.Tap (loginButton);
	app.Screenshot ("Tap 'Log In'");

        app.WaitForElement("error_message");

	return this; // stay on the Login page
}

This way, as you create page objects, you’re writing the implementation for each operation directly into that page object, and the app flows from page to page.

“Automatic” Waiting

This framework provides a default constructor template that will try to wait for the trait when the page object is created and then takes a screenshot. This supports the fluent style of calling methods shown above.

Let’s look at the constructor for a page object, so you can see how this pseudo-automatic wait protection works:

public LoginPage (IApp app)
    : base (app)
{
    Trait = new Trait (usernameField);
    app.WaitForTrait (Trait);

    app.Screenshot ("On the Login page");
}

The Trait class is constructed with the default type of a query in Xamarin.UITest, which is System.Func<AppQuery, AppQuery>, which will allow you to use any typical query from UITest as your page’s trait – such as e => e.Id("username").

Xamarin.UITest Page Objects Add-in

Behold the following Xamarin Studio add-in:

Xamarin.UITest Page Objects

This repo contains the basics for getting started with this approach, as well as an mpack file which you can install into Xamarin Studio by way of the Addin Manager.

Using the Add-in for Faster Testing

Once you’ve installed the add-in, you will have access to a bunch of helpful classes that support this programming model. This add-in will extend the default single-platform tests that come with Xamarin Studio quite nicely.

Follow these steps:

  1. Create a new platform-specific test project from the Xamarin Studio templates, for example, a new Android test project.
  2. Right-click your project and add a new file.
  3. Choose UITest Helpers on the left pane, and choose UITest Page Object with Setup. You will be prompted to enter a name. This template by default will already create a LoginPage for you, but you should specify another page to work on here. If you don’t want or need this extra page, just delete it.
  4. Type app.LoginPage() in your test, and see how you can extend it according to the approach above.

Next up, we’ll have a screencast on using these tools to your advantage to create high-quality UI automation tests faster.

Let me know if you have any issues with the add-in, or if you’d like to make some contributions to the project.

Building a Scalable Test Suite with Xamarin.UITest and Page Objects

Introduction

The act of developing an automated test suite is like an odd cousin of product-oriented software development. The skills required are just difficult enough to be out of reach of your average manual tester, and in fact, to do it well takes some serious dedication to the craft of software engineering. And yet, there are still many developers who don’t write their own tests and rely on someone else to run test suites on their work. So how can we make our tests both easier to author and easier to understand?

The BDD approach came onto the scene in a big way in 2009, and is still actively in use at many organizations, but it does require organizational coordination to make the most of this methodology. It’s not for everybody. However, imagine a test suite written procedurally with little to no reuse, filled with hard-coded query lambdas and timing values. It starts out easy but is going to be impossible to read and maintain over the long term. And unlike BDD, nobody can tell at a glance what the test does unless he or she is an engineer.

Is there a middle ground? Can we improve readability and maintainability, keep the code self-documenting, and create a bunch of composable pieces which anyone can use?

Page Objects in Xamarin.UITest

A page object is a concept. There is no page object thingy that you can instantiate or construct that Xamarin.UITest provides (although Calabash does currently provide this as a predefined framework construct).

When I think of a page object, I think of a C# POCO class that has some simple properties and several methods that model the behavior of the application under test. That’s it. This class encapsulates the queries required for your test to understand and manipulate the application.

The best part about this approach is that it can make you feel very creative. Automated testing can be very boring and repetitive, which is why software engineers try to get away with not doing it. Add a little bit of architecture work and suddenly it becomes fun!

Lastly, I will point out that just like everything else in C#, there are many valid ways to do this. Take the basic concepts and shape them into a solution that fits well with your team, design practices, and philosophy towards testing. My approach isn’t necessarily the one that will work the best for you. Later, I’ll list some common things that may need to be modified or may suit your style better.

As you begin designing your page object infrastructure, it’s important to remember a few design principles that apply to object-oriented design that can help you build great tests:

  • YAGNI (You Ain’t Gonna Need It): It’s easy to get carried away. Build only what you need, when you need it. Don’t try to predict the future.
  • SOLID: This encompasses a whole set of design principles laid out by Uncle Bob that will help you write better code no matter what you’re doing. Single Responsibility Principle and Dependency Inversion are particularly useful in tests, as is the Liskov Substitution Principle.

The Page Object Makeover, Phase 1

I’m going to show you the easiest possible way to start structuring tests as page objects.

Let’s imagine a very simple test that logs in to an app. You might start off like this:


[Test]
public void Login()
{
   app.Screenshot("On the login page");
   app.WaitForElement(e => e.Id("loginButton"));
   app.EnterText(e => e.Id("username_field"), "username");
   app.EnterText(e => e.Id("password_field"), "password");
   app.Tap(e => e.Id("loginButton"));
}

There is really nothing wrong with this code. However, when we have 100 lines of it, it starts to become a real problem. Then when we want to reuse it, it becomes quite a nightmare if you’ve copied and pasted these 100 lines into other scenarios.

Let’s move it into a new class:

using System;
using Xamarin.UITest;

public class LoginPage
{
   private IApp app;

   public LoginPage(IApp app)
   {
      this.app = app;
   }

   public void Login(string username, string password)
   {
      app.Screenshot("On the login page");
      app.WaitForElement(e => e.Id("loginButton"));
      app.EnterText(e => e.Id("username_field"), username);
      app.EnterText(e => e.Id("password_field"), password);
      app.Tap(e => e.Id("loginButton"));
   }
}

And now the test method in Tests.cs looks like this:

[Test]
public void Login()
{
   var loginPage = new LoginPage(this.app);
   loginPage.Login("demoUser", "demoPassword");
}

Now we’re cooking. That’s a lot better. Now anytime I need to log in to the app, I just instantiate a LoginPage and call this method. This is the most basic refactoring to page objects that you can do, and even this is going to save you a ton of time. So start here!

Enhancements

Looking at this code, you probably have some questions – and these will be answered in the next blog post. If I saw this code for the first time, I’d ask:

  • So every page object needs to be constructed with an IApp? It would be better if we remove that dependency. (Yes, it would).
  • How do I actually know what page I’m on?
  • How do I seamlessly handle transitions from screen to screen?
  • Is there a way to generically load page objects for cross-platform tests?

All of this will be addressed in the next blog post. In the meantime, I’ll let on that I like to use base pages (superclasses with protected IApp fields) and dependency injection (with Autofac) to decouple tests and page objects from their dependencies.

Conclusion

Automated testing, by its very nature, is repetitive. That login scenario might need to prefix every other test in your test suite, so why would you copy and paste it all over the place? Even worse, what happens when the login page changes – maybe an animation is added – and now you have to go back through 100 scenarios and add a wait before tapping the Sign In button. That makes automated testing suck so much more. Once you realize you have a grunt work task to repeat 100 times, it’s just not fun anymore. Wouldn’t it be better just to have all Login page related logic in one place? Hence, page objects.

This approach, and many like it, will ultimately lead to cleaner tests that are easy to maintain and read, and also prevent you needing to change things all over the place when simple behaviors in the app change.

Configuring backdoor methods in C# for Xamarin Test Cloud

One of the most powerful mechanisms we offer in Xamarin Test Cloud is the ability to invoke specific code in your application from your UI Tests. This is called invoking a backdoor method.

Like the Xamarin Test Cloud Agent for iOS, backdoors should never be enabled in your release build as they could pose a security risk. Your test/debug build should have all the backdoors and configuration needed for proper UI testing.

Let’s say I’ve built a Xamarin.Forms application that simply loads a WebView, and I want to control the loaded URL via a backdoor method. In other words, I want the test to control what gets loaded in the WebView.

iOS

On iOS, the method signature for a backdoor method needs to look like this:


[Export("SetWebviewUrl:")]

public NSString SetWebviewUrl(NSString webviewUrl)

{

// set the webview's .Source property to webviewUrl

}

A couple of things to point out:

  • The signature must have NSString for both the return type and the argument.
  • The Export attribute is found in System.ComponentModel.Composition, so make sure to add a reference to this assembly.
  • Note the trailing colon in the method name in the Export attribute.

The backdoor method is invoked like this:


app.Invoke("SetWebviewUrl", "http://www.xamarin.com");

Android

On Android, it’s a little bit different. First of all, the reference we need is not System.ComponentModel.Composition – you need to add Mono.Android.Export as a reference instead.

On Android, the method may take any number of arguments as well as try to type convert the return value for you. This method is nonetheless the same on Android:


[Java.Interop.Export("SetWebviewUrl")] // note there is no trailing colon

public String SetWebviewUrl(String webviewUrl)

{

// Set the webview URL

}

And this method is invoked in exactly the same way:


app.Invoke("SetWebviewUrl", "http://www.xamarin.com");

Happy Testing!

Why You Should Start Testing Your Mobile Apps Now

At Xamarin, I have a unique opportunity to speak with a lot of organizations at various levels of capability in mobile testing. One thing I’ve learned from these conversations is that mobile testing is still a relatively new thing compared to automation of workstation or web apps. Not everyone has yet adopted the mindset that mobile automated testing is important enough to actually get serious and do it.

Quality on mobile is especially hard

In the enterprise, we see a lot of organizations replatforming their line of business apps. What used to be done sitting at a desk is now done out in the field, at home, or even on an airplane. Apps that look and perform well (natively) on all form factors are the most likely to be adopted by users, versus apps that are slow, crashy, or buggy.

We know that mobile quality is difficult. Mobile development itself is difficult – that’s why Xamarin exists; we believe there is a better way to build, test, and monitor mobile apps. As a consequence of this perceived difficulty, I frequently hear from folks that while they may have a highly mature process for their web and desktop apps, they don’t feel confident in their mobile quality, don’t have their mobile apps building in CI, and in some cases it’s not even in a VCS system.

We have a responsibility and a desire to make mobile development & testing feel natural and easy. Many teams focus on feature delivery but not testing, in hopes of showing progress and getting the app out the door fast. Unfortunately, they are setting up to go to market with a poorly tested or entirely untested app – which causes weeks of last-minute refactoring and debugging, unhappy product owners, delays in the schedule and other problems.

Testing is not an afterthought. The earlier you implement proper testing, the more value you will get from it. It’s that simple.

There also is a lot of expectation placed on the success of mobile projects. For example, Gartner forecasts that less than 0.01% of consumer mobile projects will be considered a financial success – through 2018!  by their developers. This can make a dev team feel like they are up against some pretty serious odds.

The last thing you want your team to worry about is your app crashing on one of the 19,000 unique Android configurations on the market, or suddenly not working on iOS 8, or looking terrible on screen sizes you haven’t considered.

Typical approaches to mobile testing

Based on the conversations I’ve had with mobile dev teams over the last six months or so, I rarely have heard that a team embraces automated UI testing from the beginning – but those teams are quite successful. In fact, most dev teams I speak with are frankly embarrassed even with their unit test coverage. (Why? Is your unit test coverage on other non-mobile platforms equally poor?) I attribute some of this to the fact that people are still figuring out how to write mobile apps, and testing feels like a whole other skillset – but it doesn’t need to be.

Mobile apps are insanely complex. You’ve got a (hopefully native) UI, calling REST services and populating data asynchronously, sitting on top of a set of device APIs on top of an operating system which may or may not be up to date, running on a hardware configuration that may be a vanilla iPhone 5c or something completely nonstandard. A UI test has the benefit of implicitly testing each of these layers of complexity as an integration unit.

So, while unit testing is very important for testing your business logic in isolation, having functional tests is where the real power lies in ensuring your app actually works.

When should I start writing automated UI tests?

The easy answer is “start now.”

If you haven’t started your project yet, the answer to this question is pretty simple. You want to fail fast. Don’t build 10 screens and then see what they look like on the top 10 marketshare devices. You are going to be really disappointed with the amount of work you now need to do. Now, imagine doing this same process 3 weeks before you release, when your UI is baked and your app is nearly complete – would you rather spend the final 3 weeks of your project tying up & polishing functionality, or completely refactoring things to support a core set of devices that you haven’t been testing on?

You should write your first UI test as soon as you complete your first screen. It does not matter if the UI will change later; great tools exist for refactoring code and you should be prepared to refactor & improve tests in the same way you do your codebase. If you’re using some object-based testing framework like Calabash or Xamarin.UITest, you will have the benefit of tests that evolve with your UI without too many changes. A functional test should be an acceptance criteria / deliverable / Definition of Done checklist item for every new piece of UI that you build or change.

“But that sounds like a lot of work! It’s going to slow me down!”

Yes, it is work. It will take time. When you plan the project, you should be adding a 20% buffer to your estimates for quality efforts anyway (expect to spend 8 hours of a 40 hour week writing unit and functional tests). The investment is well worth it – you and your management will always know the quality of your app at any given time in your cycle, instead of flying blind and getting bad news before your release, or looking bad when demoing to a product owner when something breaks.

Start creating functional tests as soon as you can.

Not all teams have the luxury of starting from day 1 – many teams hit some inflection point where some unbearable amount of pain is felt and they realize they need to start writing tests. It’s never too late to instill a culture of ensuring quality.

But where do I start?

Say you’re 6 months into a 12 month project and you only have unit tests – no problem, don’t despair! You should start with a simple happy path test. Have a developer create a functional test that exercises the basic scenario in your app that should never break if the user’s not doing anything weird, for example:

  1. I log in with a valid user account,
  2. Browse products and select the first one,
  3. Add it to the basket,
  4. Assert that I see the item in my basket,
  5. Confirm the order,
  6. Assert that I see a confirmation of my order.

This is a really basic test and should require about 10 lines of code using Xamarin.UITest. And the best part is that every time you do this, you have an automated, codified test that lives on and can test this part of your app forever at the click of a button. As you create more tests, you add more coverage and usefulness to the test suite. Now, scale that out that with the power of Xamarin Test Cloud, and run this test suite regularly across the 1000+ real devices in our cloud for quick feedback about what your app does on real devices in the field.

In conclusion: there is no investment in testing that doesn’t result in better app quality. Starting now is going to provide massive benefits over never starting.

If you’re taking your app to market fast in a Ferrari, Xamarin Test Cloud is your airbag system.

We’re here to help. 

The team at Xamarin has years and years of expertise in the field of mobile testing. We are fanatical about mobile quality, and we want to help you get there too – in a way that feels fun and not like a chore. Get in touch with our sales team: sales@xamarin.com or check out our Xamarin Test Cloud Forums to get help from our automation engineers.