Month: August 2015

The Test Log: Xamarin Test Cloud’s Indispensable Debugging Tool

It’s a Log Way to the Top

(if you want a 5 star app)

It may not seem like much, but this feature may add quite a bit of insight to your UI Tests if you haven’t used it before.

Since launch, Xamarin Test Cloud has provided three types of very useful logs:

  • The Console Log, which is syslog relay on iOS or logcat on Android;
  • Test Failures, which appear when assertions in the test raise exceptions;
  • Stack Trace, to help pinpoint the source of errors and crashes in the actual codebase of the app.

All of that is useful and critical information to a tester. Until recently, there was no easy way to write your own custom information to a log unless you were using the app.Invoke “backdoor logging” hack.

Now, all you have to do is simply write whatever you want to stdout in your test script. In Xamarin.UITest, you would use Console.WriteLine(string). This appears in a new area called the Test Log, which is shown here in a single device view.

The Test Log button.
The Test Log button in a Xamarin Test Cloud report’s single device view.

Upon investigating the Test Log, we see this warm, fuzzy, familiar output (for NUnit users):

Our friendly console log. It's big, it's better, it's good!
Our friendly console log. It’s big, it’s better, it’s good!

This means that you can very easily write anything you like to the console, in any way you choose (NLog; whatever that crazy enterprise logging block from .NET 2 was; you name it!) The most basic approach is to call Console.WriteLine("message") from your test. As tests run across the devices, each device generates its own test log for that session.

The Test Log is free and easy to understand, so you should use it often.

How to use the Test Log

When you’re working with any remotely hosted resource, having as much log data available to you is absolutely critical. Your app running on the device can generate its own logs, which appear in Console Log tab, but what about logging from the execution of the test?  Why would we do that? Here are some common applications that you can use today.

  1. Better understand what’s happening on each device individually. If your test fails on a 3″ screen because the keyboard blocks the password field, you can know for sure with some nice logging. Just create a nice extension method called ToLogFormattedString for the Xamarin.UITest.AppResult class, and then use Console.WriteLine(app.Query().ToLogFormattedString()) in the test. This will dump out a pretty-printed control hierarchy for you to use. Since app.Query() returns only fully visible elements by default, you can use this technique to find out what elements can and cannot be “seen” by the automation framework (and by extension, your users) on each device.
  2. Generate your own diagnostics and metrics. If you’re trying to optimize tests, you can output execution timing to that log.
  3. Validate response data from service endpoints. Some E2E tests go all the way back to the database. An example use case would be creating a user: you’d use the UI to create a new user with a randomly generated, unique user ID created by the test code. You can automate the creation workflow, and at the end of the process, hit your REST API and try to pull a valid user with that same ID – you can then assert that the fields in the database match the ones entered by your test. Outputting JSON to the console log can help you understand what exactly is going over the wire.
  4. Sanity check your work. Because sometimes you need to make sure your code “got to there.”

The test log is just another way Xamarin Test Cloud gives you the information you need, when you need it, to get faster feedback and make better apps for the world to enjoy.

Add-in For Building Xamarin.UITest Page Objects Faster And More Accurately

This post is all about creating precise, stable automation suites as quickly as possible.

The explanation

In the last post (which was way too long ago), I left off with a couple of enhancements that I promised to cover next:

  • Remove the IApp dependency in pages
  • Knowing what page you’re on
  • Handling page transitions without extraneous waits
  • Generic loading of pages for cross-platform tests

I’m going to cover a simple evolution of what I described last time, but with the added bonus of a helpful add-in for Xamarin Studio. We won’t cover dependency injection or cross-platform just yet.

Much of my current role here at Xamarin involves the creation of custom Xamarin Test Cloud demos for prospective customers. I do a lot of custom demos, and my primary concern with the tests that I generate is not unlike what you’d want from your testers anyway. What I’m looking for in an approach is as follows:

  • The test is human-readable.
  • The test is easy to write and there is a reasonable pattern to follow.
  • The various methods in the test can be parameterized with different data.
  • Page objects can be composed together by someone with little coding experience and can be expected to “just work.”
  • All the difficult automation work is centralized and concerns are properly separated.

What I’m about to offer is certainly not perfect, and it’s not a prescription. It’s just what works for me every day in the context of Xamarin Test Cloud, and I’m happy to share it with you. You may want to extend it and add features, and that’s great!

Explaining The Approach

Our goal is to get to beautiful, fluent-looking tests that look like this:

[Test]
public void SuccessfullyPlaceAnOrder()
{
    app.LoginPage()
       .LogIn("testUser", "testPassword")
       .GoToHomeScreen()
       .ViewOrders()
       .SelectFirstOrder()
       .VerifyOrderDetails();
}

Notice how you can read this very clearly, and you can pass different arguments to methods that accept them. This type of approach works well in a single SDET type of environment, as well as where you have multiple people utilizing the page objects for one app to build out the test suite.

The key here is that all the page operations are centralized into that page’s class, so you never do the same thing in two different ways.

There are some more hidden thingies in here that I need to explain.

How the Fluent Syntax Works

Right off the bat, we have this weird line of code which certainly doesn’t exist in vanilla UITest: app.LoginPage()

How did our beloved app variable get knowledge about some login page? The secret is that we used an extension method.

The LoginPage() extension method actually instantiates and returns an instance of the LoginPage page object, also telling the new LoginPage to use the same app variable to perform all of its necessary operations. Let’s take a look at how this extension method works:

[Test]
public static LoginPage LoginPage (this IApp app)
{
    var page = new LoginPage (app);
    if (app.TraitExists (page.Trait))
        return page;
	
    throw new Exception ("The Login page was not found.");
}

You’re also probably going to notice this concept of a Trait. We’ll get into that later, but for now, just know that it is a strongly typed encapsulation of a query that every page has, and can be used for seamlessly transitioning between pages.

Once we have that LoginPage, we can start calling methods that exist on that object, for example, LogIn(string username, string password).

Pretty much all operations on the login page return the current instance of the login page, unless the operation results in a transition to another page. Compare the return types and return statements of these two methods:

public MainPage ValidLogIn (string username, string password)
{
	app.WaitForElement (usernameField);
	app.Screenshot ("On the Login page");

	app.EnterText (usernameField, username);
	app.Screenshot ("Enter Username");

	app.EnterText (passwordField, password);
	app.Screenshot ("Enter Password");

	app.Tap (loginButton);
	app.Screenshot ("Tap 'Log In'");

        app.WaitForNoElement("activity_indicator");

	return new MainPage(app); // transition to new page
}

public LoginPage InvalidLogIn (string username, string badPassword)
{
	app.WaitForElement (usernameField);
	app.Screenshot ("On the Login page");

	app.EnterText (usernameField, username);
	app.Screenshot ("Enter Username");

	app.EnterText (passwordField, badPassword);
	app.Screenshot ("Enter Bad Password");

	app.Tap (loginButton);
	app.Screenshot ("Tap 'Log In'");

        app.WaitForElement("error_message");

	return this; // stay on the Login page
}

This way, as you create page objects, you’re writing the implementation for each operation directly into that page object, and the app flows from page to page.

“Automatic” Waiting

This framework provides a default constructor template that will try to wait for the trait when the page object is created and then takes a screenshot. This supports the fluent style of calling methods shown above.

Let’s look at the constructor for a page object, so you can see how this pseudo-automatic wait protection works:

public LoginPage (IApp app)
    : base (app)
{
    Trait = new Trait (usernameField);
    app.WaitForTrait (Trait);

    app.Screenshot ("On the Login page");
}

The Trait class is constructed with the default type of a query in Xamarin.UITest, which is System.Func<AppQuery, AppQuery>, which will allow you to use any typical query from UITest as your page’s trait – such as e => e.Id("username").

Xamarin.UITest Page Objects Add-in

Behold the following Xamarin Studio add-in:

Xamarin.UITest Page Objects

This repo contains the basics for getting started with this approach, as well as an mpack file which you can install into Xamarin Studio by way of the Addin Manager.

Using the Add-in for Faster Testing

Once you’ve installed the add-in, you will have access to a bunch of helpful classes that support this programming model. This add-in will extend the default single-platform tests that come with Xamarin Studio quite nicely.

Follow these steps:

  1. Create a new platform-specific test project from the Xamarin Studio templates, for example, a new Android test project.
  2. Right-click your project and add a new file.
  3. Choose UITest Helpers on the left pane, and choose UITest Page Object with Setup. You will be prompted to enter a name. This template by default will already create a LoginPage for you, but you should specify another page to work on here. If you don’t want or need this extra page, just delete it.
  4. Type app.LoginPage() in your test, and see how you can extend it according to the approach above.

Next up, we’ll have a screencast on using these tools to your advantage to create high-quality UI automation tests faster.

Let me know if you have any issues with the add-in, or if you’d like to make some contributions to the project.