Capturing Browser Errors and Logs in iOS Web/Hybrid Apps

We've talked a few times about automating web and hybrid apps with Appium. One feature of such apps is the ability to write information to the browser console. Whether or not this console is visible to the end user is irrelevant---app developers often announce interesting information about the state of an app in the form of browser console messages. It would be useful to be able to retrieve this information as part of our test, for two main reasons:

1. To make assertions on app state (for example if we're waiting for a 3rd-party JS library to load, and the only way we have of determining its state is by inspecting the console)
2. To save the record of an app's behavior during a test run for future inspection (for example to assist in debugging failed tests, or in reporting bugs caught by a test).

Check out: Saving Test Data to Make Debugging Easier

Something closely related to the second purpose is the need to capture JS errors; it's extremely useful to detect whether the web/hybrid app has thrown any errors during the course of a test! Using a simple little hack, it's actually quite easy to get any JS error logged to the browser console. We simply have to run this little bit of JS in the page:

window.onerror = console.error.bind(console);

This basically tells the browser to direct information about any error it detects on the page to the console.

Read: Fundamentals of Browser, OS, and Device Fragmentation

Of course, all of this is moot if there's no way to use Appium to retrieve this information. Happily, there is! Let's take a look at how to retrieve the browser logs for Safari (or any other iOS hybrid app).

We begin by using the log retrieval mechanism built into the Appium client (by way of the Selenium client included within), and specifying a specific, custom log type safariConsole:

driver.manage().logs().get("safariConsole");

In Java, this will return a LogEntries object which contains a set of (you guessed it) log entries: one per console message which has been received since the last time we retrieved the entries. Each entry has certain data associated with it: a message, a level and a timestamp. In the case of safariConsole logs, the message is itself a JSON string, which contains lots of information (the console message itself, a stack trace, and so on). This means that we'll typically need to convert the message from its JSON string format into an object we can work with in our code. Let's say we wanted to print out the numeric timestamp, the severity level, and the message of everything that's been logged to the console. We can do so using the following code:

for (LogEntry entry : driver.manage().logs().get("safariConsole")) {
    HashMap consoleEntry = new Json().toType(entry.getMessage(), Json.MAP_TYPE);
    System.out.println(String.format(
        "%s [%s] %s",
        entry.getTimestamp(),
        entry.getLevel(),
        consoleEntry.get("text")));
}

Essentially, what we're doing is going through each entry in the list of entries, turning the message into a key/value pair so we can extract specific data from it, and using some of the general entry API methods as well (to get the time and severity level of the message). Of course, this is just one example of how we could use the driver.manage().logs() feature. We could do all kinds of things with it. But when would we want to do something like above? It would sit very naturally in some kind of teardown method. Instead of writing to stdout, we could write the console messages to a file and save it along with other test artifacts in case we need it later on to figure out why a test failed.

Also read: Browser Sandboxing - What is it, and why do we need it?

We could also use the window.onerror hack I showed above, to make sure that any JS errors are brought to our attention; we could even be very strict with our developers and automatically fail a test if any SEVERE-level log entries come through! We don't even need our developers' permission to set up this little error handler; we can do it ourselves using executeScript at the beginning of our test:

driver.executeScript("window.onerror=console.error.bind(console);");

(2 caveats here with this simplistic hack: first, we'll need to run this little script after the page has loaded, which means we won't catch errors that happen on page load. Second, we'd overwrite any other handler attached to the onerror event, which might break the intended behavior of the app if such a handler was already in use).

Also check: Selenium Testing - Everything You Need to Know

At any rate, using the safariConsole log type, we're able to get all kinds of interesting information from the browser console, whether in Safari or even hybrid apps. Have a look at the full code sample to see how I've put everything together in the context of a real-world example:

import io.appium.java_client.ios.IOSDriver;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.json.Json;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.remote.DesiredCapabilities;

public class Edition037_iOS_Web_Console {
    private IOSDriver driver;

    @Before
    public void setUp() throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "iOS");
        capabilities.setCapability("platformVersion", "11.4");
        capabilities.setCapability("deviceName", "iPhone 8");
        capabilities.setCapability("browserName", "Safari");

        driver = new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
    }

    @After
    public void tearDown() {
        if (driver != null) {
            for (LogEntry entry : driver.manage().logs().get("safariConsole")) {
                HashMap consoleEntry = new Json().toType(entry.getMessage(), Json.MAP_TYPE);
                System.out.println(String.format(
                    "%s [%s] %s",
                    entry.getTimestamp(),
                    entry.getLevel(),
                    consoleEntry.get("text")));
            }
            driver.quit();
        }
    }

    @Test
    public void testLogging() {
        driver.get("https://appiumpro.com/test");
        driver.executeScript("window.onerror=console.error.bind(console)");
        driver.executeScript("console.log('foo.');");
        driver.executeScript("console.warn('bar?');");
        driver.findElementById("jsErrLink").click();
    }
}