Running Multiple Appium Tests in Parallel

UI-driven functional tests take time, and even when we've used all the tricks up our Appium sleeves, the tests will never be as fast as, say, unit tests. The ultimate solution to this is to run more than one test at a time. Think about it this way: if you have a build in which you are constantly adding tests (as you are likely to do while ensuring quality for an app), then each additional test will add directly to your build time, if you're running them one-by-one:

Let's say, on the other hand, that for each additional test you add to your build, you unlock an additional test execution thread. In this case, you're running every single test simultaneously, and the build is only as long as the longest test!

Check out: Using Mobile Execution Commands to Continuously Stream Device Logs with Appium

It's probably not realistic to assume that you could afford thousands of test execution threads, but it illustrates the point that often the most effective optimization you can make to reduce overall build time is to parallelize test execution. Also, with the availability of cloud-based test execution, adding additional test threads becomes a problem of cost rather than technical feasibility.

In another article, we'll discuss all the ins and outs of parallelization from a testsuite perspective. For now, let's just learn the mechanics of running multiple Appium sessions at a time, and leave the big-picture stuff for later.

Also check: Tips for Speeding Up Your Remote Appium Sessions

There are basically four designs for running multiple Appium sessions at a time:

1. Running multiple Appium servers, and sending one session to each server
2. Running one Appium server, and sending multiple sessions to it
3. Running one or more Appium servers behind the Selenium Grid Hub, and sending all sessions to the Grid Hub
4. Leveraging a cloud provider (which itself is running many Appium servers, most likely behind some single gateway)

(In this article we'll explore Options 1 and 2 above, leaving Selenium Grid and cloud-based Appium testing as topics for other articles.)

Parallel Testing - Client Side

Regardless of which option you choose for the Appium server side of the equation, there is also the client side to worry about. Just because you have multiple Appium servers or a cloud-based solution doesn't mean you have parallel testing---you also need a way for your test runner or test framework to kick off the tests in parallel! This ability to run tests in multiple threads (or using an event loop) is not part of Appium itself and must be configured in your test runner itself. Each language and test runner has its own way of accomplishing this task.

In the world of Java, the most straightforward way to do this is probably with Maven and the Surefire plugin, which lets you update your pom.xml with the following kind of configuration:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <parallel>methods</parallel>
    <threadCount>4</threadCount>
  </configuration>
</plugin>

Here you can see that we can either parallelize based on methods or some other grouping (classes, for example). We can also specify how many test threads we want---this should of course match the number of Appium sessions we can handle.

Leverage digital experience monitoring capabilities for proactive resolution of app issues. Know more.

Parallel Testing - Server Side

For most of Appium's history, Option 1 (running multiple servers) was the only way to achieve multiple simultaneous sessions. The Appium server was not set up for handling multiple simultaneous sessions, so the recommended approach was to start different Appium servers listening on different ports, for example:

appium -p 10000  # server 1
appium -p 10001  # server 2

Check: Facts to Note About Using Appium for Desktop Applications

Then, each test thread would need to adjust the URL of the Appium server to include the appropriate port, for example:

@Test
public void testOnServer1() throws MalformedURLException {
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability("platformName", "iOS");
    capabilities.setCapability("platformVersion", "11.4");
    capabilities.setCapability("deviceName", "iPhone 8");
    capabilities.setCapability("app", APP);

    IOSDriver driver = new IOSDriver<>(new URL("http://localhost:10000/wd/hub"), capabilities);
    actualTest(driver);
}

public void testOnServer2() throws MalformedURLException {
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability("platformName", "iOS");
    capabilities.setCapability("platformVersion", "11.4");
    capabilities.setCapability("deviceName", "iPhone 8");
    capabilities.setCapability("app", APP);

    IOSDriver driver = new IOSDriver<>(new URL("http://localhost:10001/wd/hub"), capabilities);
    actualTest(driver);
}

Obviously, this isn't ideal from a testsuite design perspective, since we have to have some way of specifying ports differently based on different threads, meaning creating different test classes or similar. Nowadays, Option 1 is thankfully not the only option: we can also simply start multiple sessions on a single running Appium server! This means we can forget starting multiple servers on different ports and rely on the one server running on the default port, or any port we want.

Read: All You Need to Know About Appium Mobile Testing

Unfortunately, this isn't everything we have to worry about. Appium orchestrates a host of services under the hood to facilitate communication between different subsystems and drivers. In some cases, an Appium driver might need to use a port or a socket, and it would not be good for another session using the same driver to compete for communication on the same port or socket. There are thus a handful of capabilities that need to be included to make sure no system resource conflicts exist between your different test sessions.

Android Parallel Testing Capabilities

• udid: if you don't include this capability, the driver will attempt to use the first device in the list returned by ADB. This could result in multiple sessions targeting the same device, which is not a desirable situation. Thus, it's essential to use the udid capability, even if you're using emulators for testing (in which case the emulator looks like emulator-55xx).
• systemPort: to communicate to the UiAutomator2 process, Appium utilizes an HTTP connection which opens up a port on the host system as well as on the device. The port on the host system must be reserved for a single session, which means that if you're running multiple sessions on the same host, you'll need to specify different ports here (for example 8200 for one test thread and 8201 for another).
• chromeDriverPort: in the same way, if you're doing webview or Chrome testing, Appium needs to run a unique instance of Chromedriver on a unique port. Use this capability to ensure no port clashes for these kinds of tests.

Also read: Managing Chromedriver for Android Chrome and Webview Testing

iOS Parallel Testing Capabilities

• udid: as with Android, we need to make sure we specify a particular device id to ensure we don't try to run a session on the same device. For simulators, udids can be found by running xcrun simctl list devices. Actually, if you're doing simulator testing, I should have said that either the udid capability is required, or a distinct combination of the deviceName and platformVersion capabilities is required. See below...
• deviceName and platformVersion: if you specify a unique combination of device name and platform version, Appium will be able to find a unique simulator that matches your requirement, and in this case, you don't need to specify the udid.
• wdaLocalPort: just as with UiAutomator2, the iOS XCUITest driver uses a specific port to communicate with WebDriverAgent running on the iOS device. It's good to make sure these are unique for each test thread.
• webkitDebugProxyPort: for webview and Safari testing on real devices, Appium runs a small service called ios-webkit-debug-proxy, which mediates connections on a specific port. Again, make sure multiple test threads are not trying to speak to this service on the same port.

Check out: Setting iOS App Permissions Automatically

Option 1 Redux

One advantage of Option 1 (running multiple Appium servers) that I didn't mention before is that you can use it to abstract away all this port configuration from your testsuite, by leveraging the --default-capabilities server flag. Essentially, you can define unique port values on server start (rather than in desired capabilities), for example:

appium -p 10000 --default-capabilities '{"systemPort": 8200, "udid": "emulator-5554"}'
appium -p 10001 --default-capabilities '{"systemPort": 8201, "udid": "emulator-5556"}'

(This gets us back to the problem of now having to specify different Appium server ports in our driver creation, but this problem could be avoided with the use of something like Selenium Grid)

Read: Starting an Appium Server Programmatically Using AppiumServiceBuilder

Putting It All Together

Once we've got our client set up to fire off multiple tests at the same time, an Appium server running which can handle multiple sessions (or we're running multiple servers), and appropriately unique desired capabilities for each test thread, we're good to go. With the Gradle setup I'm using for Appium Pro, the Maven method I discussed above is not available to me. The most straightforward approach is to parallelize based on test classes. First, I need to update my build.gradle to include the following directive:

test {
    maxParallelForks = 2
    forkEvery = 1
}

This tells Gradle that I want to run in at most 2 processes, forking a new process for every test class I encounter. What I need to do next is organize my testsuite so that different classes contain different sets of desired capabilities. Because I am multiplying test classes based on the number and type of capability sets, that means each test class should inherit a base class which contains the actual logic (we don't want to duplicate that!), so some organization like the following:

BaseTests
|--BaseIOSTests
|  |--IOSTests_Group1
|  |--ISOTests_Group2
|--BaseAndroidTests
   |--AndroidTests_Group1
   |--AndroidTests_Group2

In a real, well-architected build, you probably want to follow a completely different approach, one that doesn't involve hand-coding multiple classes just for the sake of getting parallel testing going, but for the sake of this article it's what we'll do. Here's an example of login test ready to be run in parallel on different iOS simulators:

Edition028_Parallel_Testing_Base.java

package Edition028_Parallel_Testing;

import io.appium.java_client.AppiumDriver;
import io.appium.java_client.MobileBy;
import org.openqa.selenium.By;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class Edition028_Parallel_Testing_Base {

    protected String AUTH_USER = "alice";
    protected String AUTH_PASS = "mypassword";

    protected By loginScreen = MobileBy.AccessibilityId("Login Screen");
    protected By loginBtn = MobileBy.AccessibilityId("loginBtn");
    protected By username = MobileBy.AccessibilityId("username");
    protected By password = MobileBy.AccessibilityId("password");


    public void actualTest(AppiumDriver driver) {
        WebDriverWait wait = new WebDriverWait(driver, 10);

        try {
            wait.until(ExpectedConditions.presenceOfElementLocated(loginScreen)).click();
            wait.until(ExpectedConditions.presenceOfElementLocated(username)).sendKeys(AUTH_USER);
            wait.until(ExpectedConditions.presenceOfElementLocated(password)).sendKeys(AUTH_PASS);
            wait.until(ExpectedConditions.presenceOfElementLocated(loginBtn)).click();
        } finally {
            driver.quit();
        }
    }
}

Edition028_Parallel_Testing_iOS.java

package Edition028_Parallel_Testing;

import io.appium.java_client.ios.IOSDriver;
import java.net.MalformedURLException;
import java.net.URL;
import org.junit.Test;
import org.openqa.selenium.remote.DesiredCapabilities;

public class Edition028_Parallel_Testing_iOS extends Edition028_Parallel_Testing_Base {

    protected String APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.5.0/TheApp-v1.5.0.app.zip";

    @Test
    public void testLogin_iOS() throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "iOS");
        capabilities.setCapability("platformVersion", "11.4");
        capabilities.setCapability("deviceName", "iPhone 8");
        capabilities.setCapability("wdaLocalPort", 8100);
        capabilities.setCapability("app", APP);

        IOSDriver driver = new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
        actualTest(driver);
    }
}

Edition028_Parallel_Testing_iOS_2.java

package Edition028_Parallel_Testing;

import io.appium.java_client.ios.IOSDriver;
import java.net.MalformedURLException;
import java.net.URL;
import org.junit.Test;
import org.openqa.selenium.remote.DesiredCapabilities;

public class Edition028_Parallel_Testing_iOS_2 extends Edition028_Parallel_Testing_Base {

    protected String APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.5.0/TheApp-v1.5.0.app.zip";

    @Test
    public void testLogin_iOS() throws MalformedURLException {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "iOS");
        capabilities.setCapability("platformVersion", "11.4");
        capabilities.setCapability("deviceName", "iPhone X");
        capabilities.setCapability("wdaLocalPort", 8101);
        capabilities.setCapability("app", APP);

        IOSDriver driver = new IOSDriver<>(new URL("http://localhost:4723/wd/hub"), capabilities);
        actualTest(driver);
    }
}

Notice that both of classes with actual @Test annotations differ merely by a few capabilities. This could obviously be tightened up to reduce boilerplate, using an anonymous inner class design, or similar. But it illustrates the point. Now, we can run these tests in parallel by isolating them using gradle:

gradle cleanTest test --tests "Edition028_Parallel_Testing_iO*"

If you run this, you'll see the Appium server handling multiple sessions at the same time, their log output interleaved. This is another reason to potentially choose multiple servers over a single server---the ability to direct log output to different files, or otherwise keep it sensibly organized for debug purposes!

And that's basically it. Parallel testing is a deep topic and there's lots more to say about it, so stay tuned for other articles in this genre. Don't forget to check out the full code sample for this edition, which also includes an Android test class. You can update the build.gradle file to have a maxParallelForks of 3, fire up an Android emulator, and run a cross-platform suite in parallel too!