Optimizing WebDriverAgent Startup Performance

Some Appium users have asked me how to speed up their iOS tests, citing the length of time it takes to start tests which use the WebDriverAgent library (all tests using the XCUITest driver).

Most of the perceived speed of an Appium test can't be improved due to the base speed of booting simulators or the UI actions themselves. The slowest part, which users were asking me how to avoid, is the initial startup of a test: the time between sending the first POST /session command and the response indicating that your test script can begin sending commands. We'll call this time period the "session creation" time.

Let's start with the most basic iOS test:

private String APP = "https://github.com/cloudgrey-io/the-app/releases/download/v1.9.0/TheApp-v1.9.0.app.zip";

@Before
public void setUp() throws IOException {
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("platformName", "iOS");
    caps.setCapability("platformVersion", "12.2");
    caps.setCapability("deviceName", "iPhone Xs");
    caps.setCapability("automationName", "XCUITest");

    caps.setCapability("app", APP);

    driver = new IOSDriver(new URL("http://localhost:4723/wd/hub"), caps);
}

@After
public void tearDown() {
    try {
        driver.quit();
    } catch (Exception ign) {}
}

@Test
public void testA() {
    assertEquals(1,1);
}

@Test
public void testB() {
    assertEquals(1,1);
}

@Test
public void testC() {
    assertEquals(1,1);
}

@Test
public void testD() {
    assertEquals(1,1);
}

@Test
public void testE() {
    assertEquals(1,1);
}

@Test
public void testF() {
    assertEquals(1,1);
}

This is basically the default template we use for an iOS test in the AppiumPro sample code repository. We have our sample app hosted on a web Url for convenience, a @Before step which creates a fresh session for each test, an @After step to delete the session at the end of each test, followed by six tests which do nothing.

Check: Tips for Speeding Up Your Remote Appium Sessions

Each of the six tests take 12.8 seconds on average. We can cut this down by two thirds!

There are desired capabilities we can specify to greatly reduce the time it takes to create a session. Appium is built to cater to a large number of devices, for use in many different situations, but we also want to make sure that it is easy to get started automating your first test. When specifying desired capabilities, Appium will analyze the state of your system and choose default values for every desired capability which you don't specify. By being more specific, we can have Appium skip the work it does to choose the default values.

Also check: Why is Cloud Automation Testing gaining popularity?

Our first improvement is to set the app location to a file already on the host device. Downloading an app from a remote source is adding 3.9 seconds to each test in the suite.

private String APP = "/Users/jonahss/Workspace/TheApp-v1.9.0.app.zip";

Your test suite probably already does things this way, but for our demo repository this really speeds things up.

See: Facts to Note About Using Appium for Desktop Applications

Running the tests, it's easy to notice that the app gets reinstalled on the simulator for each test. This takes a lot of time and can be skipped. You may have certain tests which require a fresh install, or need all the app data cleaned, but those tests could be put into a separate suite, leaving the majority of tests to run faster by reusing the same app. Most users should be familiar with the noReset desired capability.

caps.setCapability("noReset", true);

This saves an additional 2.9 seconds per test.

Also see: Tips for Speeding Up Your Remote Appium Sessions

That was the easy stuff, giving us an average startup time of 6 seconds per test, but we can shave off another 2.1 seconds, which is a 35% improvement.

Appium uses the simctl commandline tool provided by Apple to match the deviceName desired capability to the udid of the simulator. We can skip this step by specifying the simulator udid ourselves. I looked through the logs of the previously run test and took the udid from there:

caps.setCapability("udid", "009D8025-28AB-4A1B-A7C8-85A9F6FDBE95");

This saves 0.4 seconds per test.

Read: Batching Appium Commands Using Execute Driver Script to Speed Up Tests

Appium also gets the bundle ID for your app by parsing the app's plist. We can supply the known bundle ID, again taking it from the logs of the last successful test. This ends up saving us another 0.1 seconds:

caps.setCapability("bundleId", "io.cloudgrey.the-app");

When loading the WedDriverAgent server, Appium loads the files from wherever XCode saved it after compilation. This location is called the "Derived Data Directory" and Appium executes an xcodebuild command in order to get the location. Again, we can look through the logs of the last test run and supply this value ourselves, allowing Appium to skip the work of calculating it:

caps.setCapability("derivedDataPath", "/Users/jonahss/Library/Developer/Xcode/DerivedData/WebDriverAgent-apridxpigtzdjdecthgzpygcmdkp");

This saves a whopping 1.4 seconds. While xcodebuild is useful, it hasn't been optimized to supply this bit of information.

Also read: Appium Tools for Rapid Development of Functional & Performance Tests

The last optimization is to specify the webDriverAgentUrl desired capability. If specified, Appium skips a step where it checks to make sure that there are no obsolete or abandoned WebDriverAgent processes still running. The WebDriverAgent server needs to already be running at this location, so we can only use this desired capability after the first test starts the server.

caps.setCapability("webDriverAgentUrl", "http://localhost:8100");

So, what have we done? Using a local file and noReset reduced the base test time from 12.8 seconds to 6 seconds, making test startup 53.3% faster.

On top of that, we performed some more obscure optimizations to improve test times another 23.9%, shaving off another 1.3 seconds.

Here's a visual breakdown of what Appium spends its time doing during one of our tests:

"Remaining test time" and "session shutdown" are all that remain in our optimized test:

private String APP = "/Users/jonahss/Workspace/TheApp-v1.9.0.app.zip";

private IOSDriver driver;
private static Boolean firstTest = true;

@Before
public void setUp() throws IOException {
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("platformName", "iOS");
    caps.setCapability("platformVersion", "12.2");
    caps.setCapability("deviceName", "iPhone Xs");
    caps.setCapability("automationName", "XCUITest");

    caps.setCapability("app", APP);

    caps.setCapability("noReset", true);
    caps.setCapability("udid", "009D8025-28AB-4A1B-A7C8-85A9F6FDBE95");
    caps.setCapability("bundleId", "io.cloudgrey.the-app");
    caps.setCapability("derivedDataPath", "/Users/jonahss/Library/Developer/Xcode/DerivedData/WebDriverAgent-apridxpigtzdjdecthgzpygcmdkp");
    if (!firstTest) {
        caps.setCapability("webDriverAgentUrl", "http://localhost:8100");
    }

    driver = new IOSDriver(new URL("http://localhost:4723/wd/hub"), caps);
}

@After
public void tearDown() {
    firstTest = false;
    try {
        driver.quit();
    } catch (Exception ign) {}
}

@Test
public void testA() {
    assertEquals(1,1);
}

//... 5 more tests

For all these tests, I kept the same simulator running. Starting up a simulator for the first test takes some extra time. I also noticed that the first test of every suite always takes a little bit longer even when the simulator is already running. For the timing statistics in this article, I omitted the first test from each run. None of them were that much longer than the average test, and the goal is to reduce total test suite time. If you have 100 tests, an extra second on the first one doesn't impact the total suite time much.

Don’t Rely on iOS Emulators & Android Simulators. Test on Real Devices

That said, all this is splitting hairs, since as soon as your tests contain more than a few commands, the amount of time spent on startup is insignificant compared to the amount of time spent finding elements, waiting for animations, and loading assets in your app.

Full example code located here.

For situations where a CI server is running Appium tests for the first time, further optimizations can be made to reduce the time spent compiling WebDriverAgent for the first test. We'll discuss these in a future edition!