Streaming Video from iOS Devices

Many testers may be unaware of a recent improvement to recording a video of the device screen on iOS devices during a test run. The commands have not changed: in order to begin recording the screen, call the startScreenRecording command. There are multiple options available for specifying frame rate and such, and when you stop recording using the stopScreenRecording command, it will return the video file or upload the file to a url you provided.

The surprising new functionality is that once you've started the recording, you can access a live streaming video of the screen. All it takes is pointing your web browser to localhost:9100 and your browser will render this stream. You could use a video-recording tool to record from this stream if you wanted.

‍

‍

After this mjpeg stream functionality was released, Jonathan was doing some work around image-based testing of games and wanted a way to speed up Appium's ability to take a screenshot. With image-based testing, Appium needs to take a screenshot every time it looks for an element. If taking a screenshot could be made faster, than the tests speed up dramatically.

The thought is, if you are already recording a video of your test run, taking a screenshot should be unnecessary. Instead, Appium could use the latest frame of your video recording. The way this was implemented was by adding a new desired capability called mjpegScreenshotUrl. Setting this desired capability to the url of an mjpeg video stream (provided by any means), Appium will follow the stream and return the latest frame instead of trying to capture a screenshot normally.

On Android, this should provide a significant performance improvement, but setting up the mjpeg stream for an Android test run is not so simple, and will be the topic of a future article.

On iOS, we already have this mjpeg stream provided by default whenever we start screen recording! We can pass in localhost:9100 as the value for the mjpegScreenshotUrl desired capability, and Appium will retrieve screenshots from the stream instead of running a screenshot command on the device.

This performance upgrade was designed for Android devices which sometimes return screenshots very slowly. iOS doesn't have this problem and counterintuitively, this performance "improvement" actually caused my tests to run slower! Interesting to learn about, but I'd advise not using mjpegScreenshotUrl on iOS. I expect great improvements to Android screenshot times though, so look forward to that in an upcoming post.

Here is the code I used to test both methods of saving screenshots:

@Test
public void timeScreenshotsWithDefaultBehavior() throws IOException, URISyntaxException, InterruptedException, ExecutionException, TimeoutException {

   DesiredCapabilities caps = new DesiredCapabilities();
   caps.setCapability("platformName", "iOS");
   caps.setCapability("platformVersion", "12.4");
   caps.setCapability("deviceName", "iPhone Xs");
   caps.setCapability("automationName", "XCUITest");
   caps.setCapability("app", IOS_APP);

   driver = new IOSDriver(new URL("http://0.0.0.0:4723/wd/hub"), caps);

   driver.startRecordingScreen(); // this is unnecessary for this test run, but included here to make this test identical to the next test

   long startTime = System.nanoTime();
   for (int i = 0; i < 100; i++) {
       driver.getScreenshotAs(OutputType.FILE);
   }
   long endTime = System.nanoTime();

   long msElapsed = (endTime - startTime) / 1000000;
   System.out.println("100 screenshots normally: " + msElapsed + "ms. On average " + msElapsed/100 + "ms per screenshot");
   // about 172ms per screenshot on my machine
}

@Test
public void timeScreenshotsWithMjpegScreenshotBehavior() throws IOException, URISyntaxException, InterruptedException, ExecutionException, TimeoutException {

   DesiredCapabilities caps = new DesiredCapabilities();
   caps.setCapability("platformName", "iOS");
   caps.setCapability("platformVersion", "12.4");
   caps.setCapability("deviceName", "iPhone Xs");
   caps.setCapability("automationName", "XCUITest");
   caps.setCapability("app", IOS_APP);

   driver = new IOSDriver(new URL("http://0.0.0.0:4723/wd/hub"), caps);

   driver.startRecordingScreen();

   long startTime = System.nanoTime();
   for (int i = 0; i < 100; i++) {
       driver.getScreenshotAs(OutputType.FILE);
   }
   long endTime = System.nanoTime();

   long msElapsed = (endTime - startTime) / 1000000;
   System.out.println("100 screenshots using mjpeg: " + msElapsed + "ms. On average " + msElapsed/100 + "ms per screenshot");
   // about 436ms per screenshot on my machine
}

The entire test file can be found in our example code repository.