Testing tvOS in Appium for Apple TV 1080 and 4K

HeadSpin offers a device cloud that can be used for testing purposes and monitoring your applications’ performance characteristics. Our device cloud currently has a wide range of real devices, such as every model of iPhone and iPad, Apple TV, and many different kinds of Android devices, including non-smartphone devices such as Fire TV.

HeadSpin supports the open source Appium framework natively. Appium is an industry standard framework for automated testing for mobile devices, as well as being a general framework for testing other kinds of systems, including web browsers.

Clients can use Appium written in a variety of languages such as Java, Ruby, or Python to interact with a Napkin server over the network with an http interface. All that is needed to run an Appium test on the HeadSpin cloud is the HeadSpin API token, which you can get from the HeadSpin UI in the URL for the Appium remote server. Appium supports iOS devices natively in all versions of Appium.

Currently HeadSpin provides versions – [backtrack]

With Appium, all you need is the HeadSpin API token and the remote driver URL. Appium supports iOS on iPhones and iPads from versions of 1.6 to 1.13, and HeadSpin currently supports these versions. In Appium 1.13 and above, there is support for tvOS running on Apple TV devices. Unfortunately, the open source Appium version can only support Apple TV 4th generation and below, but not the Apple TV 4K.

The reason for this is that the Apple TV 4K only has network connectivity rather than USB. And the open source tools currently only support USB. Luckily, HeadSpin has developed a solution that works for network devices such as the 4K for Appium and continues to work great.

If you look at the hardware of an Apple TV 4th generation, we can see that it has a USB-C connection for communication with the host, an HDMI output port for power and an Ethernet port for network connectivity. It also has WiFi support, so ethernet is optional. The 4th generation device can output a frame with a maximum resolution of 1920 x 1080

The Apple TV 4K is similar to the 4th generation but lacks a USB connection. It can output a frame size of 3840 x 2160 and can use either WiFi or Ethernet to communicate over the network. Running an Appium test on an iOS device requires that a set of configuration variables be passed to it. A minimal set of configuration variables are shown here. The configuration shows the udid of the device and name, which doesn’t have to be unique. The automation name that always must be an XCUI test, the platform of the mobile device, and an app or bundle ID. In this case, I’ve chosen the default calculator app that is installed on every iOS device.

To run on an Apple TV, we simply need to change the platform to tvOS – Calculator is not installed on an Apple TV. So, I will use the YouTube app instead. YouTube can be downloaded from the App Store on the Apple TV. Since Apple TV is only supported on Appium 1.13 and higher, it is also a good idea to specify the Appium version. If this is left out, HeadSpin will default to the highest version it has available, which may not be exactly the same version that you want to use. So, it is better to explicitly specify it with the HeadSpin Appium version capability.

Here we have a simple Python script that demonstrates how an Appium is set up. We can take this set of capabilities structured as a dictionary in Python, and pass them as an argument to the WebDriver object. We must also pass the remote URL that I had mentioned earlier, and I will show you how to get that URL in a moment.

In the HeadSpin UI, in the remote control tab, we see the list of devices we have access to. If I click on the three vertical dots on the right side, it will open up a dialog box to show me that Appium configuration.

In this configuration, we can get the remote driver URL that can be passed in the Python script, as well as the default set of capabilities that can be used for starting a test. Note that here, it does not specify the Appium version, so you may want to include that yourself.

If you look at the remote control device list, again, we can click the start button to gain manual control over the device. That will take us to this UI, where we see the device and some information about it on the right side. This screen shows it working for the Apple TV 4th generation and this next screen shows it working for Apple TV 4K.

When running an Appium test, the start button will change into a looking glass that lets you watch the test as it runs in real time. When you click the looking glass, you’re presented with a view of the device.

We can add an additional capability that will let HeadSpin capture the video as well as the Appium test runs. If we add “headspin:capture.video” as set it to true, then HeadSpin does all the work for us necessary for saving the video.

The resulting video can be viewed by going to the HeadSpin UI and clicking on the P button on the left. This will bring up the session list that shows all the recordings of devices done so far. Clicking on one of the sessions brings up the session UI. In this UI, we see that the video was saved for the session on the right, some analysis of the video on the bottom, and you download the video and Appium logs from the left side. Now I will demonstrate a running an Appium test on a 4K

Okay, so first I’m logging into the HeadSpin UI – let me get my terminal, so I can actually run the test. [Pause.] If you look at the devices that are attached, that I have access to, it tells me I have an Apple TV 4th generation, an Apple TV 4K, and two iPhones.

So when I run my test, immediately we see the looking glass. I can click on it and then it takes a couple of seconds for initialization to occur. But immediately we see the device screen. Here’s the HeadSpin loading program.

The first thing it does is go to the home screen of the Apple TV device. And now at this moment Appium is loading and it’s about to launch YouTube, and it will try to play a YouTube video for a couple of seconds. After this test has finished, it’s going to save the video and then we can look at it in the session UI.

So, our test has completed. Now we can go over to P, look at the test we just did. Here’s the video that just got saved. And there’s YouTube. And we can download the Appium log, the device log and the raw video file itself as an MP4. So, thank you for watching the demo and now Kazu will go over details about implementing an Appium test for Apple TV. Over to your Kazu.

Hi, I’m Kazu. I work as a Senior Software Engineer at HeadSpin. I also heavily contribute to the Appium project. Now, I’d like to demonstrate Appium commands for tvOS. As Jon explained before, Appium is supporting tvOS from Appium version 1.13. The only thing they should do is set platform name as tvOS. Other than that, all commands are the same for their Appium tests running on tvOS.

In this demo I will use ruby_console, which is a CRI tools. The library is written in Ruby, so it can also be used with Appium Ruby client.

The commands I’ll show you should it be easy to use in other clients too. Okay, let’s continue the demo. ruby_console supports “toml” format to set capabilities. Ruby_console will launch by arc toml and some configuration text.

When the command starts, the client sends their current session command to Appium server. After establishing their connections, we can send commands to the servers. Let’s share their capabilities. [Pause.] These are our capabilities I run from now, and the automation name is that XCUI test, and our platform name is tvOS.

And today, I will launch the TV stows provided by Apple, and this server URL is the WebDriver URL. There, we specified their URL, and then the client tries to connect to their own platform.

So let’s connect through to our platforms. [Pause.] After running the command, let’s see, there’s our platform page. And then, I can see the red icon on our device list. It means their device is locked. And let’s see, there are details. And, on this screen, we can see the progress and we can also see their Appium logs on this side. We can hide and show the logs. Let me hide the logs for now.

It looks like they already finished establishing the connection. Appium has a command to confirm the app state on the device. We could say app_state – we can send a command to tvOS to get their state. In order to ensure that TV shows up is actually running, “app_state TV shows” – then the running in foreground is returned.

This is our current test view, and actually the tvOS shows are running. If you want to launch the app, which is already in the device, you can launch it too. For example, let’s open the Settings app. “activate_app ‘com.apple.TVSettings’” – then, the Settings app launched.

Either the TV shows went to the background or it’s just not running. So, let’s see the app_state of the TV shows. The state says that apparently, there are TV shows running in the background and their Settings app is programmed.

Back to the TV shows. In the UI testing view hierarchy, it’s a very common method to write test cases, finding variable elements, attributes, etc. If the target view had standard UI elements, you can get the view – view tree – like this external formal.

The format is very complicated, but actually we can see there’s XML format that appears in their consuls. If the target app is more complicated or exceeds test framework, the response becomes slower since Appium adds various attributes in the source tree such as visibility testing or some other external attribute in the tree.

When we want to search for something in the app, we often type text in the search field. Appium supports sending a text for tvOS, with iOS. Let’s go to the search bar on this view. If you have experience with Appium, you probably wonder whether finding elements works on tvOS.

Yes, of course you can send, “findElement:” – then, let’s go to the search bar at the top, type “Search” – and the search word is here, under screen tree. If you view tree, you  can see that element in the search. So, let’s send a “findElement” – accessibility ID and search.

We can get the element. Now, let’s see the text attribute. And now, let’s go to the search.

The screensaver appears. So let’s activate our TV show again. On tvOS, we should have a physical case case for example, up, down, right, left to move the focus country. The focus is in top TV shows, so the focus isn’t in the search tab at the top. So first we must move their focus to the search, but Appium has a feature to move the focus automatically. If Appium can detect the focus element and the target element correctly, we already have our element on their search style. So, let’s just send a click.

I didn’t touch anything, and Appium has changed the focus of the search tab, and shows the search tab in the view. Now, let’s look at the source in here.

Around here, there is a search field. So from now, I want to type and send a text from the search field. Let’s find the search field in the source tree. This “XCUIElementTypeSearchField” may be the search field. So, let’s get the search field. But the element may not have a text or name attribute, so we cannot get this element using the accessibility ID. So, let’s get the element using the class name.

We can get the element, and the text, and try to send keys, for example “Spiderman” – then, their key is sent to their tvOS – Spiderman, the results are also shown. We can send a multiple byte string to their key.

“Clear” the text field and type in the text field “Spiderman” in Japanese. The search results don’t bring up anything because this is the US region.

Back on the home screen, I type the mobile command here. This command type provides Appium-specific, and device or Appium-driver-specific, clients. Most clients have execute_script command as the fundamental command name. It means they can use their mobile command without updating anything on the client side. This “pressButton” provides the feature of pressing physical buttons on their devices via exit test framework, so this “pressButton” home means pressing the home button on their remote control, which is provided for tvOS.

Now we’re on the home screen. When we want to get a current focused element to confirm where the focus is. You can call “getActiveElement” in Ruby context. “Switch to active element” is the method to get a current focused element. Currently, “Watch Now” is the focus on the screen, so let’s get the text. The text has “Watch Now” – so, the current focused element is “Watch Now.”

On tvOS, the active element means focused element – Appium transfers from getting active elements to getting focused tvOS contexts, so you don’t convert such different contexts between iOS and tvOS. Appium translates  them automatically.

Now, let’s open YouTube. What’s interesting on YouTube is we cannot get elements via source. Let’s type the source. We can only see the “source tree only” design. In this case we cannot use “findElement” and click “getElement” – for example, there are some screen icons, a home icon, a search icon. But in this case, Appium provides all of the remote control commands, so we can move the focused elements using the remote controller. As for the type of command we can do,  we can see the rest of the error message.

I just typed a long command named “pressButton” – the whole up down left right menu, so we can use them. Now, let’s play the YouTube video using this command. Currently, the focus is this very cute cat. And, when you try to send a lift, the focus moves to the home icon. And let’s send it down, for example. In that case, the power is down and the shows are on our screen. Back to the home icon – right is back to the cat view.

And, if we send our “select” command, it means just select the focused element.

On YouTube, we can see that there’s a progress bar, but tvOS has its command stop and play again. The command is named “play plause” – of course, Appium also supports that command. Once you send the command, the movie stops, and when you send it again, the movie starts again. Okay, let’s go back to the home and send an exit.

From ruby_console, I send our exit command. The command is closing the session, and after closing the session, on this view, the device view appears, and the devices go back to normal and the Appium session finished.

In this demo, I went through specific commands for tvOS. For example, find_element, click, pressButton, getActiveElement, send_keys, app_state, and page_source.

And some tips concerning test cases for tvOS. tvOS also has animation, just like iOS. So, you can use a wait method to make test cases stable. We have no official way to disable animations in the system level. And, please make sure your test target is running in the foreground before starting your test. The state will be background if the screensaver is presented. During this demo, the threshold became 1 – in that case, we must disable the screensaver using the home button or activate the app.

All right, thank you Kazu. In summary, HeadSpin provides support for running Appium tests on Apple TV and Apple TV 4K devices with minimal setup for the client. Be sure to use the mobile pressButton and Associated APIs for Appium commands to actually interact with the TV as it does not support swiping behavior that is standard on iOS devices. Lastly, even though you may not be able to test the real Apple TV 4K locally, the scripts you write for an Apple TV 4th generation will work on a 4K. Also, Appium scripts running on a simulated 4K will also work on a real 4K.

Thank you all for your time today. And if you have any questions, please send them this way.

Q&A

Q: How can I install my application?

A: HeadSpin offers an API that you can use remotely to upload an IPA to our platform, which will get installed automatically on a device of your choosing, including an Apple TV.

Q: How long does it take for the Appium session to start?

A: It can vary a little bit, but typically between 15 and 30 seconds.

‍