The Future of Automation Testing for Media Devices

Listen to Jonathan Lipps, Appium architect and director of HeadSpin University, talk about the future of automation testing for media/OTT and other specialized devices. https://soundcloud.com/headspinio/jonathan-lipps

Synopsis

One of the biggest visions for Appium is that it can basically help provide automation support for any application platform.

If we call it an application, usually that means that there’s some kind of user interface to tap or click or type into. Selenium had created a good model for automation for testing web browsers as most things have user interfaces that people interact with. When we started work on Appium, we said, “Okay, well, why don’t we just extend that kind of insight to new platforms?” Eight years ago, the obvious new platforms were iOS and Android. Now there’s a whole other set of platforms out there such as all the TV, like Apple TV, FireTV, Roku, Chromecast, etc.

Remotely run cross browser tests and debug mobile, web, audio and video applications on thousands of devices around the world. Larn more.

Providing automation tools for application platforms really helps the platform and ultimately helps the end users by making sure that app developers have the tools they need to release quickly and to release software that actually works well. Whether you are the Chrome OS team or the Roku team, you are definitely incentivized to give your application developers, who are writing apps for your platform, the tools to help make sure that their applications are high quality. Some years ago Microsoft had approached me to make test automation for Windows applications available as part of Appium. Because they saw a lot of people using Appium, they wanted to make sure that everyone developing apps for the Windows operating system had an easy way to test those applications as part of their software development cycle.

Accelerate Appium test cycles with HeadSpin

Appium has a goal of building out the tools for the vendors that don’t necessarily have a real vision for providing good automation for their platform yet and making it available to the world. And then hopefully bringing the platform vendors along into the fold, to say “Yes, this is the way we want to do automation.”

A lot of the work that I’ve been doing this last year has been on figuring out how to make Appium really easily extensible so that anybody, even if they are not on the Appium core team, can think about creating an Appium driver for a new platform and share it with the world. Then everybody can use the same techniques they use to automate web and mobile applications to automate applications on any brand new or bespoke platform.

Listen to Brien Colwell, HeadSpin CTO, weigh in on HeadSpin’s vision for supporting test automation for our customers in the media, entertainment, and gaming space. https://soundcloud.com/headspinio/brien-colwell

Synopsis

There’s a really interesting historical note here too, which is, if you look at test frameworks from 10 years ago to test frameworks that are used today, there’s really one commonality and that’s Appium and WebDriver. We’ve seen this sort of long term vision of enabling a standard testing platform that’s easy to learn that works across a broad number of devices, to be the preference of the industry, and platforms that do this do see heavy adoption in terms of testers.

Jonathan, obviously, is the visionary who is looking five to ten years into the future and saying, look, if we can get more devices into this platform we will actually have something that has a lot of staying power in the industry. It’s also incredibly powerful from an industry standpoint that we can train people to reuse their skills. If you already have a large workforce of testers who already know how to test iOS, for instance, they can easily expand their skill set to Roku and KaiOS.

Now there’s also the short term consideration which is that not every device can reasonably support Appium. In the next one to five years there’s always going to be devices, especially legacy devices, that don’t really support Appium. For many of these devices the HeadSpin solution goes much deeper on the media side, testing more from what the user sees and hears, using more cameras, using more audio, especially with our AV box, to drive these test cases. To support these devices, we’re partnering with companies who specialize in writing support for this long tail of applications.

There are a few types of applications that are interesting for us. For example, gaming applications. Gaming is historically something that’s been very hard to test generically. In the gaming industry, there’s a direction where developers build test frameworks directly into their code. And so the QA team works very closely with the development team to build out these sort of automation frameworks, as part of the application itself. That’s something that right now there really is no vision for in the industry—it’s handled on a case by case basis. So we’re having the conversation with the development team on what are the best practices for their game. They might not be able to test everything with Appium. They might need to actually modify their application to allow it to be tested fully. And we’re seeing the same direction with media applications, that this is a partnership between the development team and the testing team.

So we’re looking at working with partners who build specialized testing hardware, with HeadSpin bringing in the user experience analytics, and ultimately the infrastructure and API’s to drive automation from the end user perspective. There’s a partner we’re working with where (very similar to games) they have an SDK that you embed in your application. You essentially issue commands through this SDK and it automates your application, so the development team actually has to write code as part of their test application to drive the automation.

Right now this is a short term thing, but I think there’s actually a long term vision where Appium can also facilitate this sort of embedded testing. Right now Appium very much has this philosophy that you do not need to modify your application. But as we’re seeing in games and media, it seems the norm right now in the industry is that you do need to modify your application to fully test your application.

In Jonathan’s ten year vision, these modifications can be included as part of a common fungible framework for Appium. If you have a QA engineer who’s specialized in gaming, and they’ve built up a very specific skill set that does not translate to, for example, media testing, HeadSpin’s vision is that even this can can be folded under a consistent framework that they can be trained on. And I believe that part of the five to ten year vision of Appium is getting the device vendors on board, and also identifying parts of the industry that do testing differently and start to address those.

HeadSpin University is our vehicle where we’re trying to train engineers to a standard so that they can become skilled at testing any type of application, whether it’s mobile browser, media, or gaming, giving them a set of testing skills that apply broadly across the industry. We’re looking at Appium as the vehicle that will enable this kind of standardization of skills.

To summarize, in the short term there are these exceptions right now, like gaming and media, and for these exceptions, we approach them with our audio visual technology, as well as with partners. So those are the two other points I wanted to bring into the discussion.

Overall, I think there is a great vision here. We’re seeing a long term vision and we’re seeing short term what doesn’t fit into that vision. And I think it’s interesting to take a survey of the industry right now.

Curious about the HeadSpin Digital Experience AI Platform?

Contact us to learn more.

FAQs

1. How should testers identify test cases that are appropriate for automation?

Identifying suitable test cases for automation is crucial to the effectiveness of automation testing. Some test cases that can be automated are:

• Tests that are performed for several builds
• Tests that result in human errors
• Tests that need numerous data sets.
• Tests with higher risks.
• Tests that cannot be performed manually
• Tests that are time-consuming and labor-intensive to run manually
• Tests that are to be performed on various hardware or software platform configurations.

2. What are the various types of testing framework techniques?

Automation testing frameworks are of four types:

• Data-Driven Testing framework
• Modular Testing framework
• Keyword-Driven Testing framework
• Hybrid Testing framework

3. What are the essential modules of automated testing framework?

The essential modules of an automation testing framework are:

• Test Assertion Tool
• Data Setup
• Build Management Tool
• Continuous integration tool
• Reporting tool
• Logging tool

4. What is the standard for scripting for performing automated testing?

Testers must consider the following scripting standards for performing automation testing:

• Uniform naming convention
• Three lines of comments for every ten lines of code
• Adequate indentation
• Robust recovery scenario and error-handling
• Usage of frameworks wherever feasible