7 Reasons Why Testing Mobile Gaming Apps is Harder Than You Think

As exciting new games continue to fill App Stores at an unprecedented rate, staying ahead of the pack has never been more challenging. The pressure to innovate utilizing video, voice, touch screen gestures, augmented reality (AR), personalization, and cloud streaming technologies, has made game development more complex than ever. Combine this complexity with accelerated release schedules and frequent updates, each new release is a risky proposition, prone to bugs and slowdowns, and at the mercy of avid gamers who won’t hesitate to leave a scathing single star review.

Read more: How Can Integrated Game Testing Deliver Quality Experiences

Meanwhile, the mobile gaming industry is exploding. Last summer, Newzoo estimated there are 2.7 billion gamers in the world today—that’s fully a third of the world’s total population! — with just under half the global gaming market is comprised of mobile games.

With the stakes so high, ensuring flawless gaming experiences on day one is critical, yet many gaming companies struggle to release well-functioning apps on time and within budget.

Here are 7 reasons why testing mobile games is harder than you might think, and how HeadSpin can help.

1. Managing mobile devices can be a nightmare

Many gaming companies find themselves in the predicament of having invested in a large variety of test devices that they then struggle to keep track of and make globally accessible to all game developers and testers across the organization. These devices are often located in different physical offices, having been purchased by different teams, and the iPhone 12 that the developer in Singapore needs to test her app, may be sitting unused in someone’s desk in New York with no means for her to track it down.

Also check: Game Testing Challenges and Essential Metrics

What’s needed is the ability to manage these devices efficiently without having to physically ship devices around the world every time a remote team needs it for running tests.

HeadSpin provides an easy way to organize and utilize your existing device investment and supplement it with devices from our global device cloud.

• Our state-of-the-art device appliance can be used to securely host all devices in a particular location.

• Our scheduling and audit system allows you to see who is connected to a device and when—and easily track down whoever last had access to it and might have unplugged it!
• Devices can easily be accessed and controlled remotely for both manual and automated testing via our high-performance proprietary bridge for both iOS and Android, so you never have to physically ship devices to the locations that need them. Any type of test or debugging process can be run as if it was locally connected by USB.
• Our device cloud offers zero day premium support for new Apple and Android devices and OSes so your testing needs are always covered with access to the latest technology.

Read: Why is Usability Crucial for a Good Gaming Experience?

2. Mobile games often require special frameworks for functional testing

Many elements of game design and behavior often need to be tested manually. Automation is not always an option and when it is, it requires considerable effort and specialized test automation tools (such as AltUnity for Unity-based games) to create the test scripts. This is because game test use cases are typically not as simple as searching for a static element on a screen—you’ll usually need to embed an agent into the test script to identify objects and test if certain actions are triggered within a game. Unfortunately, gaming engines and tools are often so specialized that they are not supported by standard test automation frameworks such as Appium, or most test infrastructure vendors.

Fortunately, HeadSpin can connect to any test framework with our proprietary bridge technology, so you can easily utilize any existing tools and scripts without making any modifications, and seamlessly integrate them into your test automation workflow.

3. The competition is fierce and every minute counts

To compete effectively in the $70 billion mobile gaming market, time to market is a crucial advantage. Every additional step in the test process contributes to longer release cycles and the risk of losing critical mindshare to the competition.

A common speed bump game developers and testers often run into is the number of steps involved in uploading and executing tests. Many test automation vendors require tests to be uploaded to a server, and converted to a vendor-proprietary framework, resulting in delays due to errors and slow uploads.

HeadSpin natively supports any test framework via WebDriver and the versatile HeadSpin bridge technology. This allows you to connect directly to the test device without having to upload the test to a server, and execute the test directly on your machine on your preferred platform (e.g. Jenkins). Scripts can quickly and easily be set up to execute in parallel, and developer-friendly APIs allow for seamless integration with CI/CD workflows. Test results appear directly on your machine and can be sent to any tool of your choice.

4. Game developers are often located outside of the US

Game developers and testers often struggle with latency issues when the devices they need for testing are located in other countries or continents. When time is of the essence, it can hamper the team’s ability to get a new game or release out the door if they are held up waiting for a device to connect or a test to execute.

Commodity device farms are more often than not located in the US and developers in other parts of the world find themselves constrained by the resulting limitations. HeadSpin’s global device infrastructure hosts devices in 50+ global locations so you can almost always find low latency access to local devices close to home.

Also read: Critical KPIs that affect user experience in the mobile games

5. Game performance is more critical than for any other kind of app

Performance is crucial for gameplay.

Imagine if a game freezes in the middle of a gripping fight or challenge, or a racing game doesn’t respond immediately to your turn gestures? Game developers are painfully aware that if their game doesn’t perform, they will lose gamers much faster than a typical e-commerce vendor would lose a customer if a shopping cart checkout process got stuck on a spinner wheel. One can imagine that even an impatient shopper may return to try again later. But a gamer once burned during a climactic moment is unlikely to be so forgiving. Not only will she be unwilling to give the game another chance, she will probably share her disappointment all over social media and on App Store reviews and ratings.

Game performance needs to be comprehensively tested prior to release and continuously monitored while live. Not only is automation necessary, but also the deep visibility and smarts to detect issues and pinpoint their point of origin. Is the game jittery, doesn’t load, take too long to perform an action? What would a real gamer’s level of satisfaction be with the gaming experience?

HeadSpin provides the means to do this thoroughly and efficiently, empowering game developers and testers to deliver high-performing games on schedule. With HeadSpin you can:

• Assess the true gaming experience with our reference-free Video MOS and audio match analysis. HeadSpin’s MOS score is based on deep machine learning models that can be taught to accurately identify true gameplay malfunctions without relying on either a reference video (you won’t have one for a game) or solely on metrics such as jerkiness and blockiness that can easily produce false positives—especially in a gaming scenario.
• Find root causes of FPS drops, locate where the drop occurred, and quickly determine how to address the problem.
• Track third-party game engines. Show how programs such as Unity, Unreal and Cocos2d impact app performance and user experience.
• Determine root causes of issues whether they originate on the back end, device, OS, or network level. For example, detect whether the responsiveness of the game was impacted because the device dropped from a 5G to 4G network, because of a slow server, or because of battery drain on the device.
• Audit code-level performance and identify regressions between releases.

6. Gaming companies have a global customer base

Many location-specific factors impact game performance, from local network conditions to edge effects, such as connections to CDNs or third party APIs, and the proximity of cloud availability zones.

HeadSpin can help you identify and resolve these issues early in the development lifecycle, and make sure the game performs well anywhere in the world by testing on real devices located in real locations on real networks. HeadSpin’s extensive global device infrastructure is especially relevant to gaming companies that are focused on Asian markets, such as India and China, as most commodity device farm vendors do not offer a full range of devices in those locations.

7. Seamless payments are essential for gameplay

To support freemium pricing strategies, many mobile game vendors are shifting from advertising towards in-app purchases as their primary source of revenue. In this model, the gamer gets the basic game for free but can buy more lives, characters, stickers, additional game levels, a premium suspension or a speed booster in a racing game, or ammunition or armor in a shooter game, and so on. This requires many gameplay scenarios to have embedded processes for interfacing with external payment engines or gateways.

See: How to Launch High Performing Mobile Gaming Apps with Confidence

As such, it is essential to the gaming experience for payments to work smoothly. Local and edge effects can contribute to performance and other glitches, and it is important to thoroughly test and monitor these processes, on real devices in global locations, to ensure a happy gaming experience for your existing fans and potential new ones.

Conclusion

Ensuring flawless mobile gaming experiences across a global user base is paramount for mobile gaming companies in order to avoid poor user reviews and maximize revenue streams. However, numerous factors—ranging from device resources (memory, battery, CPU) to network latency, edge/CDN, API calls, 3rd party SDKs, and compatibility across the diversity of carriers, OSes and devices—make it difficult to reliably deliver a consistent experience. In addition, limitations with regards to the availability and accessibility of test devices and a global testing infrastructure can make it difficult to perform the thorough and efficient testing necessary to launch new releases with confidence on day one.

Learn more about how HeadSpin can help.

FAQs

1. How is gaming AV testing different from traditional AV testing?

A game’s Audio Visual (AV) performance needs to be flawless since the gaming performance relies heavily on audio and visual output. For example, a few frozen frames in a conferencing app may not be a big issue, but is experience-breaking in games. Game testing, therefore, needs a robust framework to identify AV performance issues early in game development.

2. What external factors can affect a game’s performance?

Users play games on a variety of platforms and devices. For example, a mobile game may be played on iOS and Android on thousands of devices and locations. Some common performance factors are

• Device configuration
• RAM usage
• Graphics card
• Network bandwidth and latency

It is important to test on as many different devices and configurations as possible to provide a smooth gaming experience to users.

3. What are some common video performance issues in gaming?

Gaming user experience relies heavily on the video experience of the user. Some common video issues to take into account are:

• Low frame rates
• Screen freeze
• Blockiness
• Blurriness
• Contrast
• Color Saturation

HeadSpin provides reference-free MOS scores to accurately predict the perceived video quality of games.