Pre-launch testing for mobile games: tools and best practices on Google Play
A guide to Google Play’s pre-launch tools
The months leading up to a game’s release are critically important to make it a lasting success. Due to the potential costs of launching and running a game, developers are increasingly running more layers of testing before release to ensure any problems are fixed before the public is allowed access.
At Google Play we’ve developed several ways to help game developers during this pre-launch phase, and this article will provide an overview of tools and best practices you can use. Testing options move in a scale from highly restricted (for testing basic functions and obvious bugs before anyone not closely affiliated gets visibility of the game) to largely open (when the game is near release but specific elements of the economy or design might need adjusting). This is shown on the following table, with more analysis of each approach below.
Internal testing
Whether you’re running your game by a few friends and family or giving access to key internal staff, Google Play allows you to roll out a private APK to up to 100 users in seconds via internal testing. This function is designed around what studios actually need, making it easy to send new APKs each day to QA teams on a wide range of devices. With Internal Testing you’ll also get full access to Google Play features such as the Android App Bundle support. To learn more about internal testing check out this best practices guide!
Closed beta
Once you’ve gone as far as you can with your internal teams, you might want to test your game on a small fraction of your audience prior to the full release. Google Play gives you the ability to invite up to 100k players via email or an unlimited number if invited via Google Groups. It’s simple for players, as they’ll receive a link to download the latest APK, but won’t be able to leave public reviews on Google Play. Private feedback can still be sent by players to the developer.
This is an invaluable tool for stress-testing your game, since using many public testers can help uncover bugs or usage patterns your QA team missed or simply did not anticipate. While still more likely to miss issues than a full open beta, a closed beta period is a balance between privacy and transparency — and can also be useful post-launch to trial new updates or DLC!
Open beta
When to start accepting public reviews is a turning point in the pre-launch strategy, but there is one more avenue available before you allow your players to begin publicly rating. An open beta. This allows users to search for your app on the store (listed as ‘Unreleased’ or ‘Early Access’) without giving them the ability to leave a public review — though they can give private feedback. During an open beta, developers can still set limits on the number of users who can access the game, but users no longer need to be invited or whitelisted to take part if the limit has not been reached.
Since your users in open beta are ‘wild’ (not specifically invited) this is a close approximation of how players will interact with your game once live. This makes it perfect for testing key KPIs, the popularity of different game loops, the flow of the economy and load testing. Aside from the technical benefits, it also allows you to test new markets, build up a community pre-launch and test server loads with a larger number of people — without the risk of an early build resulting in lots of 1-star reviews.
Next Games makes use of both open and closed betas in their development cycle. As Joonas Virtanen, Product Design Director says “much of the advantage comes from the fact that beta audiences have opted into a game that is known to still be in development. Their feedback is more forgiving and constructive and expectation management is less of a factor — these users also feel happy and privileged to see something before the full release.”
For Next Games, the advantage lies in learning how their product will survive in the wild. James Corbishley, Director of Business Intelligence, together with his team, developed “models to predict long-term retention based on very early D0 (day zero) metrics,” adding that “the Google Play beta platform serves as a perfect tool both for evaluating improvements after every update, as well as producing increasingly accurate retention predictions ahead of “opening” the game to the public at large.” This has been particularly useful in developing their upcoming title ‘Stranger Things’. “Even if results do not mimic the real world identically, the key is in analyzing the delta and understanding how to bootstrap that into real world expectations,” says James.
Soft launch
During a soft Launch, all players within an entire geographic market are able to search for and access the game and leave public reviews, making it only suitable for the latest stages of testing. This is also when the game begins to appear in the algorithmic collections on the Play Store (such as ‘New Releases’, ‘Recommended for you’) rather than only being searchable — this allows you to test the organic virality of your game.
At this stage, many developers get their user acquisition efforts up and running. It’s also a great opportunity to dive into monetization testing and make any final adjustments to the game’s economy. As you move through the soft launch and your confidence grows, you can continue to release in more regions. Some studios take the risk of soft launching into a tier 1 market. This can offer benefits such as providing insights into how the game performs specifically within that market, but if the soft launch is unpopular, you may struggle to overcome this early negative perception when it comes to the full release in that market. Google Play provides the tools you need to succeed whichever route you take.
To understand how to leverage a soft launch as much as possible, and quickly learn if a game can succeed or not in the market, we interviewed Teemu Hämäläinen, (Product Lead) and Burcum Taskaya (Product Marketing Manager) from Rovio, who explained their pre-launch strategy for their latest Match 3 title, Small Town Murders.
Small Town Murders is an exciting puzzle game where the main character, Nora Mistry, tries to unravel the truth about a series of bizarre crimes happening in a country town. Since Rovio has extensive experience launching puzzle games, the team was very confident with the stability of the game and strength of the core loops. After a short series of internal tests (using Play’s internal test tracks) they moved directly to soft launching the game in the US, Sweden, Finland and Poland in October 2019.
Soft launching in the US allowed Rovio to gather direct feedback from players in their largest market on the topics they felt needed testing: the strength of the IP, key KPIs and the marketability of the game (CPIs vs LTV) from day 1.
However, this strategy had its drawbacks. Launching in production in the US meant they were not able to run a Pre-registration campaign on Google Play — though Rovio still turned on Pre-registration in other regions. Changes to the core gameplay flow during soft launch also caused some backlash among players who felt their progress had been affected. Thankfully, intervention by Rovio’s customer support and game teams managed to reverse the situation and delight players with new content and game modes.
As Teemu says, “We learned a lot from this experience. We managed to leverage our existing knowledge in building successful puzzle games for the launch of Small Town Murders. But we would not have had the same pre-launch approach with a game in a genre that we hadn’t mastered yet.”
Across their various launches, Rovio has moved away from the concept of a “global launch.” Each title has different opportunities and weaknesses in each region, and therefore benefits from it’s own pre-launch strategy. Instead of working towards a single launch day, Rovio prefers to release content across various tracks in waves. In their own words:
“For us, the definition of global launch has changed drastically over time. We approach new games success as a marathon, not as a sprint, by focussing on continuously updating the game with new exciting content for players,” said Teemu.
Pre-registration
The only example on this list not focused on testing is the Pre-registration phase, and unlike what we’ve covered in this article so far, it is not focused on testing. This period is intended to build interest in the game before launch to improve engagement and retention metrics once live. Developers who run a Pre-registration campaign often see a large increase in day one and day seven retention.
This phase offers an insight into the early demand for the game as you start collecting pre-registrations and are able to observe the community’s reaction. Users are brought in organically or via an Ads campaign and can be offered Pre-registration rewards such as in-game currency or unique items for registering their interest. Those that register will receive a push notification when the game becomes available in their locale, boosting your first batch of installs. Google Play even offers players the ability to have the game automatically installed upon launch, which is a great way to increase engagement from day 1.
Pre-registration is available up to three months before launch, but we recommend to aim for a one or two month campaign to ensure interest does not fade. Studios generally see a higher conversion rate from pre-registered users, particularly if they can leverage brand awareness of a pre-existing IP to drive interest. While this is a fantastic result, it does make it critical to ensure the game is fully functional on day 1, as you may achieve far more installs than you might otherwise expect. We advise studios to minimize the game’s download size, make sure the game is stable and to test across a wide range of devices, as these are all common causes of bad review scores.
“Next Games found particular success using a Pre-registration program for games with a strong IP, such as Stranger Things or The Walking Dead”, comments Joonas Virtanen. “Since these shows have massive fan followings, news about an upcoming game won’t stay secret for long, and Pre-registration is a good way for us to capitalize on the excitement before we officially launch.”
To learn how to use Pre-registration for your own titles, check out our Pre-registration support page.
Roundup:
Google Play offers a range of ways to gather real data on how your game will perform, without risking early exposure to unpolished builds. The benefits can be carried beyond launch as well, if used to support a successful run of downloadable content or updates that need some final checks before release.
One last piece of advice comes from Next Games’ Joonas and James: “The key is in merging early intelligence and early game design: test often and test early. Even if the test results are bad, you’ll learn something new which will help with the next test or with another product.” As you take these learnings into your studio, make sure to also consider how and with which players you want to test, and take advantage of the tools offered by Google Play.
What do you think?
Do you have thoughts on pre-launch testing for mobile games? Let us know in the comments below or tweet using #AskPlayDev and we’ll reply from @GooglePlayDev, where we regularly share news and tips on how to be successful on Google Play.
