A/B testing on mobile is always the top asked question while developing new features. Most of the A/B testing tools like Apptimize, Taplytics,Optimizely, Mixpanel, are only capable of tracking variable or minor UI changes.
However when you want to measure paying user conversion between two different checkout processes or signup percentage for different login flows, you need to do this programmatically within the app.
If the user_id is even the app will show the first variant, if the user_id is odd the app will show the second variant and so on.
You can surely practice the above solution, but that’s not what I’m going to explain today.
For data-driven decision makers like you, Google Play Developer Console has something special called, Staged Rollouts.
Staged Rollouts will provide you an opportunity to publish two different versions of your app into production.
So instead of differentiating your users with some variables like user_id, and having one major apk, whenever you develop new features you can upload the newer apk to production, and you can choose the percentage of users who could be able to receive that update and increase it from 10% to 100% over time while fixing the bugs and measuring the impact of changes.
The most practical outcome of using Staged Rollouts is the ability to A/B test and measure all of those changes at the same time.
Even though you calculated your paying user conversion before the new features, when the new update comes and it is applied for all of the users, the previous version is gone.
Thus, you cannot compare the previous result with the new version with new features. By the time the new update comes, new users coming from your marketing campaigns might be better performing than the others. As a result of that, you might be mistaken evaluating the new changes.
After we have discovered Staged Rollouts at KapGel, we accelerated our app submissions from 6 weeks to 3 weeks. Plus, I am now plainly sure measuring whether our new feature affected our metrics positively or not.