A/B Testing: Some Things Need To Change

A prime example of this is the Google Play Store. In the last few weeks they’ve been messing with everything from search results to the shade of green. For some it has made using the Play Store an inconsistent experience.

Firstly, the changes aren’t just account specific but device specific too. I have the new search UI on my Pixel XL and Nexus 4, but not on my Pixel C or Nexus 5. Ideally an app should behave the same way on each device you use it on. Making it purely account based would go a long way to improve that.

Secondly, not all the changes happen at the same time. When Google started trying that fancy new shade of green, I got it on my Pixel XL. Instead of having the new colour everywhere, it was only in some places. Sometimes the action bar was the new colour but the buttons were not, sometimes it was the other way around.

You may be thinking; “that’s no big deal! Who cares what shade of green it is? Or if the search UI is the same?” That’s a valid argument. But look at things this way. Part of the very definition of a smartphone is the ability to download apps. The store is almost the focal point of everything. Preinstalled apps aside, everything you use on your phone comes from that one app. Having a consistent interface is important. You shouldn’t have to do things differently on one device when compared to the other when they’re on the same platform.

Thirdly, these changes are deployed to anyone regardless of experience with Android or smartphones in general. If you aren’t technically minded and don’t view mobile technology as an interest or hobby then having suddenly change could be bewildering. When parts of your UI change and others don’t or some devices do and others do not then that’s a bad user experience. The average user doesn’t have a clue what A/B testing is. They might even wonder if something is broken or perhaps they’ve accidentally changed something.

The same could be said with the way Facebook rolled out what they try to pass off as material design. Some had the tinted status bar and some didn’t. Those that did only had it in one screen. Some had a FAB and others did not. Sometimes the app would change on me twice in one day. When you’re the biggest social networking platform there is you should want to offer a good user experience.

Finally, there are things that could improve this process. It wouldn’t even be that hard to implement.

• Don’t half-bake the change. Make sure things are still smooth and match up properly.
• Instead of what can seem to be random server switches, use beta builds of the app correctly. That might not be possible with something like the Play Store, but it’s more than possible with other apps. Facebook even use that feature, just not very well. At the very least, make it an opt-in feature.
• If you’re going to use server-side switches make it account based. At least then all of your devices will work the same way as each other.

This isn’t likely to change in the near future, nor is it a big problem. I just think there are better ways of testing new features than what Google are doing now. Correcting this would surely help resolve the image of inconsistency Android has in some groups.