Sandeep Shah

Doing this is not nearly as difficult as most people would have you believe. Is your page relevant to every single person that comes to the website? Probably not. Start with a group of people you can identify. My favourite exercise is to throw a screenshot of the page onto paint/powerpoint/etc and just put red boxes around things that aren't relevant and green boxes around things that are. More often than not, you'll find that 75%+ of most pages are not relevant to each audience, it's just that something on the page is likely to be relevant to someone. Once you know what's wrong and not useful, it's then not hard to ideate on what to do about it.

Ongoing monitoring is hugely important. Far too many people will launch their experiments and then walk away completely for 2 weeks, forgetting that there could be a plethora of things that go wrong. Whether you're dipping in manually to check guardrail metrics, or using auto-alerting in a platform like Webtrends Optimize to be informed that things are misbehaving, being able to detect problems early and do something about them will go a long way to limiting the potential damage your experiment can do. And, don't underestimate just how bad you can make things. Broken ATB buttons and zero purchases, broken pages, broken checkouts - I've seen it all before as a result of an experiment someone has run.

It's interesting that an article on selecting an a/b testing platform focuses entirely on how to calculate statistical power. There are a considerable number of other factors to consider, and I'd argue that as most testing platforms calculate power in a reasonable way, this should be the least of your concerns. Instead, consider your intended audience, their skillset, and if the right tooling exists for them. E.g.: - Feature flags for developers, APIs, good code editors, etc. - WYSIWYGs and integrations with simpler platforms for marketeers. - Good options for advanced analysis for developers, etc. Ease of implementation, strength of support, integration with your business and being part of the team - these things matter too.

Sending data into Google Analytics from most 3rd party AB Testing platforms such as Webtrends Optimize is pretty straightforward. Either with a push-button integration, or by using a few lines of code to send specific values into GTM, it's easy to get data across whenever people enter an experiment. From there, as long as data is being pushed into the right dimensions, you'll be able to segment, slice, etc. within the Google Analytics interface.

The best testing programmes I've seen leverage experiments being run as a vehicle for capturing even more data. Close clicks, something downstream - there's never enough when it comes to data and insight of user behaviour, and it's an excellent opportunity if you're already running an experiment and so going through the cycle of dev/QA to push some additional metrics to capture into the loop, so you can feed the roadmap of future experiments. For example: We're testing on a basket page - go and track errors that people are seeing on the next step, so we can then work out if it's a big problem and what to do about it.

The best way to understand multivariate testing is by the limitations it overcomes. Let's say you're planning a really big AB test, where you've developed a brand new page and you want to send half of your traffic to it. How do you know what aspect of that new page is responsible for the improvement? And worse - what if one aspect of the page is single-handedly responsible for dragging down the performance of your experiment? These are not things that surface from a normal AB test that you run, but it's where MVTs shine. By breaking down your experiment into it's component parts, you get to point to specific changes as responsible for a proportion of the improvement, and specific changes as dragging the experience down.