Lily McIlwain, content manager at Triptease, discusses her company’s latest report, which aims to tackle the most overused term in digital marketing: A/B testing – and how it may not be relevant for most travel websites.
It would take the average independent hotel two years to run a statistically significant A/B test. That’s because the vast majority of small properties just don’t have enough traffic to their website to be able to perform a trustworthy test.
One of the most damaging misconceptions about A/B testing is that anybody with a website can perform one. In fact, running an A/B test on too little traffic is a waste of a hotelier’s valuable time – and worse, could lead them to make important business decisions based on seriously dodgy data.
As A/B test providers to some of the world’s largest hotel groups, Triptease is uniquely poised to comment on the myths and misconceptions surrounding split-traffic testing. The report Spotlight on… A/B testing sets the record straight on why independent hotels are unlikely to be able to run statistically significant tests, why hotels and OTAs are on a very different playing field when it comes to testing, and why a 10% uplift in an A/B test doesn’t translate to a 10% uplift in your bottom line.
Spotlight on… A/B testing was written in collaboration Triptease’s dedicated testing squad, who regularly perform tests on some of the biggest hotel websites in the world. The report is designed to address the questions and concerns many hoteliers hold about A/B testing.
“If it’s there, it’s been tested”
One example of such a concern is the idea that hoteliers need to be constantly testing in order to keep up with OTAs like Booking.com, who famously run ‘thousands’ of tests each year.
At ITB 2018, Booking.com’s CEO Gillian Tans made no bones about the extent to which her team are constantly iterating on their product: “Everything you see in our product, in our advertising or on our website has been tested. If it’s there, it’s been tested.”
This emphasis on testing creates a pressure on small hotels that is difficult to ignore. There are several reasons, though, why hotels and OTAs aren’t on the same playing field when it comes to testing. But that doesn’t mean hoteliers should lose hope.
In the following extract from Spotlight on… A/B testing, Triptease outlines some of the main differences between hotels and OTAs when it comes to A/B testing:
OTAs have the necessary economies of scale to invest heavily in data infrastructure and achieve accurate testing results for minute UX (user experience) changes. They can change the colour of a button and test the impact because they can achieve a sample size that enables them to detect tiny uplifts.
The vast majority of hotels don’t have the necessary traffic to be able to do this. While measuring tiny uplifts, such as a 0.1% increase in conversion rate is worthwhile for huge OTAs, it almost certainly isn’t for most hotels. Hoteliers should be focused on identifying areas for significant improvement rather than testing each button change.
Due to their scale and the breadth of their user base, OTAs have a far greater number of reference points by which test results can be assessed. Traffic to OTAs is made up of millions of different types of consumer, which means OTAs can test the impact of a change in any number of different user groups.
OTAs can see which changes have the biggest effect on which type of person, and why. The largest hotel groups can perform this kind of analysis, but a single property or a small group only has a very narrow traffic base to test on. It is difficult to capture many confounding variables on such a small base.
Knowing the guest
This section may be slightly dispiriting reading for hoteliers. But it shouldn’t be. Hotel websites are about far more than just converting a guest. They’re about providing a great experience. One thing an OTA will never be able to do is actually meet and speak to the person who is booking a room through them.
While you might not have millions of visitors to your website each month upon which to test, your hotel is full of potential research subjects. One of the most simple things a hotelier can do is just ask each guest about their booking experience as they check in.
“Really understanding your guests will set you well on the way to an optimised website.”
The hospitality industry is defined by human relationships. While OTAs are only focused on one specific part of the guest ‘journey’, hotels are necessarily invested in the entire trip – from booking, to pre-stay, to stay, to after the guest has gone. The knowledge and understanding that this gives hotels are often underestimated. A hotelier can apply their ‘soft’ knowledge of guests to how they optimise their website in a way that OTAs cannot.
A hotel website does not have to be all things to all people, as an OTA does. Really understanding your guests will set you well on the way to an optimised website.
To read more about A/B test mythbusting, Best Western UK’s view on A/B testing and Triptease’s secret to a successful A/B test, download the full report.