A/B Testing: How Much Data Do You Need?
I’ve started an A/B test for a client almost 3 weeks ago. We are trying to determine if we can bring overall reservations and revenue up by changing the layout of the homepage. Our hypothesis is that if specials were promoted in a more visible fashion on the homepage, conversion rate would increase, as would overall revenue.
RESULTS - 2 DAYS
After two days of data, we saw that the “B” version of the homepage had twice as many specials booked than the “control,” or “A” version. As a percentage of bookings, version B had 17% more specials booked than A. Version B also showed 71% more total bookings, and an increased conversion rate of 71%! This was looking to be very promising for our hypothesis.
RESULTS - 12 DAYS
With 12 days of data, we saw that as a percentage of bookings, version B had 23.22% more specials booked than A (up from 17% at the last check). However, overall conversion rate had dropped to be 8.1% below the control version (previously 71% higher). I decided that with such a swing in numbers, we probably didn’t have enough data to be able to draw a conclusion, and we should let the test continue to run for a few more days.
RESULTS - 19 DAYS