A/B Testing: How Much Data Do You Need?

• facebook
• Stumble It!
• del.icio.us
• View blog reactions

I’ve started an A/B test for a client almost 3 weeks ago.  We are trying to determine if we can bring overall reservations and revenue up by changing the layout of the homepage.  Our hypothesis is that if specials were promoted in a more visible fashion on the homepage, conversion rate would increase, as would overall revenue.

RESULTS - 2 DAYS

After two days of data, we saw that the “B” version of the homepage had twice as many specials booked than the “control,” or “A” version.  As a percentage of bookings, version B had 17% more specials booked than A.  Version B also showed 71% more total bookings, and an increased conversion rate of 71%!  This was looking to be very promising for our hypothesis.

RESULTS - 12 DAYS

With 12 days of data, we saw that as a percentage of bookings, version B had 23.22% more specials booked than A (up from 17% at the last check).   However, overall conversion rate had dropped to be 8.1% below the control version (previously 71% higher).  I decided that with such a swing in numbers, we probably didn’t have enough data to be able to draw a conclusion, and we should let the test continue to run for a few more days.

RESULTS - 19 DAYS

After 19 days of running the A/B test, and roughly 10,000 sample website visits, we have solid data that proves version B converts a higher percentage of specials than version A.  However, the overall conversion rate of version B has now dropped even further below the control. On the upside, the data also showed that average order per reservation was 5% lower than version A (an improvement from the 6% shown a few days earlier), and average revenue per night booked is now 0.1% higher for version B (previously 4% lower).  So...more swinging of numbers.

CONCLUSION?

So, what have we been able to conclude from all of this data?

From these raw numbers, we might conclude that we should stick to the original version of the homepage because, overall, it converts better.  However, when we look at the behavior of the 2 different groups of visitors, it opens a whole new theory.  We think that because version B highlights the specials more openly than the original version, it’s possible that we are changing the way visitors are booking their vacations.  Instead of going through a linear booking process, it’s possible that version B visitors are now “shopping for deals."  A thought-process something like, “Hmmm, I remember seeing a bunch of specials on the homepage.  I think I’m going to check all of them to see which one I like the best.”  We’ll implement some new tracking features that will allow us to better get in the minds of these visitors. I'll be commenting on this subject in another blog post.

With that theory, and the fact that the numbers are still flip-flopping, the existing test is going to continue to run a little longer.  

Stay tuned for final results…In the meantime, unless you are experiencing 100% conversion rates on your website, I encourage you to start A/B testing.  Just don't jump to conclusions after 1 day of testing.  Depending on the size of your website, the number of visitors, and seasonality, it may take longer than you think to get "normalized" results.  No matter how long it takes to run the test, I’ll bet that you’ll be surprised at the outcome!