Website Analytics: Time For an A-B Workout

The web and design teams are hot on the heels of upgrading liferay.com, with a full redesign soon after.  We're still running our site map through executive approval, but our hope is to make a shift from how our current site flows.  We want to streamline our navigation and content so everything is focused on and around our flagship product: Liferay Portal.  By doing this we can serve our first-time visitors clearer messaging, and also set ourselves up for quicker turnarounds in refreshing and improving site content.

Another another main focus we have is beefing up our site analytics so we can see how people are flowing through our site, where the roadblocks are, and renovating those areas.  One thing that we've been testing and am excited to share is a simple way to do A-B testing via web content templates.

So what's A-B testing?  If you're unfamiliar with A-B testing, it's a little bit like blind-folding people and having them compare Coca-Cola versus Pepsi.  Or a hamburger with special sauce versus a hamburger with new improved special sauce.  Or a website banner that uses a stock image, versus a website banner that is done with in-house illustration.  The idea is to serve users a 50/50 sampling of two versions of content, and sit back and watch the comparison results come in - the version that gets the most hits/clicks/downloads wins.  Then you can keep the "winner", and if/when needed, you can pit the winner against newer revisions of content.

So pictured above is one example of where we've implemented this: our Enterprise Portal signup form.  The pre-existing form was looking pretty plain and we thought it could use a minor refresh.  The issue is that we've seen that lots of people land on this form just to leave it without ever filling it out.  So as a really quick update, Design passed us some prettier styling, and we incorporated it into one of our two CMS template velocity blocks like so:

#set ($number= $mathTool.random(1,3))
 #if ($number == 1)
   <!-- old content goes here -->
 #else
   <!-- new content goes here -->
 #end

It's pretty darn simple, but it works like magic. We added disctinct google event tracking scripts to each of the submit form buttons, with an onclick like so:

onclick="_gaq.push(['_trackEvent', 'A-B Tests', 'Get It Now Form Submit', 'Original']);"
	onclick="_gaq.push(['_trackEvent', 'A-B Tests', 'Get It Now Form Submit', 'Revision 12-20-2012']);"

....and the results have been rolling in over the past few weeks.  What does the final tally look like?

So over 21 days of testing, the results are super close.  The new form has so far been .8% more successful that the pre-existing form (using my math, not google's math).  This tells us that, well, (1) the styling change in this case doesn't matter, and (2) at least our velocity function is great at giving a nice 50/50 of version A versus B!

Though this particular test didn't yield very riveting effects, the goal was to just set ourselves up for future experiments and start to include hard metrics in guiding and proving our website decisions.  Hopefully this can help some of you out there who are trying to incorporate the same type of things to improve your website.

 

Thanks,

Webteam

ブログ