´óÏó´«Ã½

A little while ago I was talking to our interaction designer Pekka about a link we had in the masthead that was under-performing. We suspected it was the wording that wasn't working, and after thinking up a few possible options Pekka said it would be great to be able to "A-B test" some of them. This led to some blank gawping from me, and so in turn to an explanation from him.

It turns out '' is a way to show different options for part of the UI to two (or more) groups of users, and then monitor what they do as a result. This can be done in normal of course, but more interestingly it can also be done on a live site with large numbers of users - getting real world results on a statistically valid scale. In our very simple case we could show the different wording options to different groups and see which one generated the most clickthroughs.

This sounded like a great idea, but not something I could fit onto the project backlog at the time. Luckily, some teams here set aside a little time within their project planning specifically to let developers 'scratch that itch'. No, not like that. I mean work on something that's bothering them rather than whatever's needed next in the project. Our team uses a system within our normal project planning process, , to give us up to a day a fortnight for this.

Thus I decided to spend a day writing this mechanism for doing A-B testing on bbc.co.uk.

The code

The JavaScript is split into two parts. The first is used for new visitors, and, aside from checking the function's configuration is valid, simply uses the probability value to decide whether the user is going to be part of the test, and cookie them accordingly.

The second is used for those who already have a cookie, and have been randomly selected for testing. It cycles through all the links on the page, checking for any that have a class starting 'abtest_'. When it finds one it adds an onclick which, when activated, makes a call to a tracked image so that the click is counted.

This tracking is done using the ´óÏó´«Ã½'s stats system, known internally as 'go-tracking', which allows us to easily get reports from the server logs showing how many times particular URLs are visited.

To make these calls I used img.src="..." to make an asynchronous call from JavaScript. Note however, if the link in question goes to a new page about 1/3 of browsers won't complete the request before the page unloads. If this was an issue for a particular test the links' hrefs could be changed, so they go via a tracked URL instead.

Finally, in the page itself we place some small snippets of SSI (though we could easily use more Javascript or PHP) that check for the cookie, and change what the user gets shown according to its value.

Example use

For 5 in 100 people to get a two-option test running for 24 hours the function is initialised like this:

abtest.init(
    'mydemo', {
        probability:5, 
        numOfOptions:2, 
        path:'/includes', 
        duration:24
    }
);

Within the markup, content is changed according to the cookie value:

<!--#if expr="$HTTP_COOKIE=/abtest_mydemo=a/" -->
	Text a
<!--#elif expr="$HTTP_COOKIE=/abtest_mydemo=b/" -->
	Text b
<!--#else -->
	Default text
<!--#endif -->

And classes like this are added to links that are to be counted:

class="abtest_mydemo"

Any clicks on that link will result in an asynchronous call to an image with a path that includes "mydemo" from the class and the option value (e.g. "b") from the cookie:

/go/abtest/int/abtest_mydemo_b/

Thus after the test we can count the totals for each option, and use those figures to help decide which solution to go with.

And the winner is...

Well, whaddya know: the design changed, removing the button in question, so I'm still waiting for my first excuse to use it. However, my friend Manjit over in World Service has just run a test of the wording for the main link to their syndication widget, and immediately proven a small tweak would be three times as effective at getting users' attention. Not a bad first outing.

As a method it's not without its dangers of course. Your presented options must be realistic, and not just chasing clickthroughs for example - but with an experienced information architect at the helm I think this little snippet may come in quite handy.

What do you think? Have any of you used this kind of testing?