One of the tasks that I am currently working on involves using A/B testing and investigating areas where this can be used on the current websites. For those who have not heard of A/B Testing, A/B Testing is the ability to test changes to the user interface on a website in order to determine if these changes are beneficial to the users. This tests one version (or aspect of the user interface) with one or more versions; a defined percentage of users will randomly get one version. The test results statistics will be examined to determined if the user interface change should go live. A/B Testing will highlight whether or not the change draws users in or sends them running away.
However, there are certain aspects of A/B Testing that should be followed. One of the clients had mentioned making several changes to several different areas of the website and tracking which version of the website was more useful to users, but multiple changes are not recommended. With multiple changes, it is unclear which aspects or changes are beneficial.
In "The Dark Side of A/B Testing: Don't Make These Two Mistakes!", the author mentions a retailer website's multiple A/B tests; the company's revenue increased, but they were unable to conclude which aspect of the test increased their revenue because major multiple changes to the website had been made (1). In short, developers and business owners need to be very careful when writing their tests.
However, if multiple changes need to be made, developers can use Multivariable Testing. This form of testing allows multiple versions for user interface changes to be created, categorised, and tested over a period of time. According to "Multivariable Testing", A/B Testing is superior to Multivariable testing because A/B Tests are easier to set up, results are quicker, clearer and more accurate; however, Multivariable Testing is acceptable if used with care and when multiple changes are absolutely necessary (2). The article also provides a checklist of notable considerations and influences when using A/B Testing as opposed to Multivariable Testing, and this is well worth a read.
In short, business owners need to be clear of the A/B Testing objectives and to take care when making decisions about testing their websites.
(1) Balkin, Matt. "The Dark Side of A/B Testing: Don't Make These Two Mistakes!". http://blogs.omniture.com/2006/11/02/the-dark-side-of-ab-testing-dont-make-these-two-mistakes/ [2 November 2006].
(2) Alt, Brian and Usborn, Nick. "Multivariable Testing". http://www.marketingexperiments.com/improving-website-conversion/multivariable-testing.html [29 December, 2005].
Leave a comment