Now an Adobe Platinum Partner. Click here to learn more.


Marketing Madness: Website Usability Testing

By: David Lucius
< Back to resources

From UMBC’s historic round one win over Virginia to Loyola Chicago making it to the Final Four, this year’s NCAA Men’s Basketball tournament has been exciting and unpredictable. Since my bracket busted early, I’ve happily found another, more productive, outlet to channel my competitive spirit… website usability testing.

Is your company constantly comparing its website and ecommerce store to a handful of competitors’ sites? Do you have a #1 seed competitor who always seems to be one-step ahead? Instead of just making assumptions about why these organizations are successful, why not put it to the test – March Madness style.

Why is usability testing valuable? The key to a winning strategy is to discover how prospective customers interact with your site and compare it against how they interact with your competition. One benefit of usability testing is that it doesn’t require any analytics tags, so you can run tests on any site and gain valuable information about how your competition is perceived.

I challenge you to setup your own mini-bracket, aka usability test, and pit yourself against your competition to see who emerges as the winner.

Step 1: Select the right testers and “teams” for the competition

Identify your customer audience and find testers who fit the profile. If your goal is to attract customers from a different demographic set, make sure to include this segment of customers in the mix too.

Next, identify the competition against which you want to test. This can come from competitor comparisons heard around the office and by looking online to see to who your company is compared to in search results and blogs.

Step 2: Setup the test “brackets”

Setup your test by pitting your competition one-on-one against each other. There are a few different ways to approach this type of test. A method I’ve found beneficial is to have testers visit two sites each. Ask each tester to navigate first to one site, and then to the other verbalizing their thoughts and impressions, and comparing the sites as they go. It’s important each tester attempts comparable tasks on both sites – i.e. signing up for a free trial, purchasing a product, etc. Along with this, ask your testers the same questions about each site. Since you’re asking for people’s opinions, asking the same question of each site allows you to easily compare and stack rank. Though this is not an A/B test, one thing you should not overlook is adjusting for the inherent bias that exists by having a user go through one experience before the other. People have a tendency to prefer a process they are familiar with, even if only by a matter of a few minutes. To account for this, the tests should run in pairs, with each tester seeing the other site first.

Step 3: Analyze the results to determine the winner

Once you have the data, it’s time to assign scores and compare. The quantitative metrics (i.e. any of the questions that have a numerical answer) employed in the test can be compared directly.

Website Usability Testing

Assign weighted values to any qualitative metrics and score accordingly. It’s also a good idea to include a Net Promoter Score question as part of the question set, which can be used to compare directly between companies.

Website Usability Test

Armed with the final scores you can determine how your company’s site stacks against your competition, including that #1 seed, in terms of overall score. You also can dive deeper into each individual user’s test data. This allows you to learn some of the less obvious strategies from the competition that your company could benefit.

So, how did you do? If your company came out as the champion, congratulations! You get to cut down the net and take home the trophy! Though, even if you came out on top, there are likely still learnings and plays to improve. If you were not the clear winner, now is the time to get to work. Use data along with traditional analytic KPIs to formulate an optimization plan. If any “slam dunk” insights came to light, consider implementing those immediately. If you tested against companies outside of your usual demographic, you can be strategic when making enhancements and build a Conversion Optimization Strategy of A/B (or MVT) tests to measure how your actual customers respond to changes you implement. In any case, hopefully you’ll have gained new ideas to better improve your site experience to ensure that you always end up on top.