A/B testing in 10 steps

1) Deep Dive into the Analytics

The first phase of any Conversion Rate Optimization project involves a deep dive into the analytics of the digital property.

Some of the things I like to look for first in no particular order:

• Pages with high value and high exit rate: This will often give you a good idea of where to start.
• Traffic sources: What effect does PPC or organic traffic have on conversion goals?
• Device: Are there different patterns or blockages?
• Geolocation: Does location make any impact? Why are people ordering at 2am on Sundays?
• Time of Day/Week: What influence does time have? Weekend traffic to weekday traffic.
• Custom Segments: A great idea is to look at reverse goal paths or segments.

2) Divide the digital property into areas of interest for testing (Do this for mobile, tablet and desktop).

This helps get the testing organized and prioritized.

• Store Pages
• Product Detail Pages
• Home Page
• Site-wide (Header/Footer/Navigation)
• Check Out Funnel
• Category Listing Pages
• Campaign Landing Pages
• Goal Pages - email sign up or registration and any of their funnels

3) Ponder these areas using common pinch points.

• Is the user confused?
• Why should the user stay on this page?
• Does this page give the user the heebee-jeebies?
• What are the next steps?
• Is the page simplicity to use?
• Is there purpose to users tasks?
• UX experience
• UI experience
• Functionality

4) Rank these by rating the easiest pages to test on that will produce the highest return?

I like to use a spreed sheet for this.

5) Build a hypothesis. If we change x, y will happen.

Look to build experiment with psychological triggers and marketing tactics. Depending on the context of the test not everything will work but you might gain useful insights for the next round of testing.

6) Construct an experiment to test your theory.

This is the fun brainstorming part.

7) Design, Develop and QA.

Make a checklist and just like santa check it twice or better yet a thousand times.

8) Run the test.

Don't peek and make sure you leave it long enough to run. Watch out for traffic pattern changes and phantom winners.

9) Is there a statistically valid winner?

If yes do you know why it won? Can you see anything in the data? Use a custom dimension in Google Analytics to show the user paths through the variations.

If no what insights did you gain? Why did what you expected to happen not happen? How can you use that knowledge?

10) Iterate

Don't give up. Often the best profits come after 6 or 16 rounds of testing.