Multivariate testing is far less extreme than A/B testing. Even in cases where you are varying a number of variables, they’re still limited to the context of a single page — in contrast to varying an entire experience with A/B. So the potential gains and losses are less extreme as well. A lot of lift can be found using MVT when the interaction effects of multiple variables come together in a powerful and perhaps unexpected way. In relatively minor MVT scenarios (where you’re testing a few versions of a headline for example) you’ll likely come out of your test with a pretty clear understanding of causation. In more complex cases, where many elements are varied on a page to produce a winning combination, your inference of causation may be flawed. But again, why something won or lost is less important than the reason we’re testing in the first place — to generate more business from less marketing spend. 

 

Before We Begin

Start with a test plan — know what you want to test and why.  If you are new to testing, you can download a copy of ion’s “Guide to Online Testing” for more information on formulating your test strategy.

Let's Get Started!

To get started, click into the creative you are ready to set up an MVT experiment on. In the bottom, left corner of your page thumbnails you'll find a beaker icon. An empty beaker indicates an experiment is not set up. A green, full beaker indicates a live MVT experiment is running. A yellow beaker indicates a paused experiment. Click over the beaker to open up your MVT editor.

You'll land on your variations tab where you'll find each element on your page. Click on the green plus sign beside the element you'd like to get started with. You'll find that the content, image, form and link editors function the same way as they do right on your template. The element area will pre-populate with the original content and you can replace it to add a new element. Add as many element variations as you wish.

 

Add variations for each element you'd like to include in your experiment. To the right, you can preview each combination that has resulted from the variations you've created. The more combinations you've created, the longer it will take to achieve a winning combination. If you click over the preview of a combination, you'll be able to see a larger, full-page view of the combination you are previewing.

There is an Estimated Experiment Duration Calculator available as well. You can input the pages avaerage daily views and LiveBall will calculate an estimated duration. The more more traffic you receive and the greater difference between the elements you are testing, the sooner you will see results.

When you're ready to launch your experiment, click on the play button beside the status bar. Your experiment’s status will switch from designing to running.

You have two options when running an experiment—manual or automatic optimization. By selecting manual optimization you’re electing to monitor your results yourself. This means that you’ll take combinations out of rotation as well as apply the winning combination to the page. By selecting automatic optimization LiveBall will retire combinations that are not winning when statistical significance is reached. 

Now that your experiment is running, you can view your elements and combinations tabs.

The elements tab displays each element you're testing and the variations of it that you have created. There's an influence bar that indicates the elements influence over conversion performance. In the example below, the headline has a greater influence on conversion results than the subhead.

The variation on the elements tab that is greyed out is the original. You'll be able to see estimated conversion rate, chances to win, lift and conversion rate for the original and each variation. The estimated conversion rate bar is color coded.

• Green means the variation is performing better than the original.
• Yellow indicates confidence has not yet been reached to determine if this variation is performing better than the original or not.
• Grey shows that the variation is performing the same as the original.
• Red indicates this is variation is performing worse than the original.

In the example below, there is statistical confidence that the variation is performing worse than the original. There is a margin of error to the right of the estimated conversion rate. The colored bars do not overlap.

The combinations tab displays each combination of elements that is being tested. You can scroll your mouse over the combination for a preview of it. You'll be able to see estimated conversion rate, chances to win, lift and conversion rate for each of these combinations. The color coding works the same way here, except it reflects the prediction for an entire combination against the original page.

You can manually disable lesser performing combinations here. You can also apply the winning combination to your page here.

If you're using automatic optimization, LiveBall will disable poorly performing combinations for you! 

You can pause and reset your experiment any time. If you reset your experiment, the stats of your current experiment will be reset to zero and you can begin a new experiment.

A helpful note: back on your campaign management screen you'll find an MVT column. Green checks in this column indicate a live MVT experiment. Exciting! Yellow checks mean a paused experiment and grey checks indicate an experiment with a status of "designing".

A note about test duration:

The major factors that determine how long a MVT experiment may need to run are:

• the number of combinations
• the traffic rate (i.e.# of respondents visiting the page per day)
• the original combination's conversion rate
• the expected lift percentage by at least one of the competing combinations
• the console's confidence level all factor into the duration estimate

 

In general an experience needs to run longer when there:

• are a large # of combinations
• is a small traffic rate
• is a conversion rate that is close to 0% or 100%
• is a small expected lift percentage
• is a large confidence level