Set up multivariate tests

Set up multivariate tests to simultaneously test multiple page element variations and identify the best-performing combination.

For more information on the differences between A/B and multivariate tests, watch the Get started with web experimentation course in the Kameleoon Academy.

What is a multivariate test?

A classic A/B test compares two versions of a page. A multivariate test (MVT) tests multiple variations of several page elements at the same time to find the combination that maximizes conversions.

For example, suppose you want to test a call-to-action (CTA) button with two colors (red and orange) and two wordings (Shop and Add to cart). A single MVT can test and compare all six possible combinations.

MVT glossary

• Multivariate test: An experiment that measures the conversion of one or more combinations of several elements.
• Section: A group of variations for a single element.
• Variation: A suggested change for a given element. An MVT can include several variations, grouped into sections.
• Combination: A set of variations from different sections that is displayed to exposed visitors.

Create a multivariate test

To create a multivariate test:

1. Click New experiment > In the graphic editor.
2. Select Graphic MVT from the Experiment type dropdown.
3. Fill in the required information and click Continue.

Set up sections and variations

Each section groups the variations of one element. The following example tests the color and wording of a CTA. There are two colors (red and orange) and two wordings: Shop and Add to cart.

To test the colors, create a first section (Section A, called Color) that includes your reference version and the variations red and orange.

To test the wording, create a second section (Section B, called Wording) that includes your reference version and the variations Shop and Add to cart.

Click Create. The MVT version of the Graphic editor opens.

Use the MVT editor

You can create your variations’ content the same way you would in an A/B experiment. However, the MVT editor is slightly different.

Sections

Click a section in the left panel to select it. The following options are available:

• Click the duplicate icon to duplicate the section.
• Click the three-dots menu to:
  • Rename the section.
  • Delete the section.

An MVT experiment must have at least two sections.

Variations

The selected section’s variations are displayed in the left panel. The following options are available:

• Click the duplicate icon to duplicate the variation.
• Click the three-dots menu to:
  • Rename the variation.
  • Add custom code (JavaScript or CSS) to the variation.
  • Preview the variation in a new tab.
  • Search and replace words on the page.
  • Delete the variation.

Deleting all of a section’s variations also deletes the section itself.

Kameleoon recommends changing no more than two elements on your page (a maximum of two sections), with three to four variations per element. Kameleoon combines the variations across sections to find the best combination, so the more sections you create, the more traffic you need to get usable, reliable results.

To learn more, read the article on statistical significance.

Kameleoon automatically creates all possible combinations from the variations you design, using a full factorial design that tests every combination simultaneously.

Allocate traffic

For more information on traffic allocation in MVT experiments, watch the How to finalize your web experiment course in the Kameleoon Academy.

Once you have created your variations, finalize your experiment the same way as a classic A/B experiment. The only difference is your experiment’s traffic distribution.

Traffic allocation for the experiment

To manage the experiment’s traffic allocation, click and drag the slider next to Exclude a percentage of the exposed visitors to your desired percentage.

Traffic allocation for exposed visitors

You must choose one of two ways to distribute traffic among exposed visitors:

• By section: You set the traffic share of each variation within each section. Kameleoon then derives each combination’s traffic from the shares of the variations it contains. Use the Sections tab when you want to weight individual variations, such as showing the red CTA to more visitors than the orange one.
• By combination: You set the traffic share of each full combination directly. Use the Combinations tab when you want precise control over specific combinations, such as excluding or boosting a particular pairing.

The two tabs are independent: editing the distribution in one tab doesn’t affect the other, and only one applies to your experiment. Kameleoon uses the tab that is selected when you click Launch. By default, the Sections tab is selected.

Distribute traffic by section

Click Sections to distribute traffic by section. Check the checkbox next to a variation to include it in your experiment.

To edit a variation’s traffic allocation, do one of the following:

• Click and drag the variation’s slider.
• Enter a percentage in the text field.

Click the padlock to lock a variation’s traffic allocation. Click the padlock again to unlock it.

In the following example, 30% of exposed visitors see a red CTA and 40% see an orange CTA.

Click Allocate equally to distribute traffic equally between variations. Clicking Allocate equally undoes your changes.

Distribute traffic by combination

Click Combinations to distribute traffic based on the combinations you’ve created.

Suppose you want to test a CTA’s color and its wording. You would have two sections: A (color) and B (wording).

Section A: COLOR	Section B: WORDING
Variation A0: Original	Variation B0: Original
Variation A1: Red	Variation B1: Add to cart
Variation A2: Orange

The generated combinations have the following names:

• A0B0 Original / Original
• A0B1 Original / Add to cart
• A1B0 Red / Original
• A1B1 Red / Add to cart
• A2B0 Orange / Original
• A2B1 Orange / Add to cart

You can exclude combinations in the Combinations tab. For example, in an MVT that changes both the background and text colors, you can exclude combinations where the background and text colors are the same.

Check the checkbox next to a combination to include it in your experiment.

To edit a combination’s traffic allocation, do one of the following:

• Click and drag the combination’s slider.
• Enter a percentage in the text field.

Click the padlock to lock a combination’s traffic allocation. Click the padlock again to unlock it.

Click Allocate equally to distribute traffic equally between combinations. Clicking Allocate equally undoes your changes.

Simulate, launch, and analyze your MVT

Once you’ve created your variations and distributed traffic, simulation, launch, and analysis work the same as in an A/B experiment.

Simulate your MVT

Kameleoon recommends simulating your experiment before going live.

The simulation panel generates your experiment’s combinations. Click Currently displayed in simulation to check the display of the combination of your choice.

To learn more, read the article on simulation mode.

Launch the MVT

If everything displays correctly and your targeting, goals, and reporting tools are set up, you are ready to launch. Click Launch.

Analyze the results

If you selected Kameleoon as your reporting tool, your MVT’s results are transferred to the A/B experiment Results page. Each combination is considered a variation, as defined for classic A/B experiments.

You might experience a short latency (up to 10 minutes) between launching your experiment and its visibility on your website. Don’t worry if your experiment does not appear immediately.