Working with experiments

Brightspot offers built-in experimentation functionality to help you A/B test your content. This functionality empowers you to test, iterate, and optimize your digital experiences right within the same CMS in which you create content.

Configuring experiments

Experiments help you make strategic content decisions in order to better capture visitors' attention. Each experiment is comprised of at least one variation, and each experiment will have results that inform your content decisions.

To configure an experiment:

1. Search for and open an existing asset for which you want to create an experiment.
2. In the side toolbar, click .
3. In the Variations widget, click Create a Variation.
4. In the New Variation pop-up, in the Name field, give the variation a name.
5. Edit the asset as desired.
6. Click Publish.

Note

Clicking Publish does not make the variation live or override the existing, default version of the asset. It simply saves the variation so that it may be used in experiments.

7. Repeat steps 3–6 to add any additional variations.

8. In the Experiments widget, click Create an Experiment.

9. In the New Experiment pop-up, do the following:

   1. In the Name field, give the experiment a name.
   2. In the Description field, describe the experiment.
   3. Under Schedule Start On, select a date and time when the experiment will begin; however, if you want to run the experiment right away, retain the default Start Immediately.
   4. Toggle on Auto Promote Winner to automatically promote the experiment winner to be the asset's primary variation.
   5. Click Next.
   note

   If the asset does not have its own page (for example, a shared asset reused across multiple pages), a Run On Pages field appears at the top of this page. Use it to define which pages the experiment runs on. See Targeting multiple pages with Run On Pages for details.

   6. Under Delivery Rule, fill out the following fields:

      1. Under Target, select one of the following:

         • All Visitors—All visitors are included in the experiment.
         • Segment—Define a segment of visitors that are included in the experiment.

      2. Under Exposure, select the percentage of this traffic that will be included in the experiment sample. By default, this percentage is set as 100.

      3. Under Traffic Allocation, select one of the following:

         • Allocate traffic dynamically—Automatically manages your traffic allocation using a multi-armed bandit in real time based on the performance of the variations.
         • Manual—Manually allocate traffic by fixed percentages to show for each variation and control.

      4. Under Variations, select the variations you want included in the experiment. If your role has the Create Multiple Running Experiments permission, then you will only see unused variations. This is to prevent multiple experiments from using the same variations.

      5. Under Control, select the variation that serves as the default. By default, the first version of the asset is set.

   7. Under Mutually Exclusive Group, optionally select a group to prevent this experiment from running at the same time as other experiments in the same group. If you leave this blank and the experimentation account has a default group configured, Brightspot uses the default. See Creating mutually exclusive groups.

   8. Click Next.

   9. Under Goal, select one of the following placements:

      1. Shared—Select a goal that was already created for this account. This is the default placement. Under Goal, select the shared goal. See Creating shared goals.

      2. Inline—Define a new goal directly on this experiment. Under Goal Type, select one of the following goal types:

         1. Number of Page Views—Tracks the number of pages a visitor has viewed during a session. For example, if you want visitors to engage with more of the site and wish that they visit at least five pages, you would create the following goal:
            • Comparison: Greater Than
            • Page Views: 5
         2. Access To Page—Tracks whether a visitor has accessed a specific page/URL regardless of whether a button was clicked. For example, if you want to increase engagement to a page, you might add a module that only appears on pages that match a referral pattern. You would then create the following goal:
            • Comparison: Contains
            • URL: Example Sustainability Path (/sustainability)
         3. Click Tracking—Notes whether a visitor clicks on a specific element on a page (like a button or a link). For example, if you want to test whether introducing a different call to action design will motivate users to click on the demo request form, you would create the following goal:
            • Page URL: Target Page
            • CSS Selectors: a.Button[href="target page"]
         4. Scroll Tracking—Notes how far a visitor has scrolled down a page (for example, 50%, 70%, and so on). For example, if you want visitors to spend more time on a page and scroll through at least 70% of it, you would create the following goal:
            • Page URL: [Current permalink]
            • Scroll Target Percentage: 75
         5. Time Spent—Tracks how long a visitor stays on a site or a specific page. For example, if you want to redesign a page to see if users view more of its content, you would create the following goal:
            • Comparison Type: More Than
            • Duration: 5
            • Time Unit: Minutes

   10. Click Done.

Note

If your role has the Create Multiple Running Experiments permission, you can repeat the previous steps to create additional experiments. Please see Experimentation best practices at the bottom of this topic for additional information.

Targeting multiple pages with Run On Pages

For most experiments, Brightspot determines which page an experiment runs on directly from the asset's own permalink. Some assets, however, do not have a page of their own (for example, an asset that is shared and reused across several pages). When you create an experiment on this type of asset, the New Experiment pop-up displays a Run On Pages field so you can define which pages the experiment runs on.

Under Run On Pages, select one of the following options:

1. Selected Pages—Runs the experiment on the permalinks of a fixed, editorially curated list of assets. Assets without a permalink are ignored. In the Pages field, select the assets you want the experiment to run on.

2. Pages Matching Query—Runs the experiment on the permalinks of the newest assets that match a content query, up to the account's configured page limit. The set of pages is refreshed automatically unless you select the Disable Auto Refresh checkbox. Configure the query, and, using the following table as a reference, complete the fields as needed:

   Field	Description
   Max Content Age (Hours)	The number of hours after publishing before a page is removed from the experiment on the next refresh. Type 0 to keep pages regardless of age. Defaults to 72.
   Disable Auto Refresh	Select this checkbox to only recalculate the matching pages when you save the experiment.
   Refresh Interval	From the list, select how often the matching pages are recalculated: Hourly, Every 6 Hours, or Daily. Defaults to Hourly.

3. Pages Matching Pattern—Runs the experiment on every page whose path matches a regular expression you provide. In the Path Pattern field, type a regular expression to match against normalized request paths, for example, ^/hub/.*. Paths start with / and have no trailing slash.

4. All Pages—Runs the experiment on every page on the site.

note

Brightspot limits the number of pages Selected Pages and Pages Matching Query can resolve to a maximum configured on the experimentation account. An administrator must also enable unbounded experiments on the account before you can select Pages Matching Pattern or All Pages. When enabled, the account also limits how many variations an unbounded experiment can define, how many unbounded experiments can run at the same time, and the shortest ramp duration allowed—if you exceed any of these limits, Brightspot displays a validation error when you try to save the experiment.

Because Pages Matching Pattern and All Pages do not target a fixed, enumerable list of assets, Brightspot ramps them in progressively instead of enabling every matching page at once, spreading the CDN cache impact of the experiment out over time. In the Ramp Duration (Hours) field, type the number of hours the ramp takes to complete after the experiment starts. Longer ramps spread the CDN cache impact further. This field defaults to 48.

Before any option is configured, the Run On Pages field displays a warning that the experiment will not serve on any page until Run On Pages is set up.

Once configured, the field's note depends on the targeting option. For Pages Matching Pattern and All Pages, before the ramp starts it states how many hours the progressive ramp will take once the experiment begins; while ramping, it shows the percentage of targeted pages currently running and when full coverage is expected; once complete, it confirms the experiment is running on all targeted pages. For Selected Pages and Pages Matching Query, it instead displays a count and sample of the pages the experiment currently targets, or a warning if the option currently matches no pages.

Checking experiment status

Once you have created an experiment, you can check its status.

To check experiment status:

1. Search for and open an existing asset that has an experiment of which you want to check status.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose status you want to view.
4. View status.

Note

Statuses include the following:

• Running—The experiment is currently running.
• Running (MTU Limited)—The experiment has been temporarily paused because the monthly tracked users/site traffic limit has been reached.
• Paused—The experiment is paused.
• Planned—The experiment has been scheduled, but has not started yet.
• Planned (MTU Limited)—The experiment has been scheduled, but has not started yet, and will not start until the monthly tracked users/site traffic limit is reset.
• Completed—The experiment has completed.
• Archived—The experiment was archived.

Editing experiments

Once you have created an experiment, you can update it.

Caution

Modifying an experiment in progress may produce skewed results. See Experimentation best practices at the end of this topic for more details.

1. Search for and open an existing asset that has an experiment of which you want to check status.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose status you want to view.
4. Click > Edit Experiment.
5. Make the desired changes.
6. Click Save.

Pausing experiments

You can pause an experiment when needed.

1. Search for and open an existing asset that has an experiment of which you want to check status.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose status you want to view.
4. Click > Pause Experiment.

Archiving experiments

If an experiment is not stable, you can archive it.

1. Search for and open an existing asset that has an experiment of which you want to check status.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose status you want to view.
4. Click > Archive Experiment.

Searching and filtering for variations and experiments

At times, you may have many variations and experiments. In such cases, you can search and filter for variations and experiments so you can quickly take the appropriate actions.

To search for variations:

1. Search for and open an existing asset for which you want to search for a variation.
2. In the side toolbar, click .
3. In the Variations widget, use the search bar to search for your desired variation.

To filter for variations:

1. Search for and open an existing asset for which you want to filter for a variation.
2. In the side toolbar, click .
3. In the Variations widget, click on the Any Experiment drop-down list.
4. Select the desired experiment.

Brightspot displays only those variations used in the experiment you specify.

To search for experiments:

1. Search for and open an existing asset for which you want to search for an experiment.
2. In the side toolbar, click .
3. In the Experiments widget, use the search bar to search for your desired experiment.

To filter for experiments:

1. Search for and open an existing asset for which you want to filter for an experiment.
2. In the side toolbar, click .
3. In the Experiments widget, click on the All Statuses drop-down list.
4. Select the desired status of the experiment you want to view.

Brightspot displays only those experiments with the status you specify.

Ending experiments

At times, you may find it necessary to end an experiment before a winner has been determined.

To end an experiment:

1. Search for and open an existing asset whose experiment you want to end.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment that you want to end.
4. Click > End Experiment.
5. In the End Experiment pop-up, toggle on Archive variations associated with this experiment if you want to archive associated variations.
6. Click Confirm.
7. Click Done.

Brightspot marks the experiment Status as Completed but will report the Winner as Undetermined. In this case, you must manually promote a variation as the winner of an experiment. For details, see Manually promoting a variation as the winner of an experiment.

Manually promoting a variation as the winner of an experiment

If an experiment is not yielding the expected results and you want to end the experiment prematurely, you have the option to complete it before a winner is determined. This action archives the experiment.

To manually promote a variation as the winner of an experiment:

1. Search for and open an existing asset for which you want to manually promote a winner for a running experiment.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose winner you want to promote.
4. Click > End Experiment.
5. In the Variations widget, find the variation you want to promote as the winner.
6. Click > Promote.
7. In the Promote Variation pop-up, click Confirm.
8. Click Done.

Viewing experiment results

Once an experiment has ended, you can view its results to inform future content strategy decisions.

To view an experiment's results:

1. Search for and open an existing asset that has an experiment of which you want to view results.
2. In the side toolbar, click .
3. In the Experiments widget, find the existing experiment whose results you want to see.
4. Click the bar chart icon to open the Experiment Results pop-up.
5. View results.

Tip

The following descriptions may help you interpret the results:

• Reliability—A score lower than 95% means that the test is not stable and requires more data.
• Confidence Interval—A small interval is better because it indicates a more reliable test.
• Recommended Variation vs. Winning Variation—A recommended variation indicates a variation that is currently performing well, but could still change. A winning variation is confirmed when all metrics are green.

Experimentation best practices

Experimentation is a powerful feature, and Brightspot recommends being mindful of the following best practices:

1. One experiment per asset— Running multiple experiments on the same page or component can interfere with results; however, you may give a role the ability to create and run multiple experiments simultaneously if desired.
2. Use a 1:1 variation split whenever possible—A 50/50 traffic split between control and variation helps you reach statistical significance faster and reduces sampling bias.
3. Estimate test duration upfront—Use the duration calculator to determine the minimum time your test should be run based on expected traffic and conversion goals.
4. Test during normal traffic patterns—This will reduce skewed data that may occur when running tests during atypical periods like holidays, product launches, or major announcements.
5. Avoid mid-test changes—Modifying a variation or experiment while it is running can invalidate your data. If a major change is needed, stop the experiment and launch a new one.
6. Test one variable at a time—To best understand what is driving results, isolate your changes. Testing multiple elements at once makes it difficult to attribute impact to a specific change.