A Deep Dive into Sitecore Personalization Effect Tracking

Everyone wants to assume that personalizing their website produces happy outcomes for their audiences, but how can you be sure? Sitecore 8 introduced the concept of Personalization Tests which provide a statistically sound way to answer this exact question during a set testing period.

However, what if you’d like to track the ongoing effect of your personalized content without an end date? This is what Personalization Effect Tracking, a feature introduced in Sitecore 9, can achieve. In our experience, it's less understood than its close cousin, Personalization Tests, so we’ll dig in and show you how it works.

Defining Personalization Effect Tracking

Personalization Effect Tracking works by running an ongoing Multivariate Test on your personalized component. By default, 10% of users exposed to the component in question are entered into this test and either shown the personalized experience or the “Default” experience as defined by the “personalize a component” dialog. The trailing engagement value is then compared for these two populations to demonstrate the increase or decrease in engagement value generated per visit.

Enabling Personalization Effect Tracking

To enable personalization effect tracking, click the checkbox in the “personalize a component” dialog that says, “Track effect of personalization on this component.” This checkbox is checked by default which brings us to an important call out: You may be running Personalization Effect Tracking already without knowing it. This may be a pleasant surprise for some but could lead to problems for others who expect their personalized experience to appear for 100% of users. Remember, 10% of users will be entered into a test and some of those users will see the “Default” experience no matter what.

Note: You can’t run a Personalization Test and Effect Tracking at the same time.

An Example of Effect Tracking

Here’s an example demonstrating how Effect Tracking works and how you can see where the results are recorded against contacts in Sitecore Experience Database (xDB).

First, we’ll create personalized conditions on a component. We set the personalization condition to “where true (actions always execute)” so that it always evaluates. That way, we can confirm that if a user sees the default condition, it was because they were entered into a test.

After publishing this change, the next step involves a lot of incognito windows. Because MV Tests are sticky to the user’s cookie, we need to start and stop quite a few new sessions before we find a lucky user who triggers that 10% threshold and sees the default experience. In this example, we started about 15 sessions before experiencing this scenario.

How can you confirm that your contact has taken part in an Effect Tracking test? It’s stored against their Experience Profile in Sitecore as an “MV Test Triggered” event.

As you can see above, xDB recorded that the user received a personalized experience and was placed into an MV Test on the /africa page. In this instance, the user received the “Default” experience rather than our personalization condition set to “where true (actions always execute)”. What’s notable is that an “MV Test Triggered” event was recorded even though we weren’t running a Personalization Test or a Multivariate Test on this page. The event was produced by our Effect Tracking.

After receiving the “Default” experience, this user went on to fill out the ‘Contact Us’ form which generated 100 engagement value points. You can view the results of Effect Tracking by clicking on the “Personalization” dialog from Experience Editor on any page where components are personalized.

In the results above, the personalized condition demonstrated 100 less engagement value than the default experience. In reality, the default condition yielded 100 more engagement value, but Sitecore reports the results as compared to a baseline of “0” which explain the results above.

Know that Effect Tracking results show far less detail than the results you can expect to see with a Personalization Test. It’s meant to provide a quick way to evaluate personalization performance without the traffic or time requirements of a Personalization Test. This is convenient but carries some risk as statistical significance of the results is not evaluated.

What’s Stored in xDB?

For advanced use cases, it may be helpful to know exactly what data is stored in xDB when a user is placed in a Personalization Effect Tracking test. Below we see the xDB event data from the user on the left who received the “Default” condition and the user on the right who was placed in a test and saw the “where true (actions always execute)” condition.

Within this data, there are a few interesting takeaways:

• Sitecore stores which personalization variant the user saw in the ‘PersonalizedEvent’ data, not the ‘MVTestTriggered’ data
• The ‘MVTestTriggered’ data stores the eligible rules available
• What’s missing are ruleset or condition names. This data is stored in the Final Rendering data on the page itself.

This data may be useful if anyone wished to “play back” their testing data and compute the results themselves. Sitecore will reset the Effect Tracking results if any changes are made to the page, so it’s likely that an organization may wish to hang on to past data points when only slight changes are made to a page.

Conclusion

If you want to know whether your Sitecore personalization is resonating with audiences, there are a couple of methods you can rely on. You can use Personalization Tests, which provide statistically sound measurements of personalization success during a specific period. But if you want to know how your personalized content is performing over an extended period, you can leverage Sitecore's Personalization Effect Tracking. It's important to note that you can't run both at the same time, so make sure you know which one is right for you before you activate Personalization Effect Tracking. Hopefully by providing you with a deep dive into what Personalization Effect Tracking is, how to turn it on, and an example of how to use it, you’ll better understand this useful feature introduced in Sitecore 9.

If you have additional questions about Personalization Effect Tracking, and want help activating it or interpreting its results, be sure to reach out to Velir and inquire about our Data Integration and Activation services.

You can also watch this video from our Public Interface series, where we talk about Data Integration and Activation.