How to set up a variant page for A/B testing

Overview

The different types of A/B testing can be generally categorized into the following groups:

• Variant A/B test
• Multivariate A/B test
• Redirect A/B test
• Multi-page funnel A/B test

In this tutorial, we will be setting up rules in KurocoEdge for one of the most used and well known A/B test that is the "Variant A/B test".

Benefit of using KurocoEdge for A/B testing

KurocoEdge uses the latest Edge computing technology that allows users to modify the webpage on the Edge, on-demand.

The traditional method of dynamically injecting Javascript in the browser on the client side may not be ideal as it may require developers to modify the source code by incorporating an A/B testing library, furthermore, the A/B test scripts are executed on the client side which leads to slower performance as the script will need to be executed on every client machine and adding a new element or modifying an existing element's height or width that would shift the overall layout of the page may negatively affect the Google's Core Web Vital's Cumulative Layout Shift (CLS) performance score.

In comparison, KurocoEdge processes all changes on the Edge server, which provides user with a smooth experience and developers do not need to worry about affecting Google's Core Web Vital's performance such as Cumulative Layout Shift (CLS) negatively even if new elements are added or existing element's height or width is modified, as the changes do not happen on the client side using KurocoEdge. The changes are all processed on the Edge server and the result is delivered to the user directly.

KurocoEdge is ideal for users who do not want to touch, are not allowed to touch the source code, or simply do not want to install libraries/modify the source code. KurocoEdge is a plug and play tool and allows user to modify their website extensively with no risks as any changes can be previewed before actual deployment, disabled, or rolled back instantaneously.

What You'll Learn

You can set up a URL redirect in KurocoEdge by following the steps below:

• Overall flow
• Create a rule to check if show_variant cookie exists
• Create a rule to generate random value for show_variant cookie
• Create a rule to change the banner size
• Check the AB variant display and Cumulative Layout Shift performance

Before you start

This tutorial assumes you understand how to perform an A/B testing, from research, hypothesis formulation, A/B test setup, testing, result analysis, and drawing conclusions.
This document will guide you through how to set up a variant page for A/B testing using KurocoEdge.

note

This document does not include information about A/B test research, hypothesis formulation, result analysis, and how to draw conclusions.

We will be setting up rules that modify cookies on the user's browser which will require the paid plan of KurocoEdge. For more information, refer to the plan pricing page.

Overall flow

To determine whether to display the control page or the variant page to users visiting the website, the decision is made based on a cookie named show_variant, which is added and checked using KurocoEdge. The value of this cookie is set to either true or false.

First, we will create a KurocoEdge rule that check the show_variant cookie exists. Next, if the show_variant cookie does not exist, a rule is added to randomly assign either true or false to show_variant and add it as a cookie. If the cookie already exists, it means the user has previously visited the website and the value for show_variant has already been assigned, so the process is skipped.

Lastly, we will create the KurocoEdge rule for the variant page's modification.

Create a rule to check if show_variant cookie exists

First, we will create a new rule to initialize the show_variant_cookie variable and capture the show_variant cookie's value in that variable.
This allows determining whether a user has visited the website once or not based on whether the variable defined as show_variant_cookie is null or not.

Navigate to the deployment editor page and click the [+Add] button in the 'Rules list' section to create a new rule:

In the 'General' tab, set the [Execution priority] to 15.

By setting the priority to less than 15 in the subsequent rules, the rule to check the existence of this cookie will be executed first.

info

For more information on how execution priority works in KurocoEdge, refer to the documentation here.

Go to the 'Request' tab and expand the [Condition / Variable] section then click the [Add Field] button:

Create the following 2 fields. The configured fields within "Conditions/Variables" will be executed in order from top to bottom.

Create the following 2 fields:

• First, we will initialize the show_variant_cookie variable using the Static variables field. This is to ensure that the variable is always created, even if the show_variant cookie does not exists.
• Second, we will use the Cookie field and capture the show_variant cookie's value in the show_variant_cookie variable.

Field	Settings
Field settings	Static variables
Variable name	show_variant_cookie
Value	(Leave blank)

*Initialize the show_variant_cookie variable. The variables defined here will be available for use in the subsequent items.

Field	Settings
Field settings	Cookie
Key	show_variant
Value	(?P<show_variant_cookie>.*)

*capture the show_variant cookie's value in the show_variant_cookie variable.

info

To learn more about capturing variables, refer to this document Using variables in KurocoEdge - Captured Variables

Once you have completed the configurations above, click the [Update] button at the bottom left of the page to save the changes.

Create a rule to generate random value for show_variant cookie

info

For this tutorial, I've prepared an API endpoint beforehand using KurocoCMS's custom function API endpoint feature to return a randomize result of true or false, which will be used in this A/B test to determine if the user will see the control or variant page of the A/B test.

The response body of the API I've prepared beforehand returns data in the following format:
{"errors": [], "messages": [], "data": "true"}

tip

It is possible to use any external API endpoint of your choice to generate the random data. Note that the domain of the external API will need to be added and authorized via the backend list.

At the time of writing this document, November 1st 2023, KurocoEdge only supports string format when capturing variables in the API's response body, therefore, ensure the data is returned in string format.

Start off by selecting a draft deployment and click the [+Add] button in the 'Rules list' section to create a new rule:

In the 'General' tab, set the [Execution priority] to 10. This will ensure that this rule is executed after the previous rule that checks for the show_variant cookie value in the user's browser.

First, we will need to add a field to check if the cookie exists in the user's browser by checking if the variable show_variant_cookie is empty or not.
To do this, click on rule's the 'Request' tab, then click the [Add Action] button to create a new action.
Expand the [Condition / Variable] section and click the [Add Field] button:

Field	Settings
Field settings	Match value
Expression	{$show_variant_cookie}
Value pattern	^$

note

The show_variant_cookie variable will contain the value of trueorfalse if the user has visited the website at least once, and the show_variant_cookie will be null for users who is visiting the website for the first time.

Next, we will need to call the randomizer API endpoint in KurocoEdge to get the randomized result.
Add a new field under the [Condition / Variable] section.
Select the [API variable] option in the dropdown selection, then click [Add API] and click [Edit] to open the editor.
Fill in the following fields in the editor:

Field	Settings
Field settings	API Variables
API endpoint	https://your-domain.com/randomizer
Cache	No cache
First byte timeout	15 sec(s)
Method	GET
Expected variable(s)	Variable name in API response: data Rename: show_variant_cookie

In the above configuration, the API endpoint returns a variable called data which is captured in KurocoEdge and the variable saved as show_variant_cookie.

Ensure that the API variable field is sorted at the bottom:

The items in the [Condition / Variable] section are executed in order from the top to bottom, and if the result of a condition is false, then all other condition/variables fields below it will be skipped.
With the above rule setup, show_variant_cookie variable with only be populated with the random value of true or false if the show_variant_cookie variable is empty (i.e. the user has never visited the website and no cookies have been created).

Next, we need to set a cookie in the user's browser that will be used to determine if the variant page should be displayed to them.
Navigate to the 'Response' tab, then click the [Add Action] button to create a new action.
Expand the 'Execution' section, click the [Add Field] button, and set up the following configuration:

Field	Settings
Field settings	Cookie
Key	show_variant
Value	{$show_variant_cookie}
Secure	Enable

Once you have completed the configurations above, click the [Update] button at the bottom left of the page to save the changes.

The above configuration allows setting a rule for the show_variant cookie to assign its existing value if it already exists, or, if it doesn't exist, to assign a randomly retrieved true or false to show_variant cookie.

Test the rule that you have just set up by clicking the [Preview] button at the top right of the page.

You should see that the show_variant cookie is added in your browser and assigned with the randomized the value of true or false based on the result from the randomizer API.

Refresh the page multiple times and verify that the cookie's value is not being overridden by the value returned by the randomizer API endpoint if the show_variant cookie already exists in the user's browser.

Create a rule to change the banner size

Now that we've configured the rules to generate the A/B test cookie, we will need to create a new rule to modify the banner size of the webpage for the variant page.

For this exercise, we will be using the default backend that is provided by default in your KurocoEdge account: https://astro-shopify-diverta.vercel.app/.

The banner size is modified in this practice to verify that changes made by KurocoEdge does not negatively affect the Cumulative Layout Shift scoring of Google's Core Web Vitals.

Navigate to the deployment editor page, click the [+Add] button in the 'Rules list' section to create a new rule:

In the 'General' tab, set the [Execution priority] to 5. This rule will have the lowest execution priority out of all the 3 rules.
The first 2 rules will run first to retrieve or generate the cookie data, then this third rule will execute the HTML modification if the variant page is to be displayed to the user.

To determine if the variant page should be displayed, we need to check if the show_variant_cookie variable captured from the user's browser cookie has the value true.
Go to the 'Response' tab, expand the [Condition / Variable] section and add the following field:

Field	Settings
Field settings	Match value
Expression	{$show_variant_cookie}
Value pattern	true

In the above configuration, we've added the logic to determine the display of variant page to the user.
Next, we will need to configure the field settings to modify webpage element for the variant page.

First, we will need to identify the element's CSS selector. We can do this by right clicking the top banner and clicking [Inspect] to open the developer tool.

In the developer tool, right click the element and select [Copy > Copy selector].

Go back to the KurocoEdge admin panel's rule editor page.
Go to the 'Response' tab, expand the 'Execution' section and add the following field:

Tab	Field	Settings
	Field settings	HTML editor
General	CSS selector	body > div
Tag attributes	Tag attributes	Operation: Set Attribute Name: style Value: height:150px;font-size:30px

Once you have completed the configurations above, click the [Update] button at the bottom left of the page to save the changes.

Check the AB variant display and Cumulative Layout Shift performance

Verify that the banner height and font size is now larger.

Finally, we will test verify Google's Core Web Vitals scoring.
If you are using the Google Chrome browser, you can use the Lighthouse tool in the developer tool to test the performance.

As you can see in the above analytics, the Cumulative Layout Shift has not been affected even though the height of the banner element on the page has increased significantly. This is because KurocoEdge makes all the modification on the Edge server instead of injecting scripts on the user's browser, which provides better performance in terms of delivery speed and SEO.

Summary

In this tutorial, we went through how to set up a rule to determine the display of the variant page to the user. Followed by a rule to identify and the modification the variant page displayed to the user.