User based and Cross device tracking

Problem

It's quite common today for users to use the same web app across different devices. Users might create an account on the desktop and do a few things. Then login again from their mobile or tablet and continue from where they left, or vice versa.

Challenge

This creates a challenge for A/B testing for two main reasons:

1. With an A/B test running, the same user might get a completely different experience when they switch between devices. This is confusing and potentially frustrating for users.
2. Tracking conversions is inaccurate. A user might use variation A on one device, but then convert on another device where variation B is running. The conversion will therefore be wrongly attributed.

user-based or cross-device tracking can solve both those problems, as long as the user can be uniquely identified across all devices. This isn't particularly difficult given that most web apps require logging-in to use the same account.

Method

With this in mind, Alephbet now supports user-based experiments. To do this, Alephbet does this:

1. Gets a user identifier (user_id, email, any identifier that uniquely identifies the user)
2. Uses this identifier to:

• pick a variant that is psuedo-random, but consistent for the user_id. This is done with deterministic assignment through hashing, instead of completely random. See planout
• generate a pseudo-uuid for goals. When goals are tracked, we generate a pseudo-uuid for the user and experiment, so even if the user completes a goal across different devices, it will only be counted once

Limitations

There are a couple of limitations with this approach:

1. Some experiments are designed for visitors, before they register. This approach won't work in those cases. Experiments that employ user-based tracking must target only already-identified users.
2. Not all backends support uuids. Gimel and Lamed do. And so does Keen.io. Google Analytics internally counts unique events, so I'm not sure if it can work with a custom uuid(?)

Usage

If you're using Gimel, Lamed, or keen.io - the build-in tracking adapter supports user-based experiments out of the box (as of v0.15.0).

Simply pass a user_id to your experiment, e.g.

import {Experiment} from "alephbet"

const button_color_experiment = new Experiment({
  name: 'button color',
  user_id: get_user_id(),  // pass over the unique user id bound to this experiment
  trigger: function() {
    // do not trigger this expeirment without a user_id
    return get_user_id() && other_condition();
  },
  variants: { // ...
  }
});

Caveats and special usage notes

• Use only with tracking backends that support uuids (see above).
• Do not mix visitors into your user-based experiments, or it can lead to unreliable results.

It's easiest to explain why with a few examples

Example 1: mixing visitors and logged-in users

import {Experiment, Goal} from "alephbet"

const experiment = new Experiment({
  name: 'my experiment',
  user_id: get_user_id(),  // pass over the unique user id bound to this experiment
});

const goal = new Goal('clicked on button');
experiment.add_goal(goal);

In this example, there's no trigger defined, so all users (visitors and logged-in) will participate in the experiment. On the same device, this won't cause a problem. Only when someone switches devices.

We have a visitor who went on the page. The experiment started, and we track the participate event on the backend. We only track it once on this device. Even after the user logs-in. No problems there.

As soon as the user switches to a different device, we track the participate event again (whether they're logged-in or not). The same goes for goals.

To resolve this, make sure you use a trigger function that checks the user_id, or wrap your whole experiment inside a conditional. Don't start it if user_id isn't present.

Example 2: registered user who logs out and continues to use your website

import {Experiment, Goal} from "alephbet"

const experiment = new Experiment({
  name: 'my experiment',
  trigger: function() { return get_user_id(); },
  user_id: get_user_id(),  // pass over the unique user id bound to this experiment
});

const goal = new Goal('clicked on button');
experiment.add_goal(goal);

In this example, the experiment trigger checks that we have a user_id before we assign the user to the experiment. So new visitors will not start the experiment.

Our user is logged-in however, so the experiment will start, and the participate event will be recorded for our user. Our user then logs out of their account on the same device. get_user_id() will return an undefined.

Since the user already participates in the experiment, the goals will continue to get tracked. If the (now logged-out) user clicks on the button, the goal will be marked as complete and sent to the backend.

Not the end of the world, since it's still the same person.

The problem can arise when the same user logs into another device. They will again trigger the experiment on the second device. Our backend will not track the participate event again (which is good). But if the user clicks again on the button, on their 2nd device, another goal will be recorded as completed. This is undesirable and will skew our results.

To resolve this, you can either wrap the whole experiment inside a condition, or at the very least make sure goals are not assigned to experiments unless the user_id is present. e.g.

if (get_user_id()) { experiment.add_goal(goal); }

The safest way to resolve it is to always wrap your experiments and goals inside a conditional, and check whether you have a valid user_id

if (get_user_id()) {
  experiment = new AlephBet.Experiment(...);
  goal = new AlephBet.Goal(...);
  experiment.add_goal(goal);
  ...
}