Add A/B testing for email design #633

totten · 2024-02-02T07:11:16Z

Overview

Expands support for A/B testing.

~~This is currently marked "Draft" because there are a couple TODOs (noted at bottom of PR). However, it should work well enough that one can do some testing.~~ (Resolved)

ping @kurund

Before

You may add a second "Subject" field, which enables A/B testing. (By default, it copies your current subject.)

After

You may add a second "Subject" field, which enables A/B testing. (By default, it copies your current subject.)
You may add a second "Design" section, which enables A/B testing. (By default, it copies your current design.)

After doing this, you will see "Design (A)" and "Design (B)".
It is possible to mix/match testing at the "Subject"-level and the "Design"-level. The GUI provides separate options for enabling/disabling each of these. These should be working in both "Single Page" and "Wizard" layouts.
(Aside: I moved the "A/B testing" percentage-sliders toward the end of the "New Mailing" screen. When there are multiple ways to enable A/B testing, it's a bit confusing if the sliders popup earlier in the UI. The new placement ensures that the sliders appear you've made all the decisions about whether to test subject and/or design.)

Technical Details

Within a Mailing record, there is a section template_options.variants[0,1]. The existing AbDemux component reads this property and generates "A" and "B" mailings (if present). This is how the subject-testing works. This PR follows the same path for testing two values of body_html. The existing AbDemuxTest already has scenarios for multiple body_html variants.

Here are a few examples of how the content ought to look:

Example: JSON for single subject and single design

{
  id: 123,
  subject: "Apples are the best fruit",
  body_html: "<html>...yummy...</html>",
  template_options: {
    mosaicoTemplate: "basic-versafix-1",
    mosaicoMetadata: "{...}",
    mosaicoContent: "{...yummy...}"
  }
}

Example: JSON for dual subject and dual design

{
  id: 123,
  subject: "PLACEHOLDER",
  body_html: "PLACEHOLDER",
  template_options: {
    variants: [
      {
        subject: "Apples are the best fruit",
        body_html: "<html>...yummy apple...</html>",
        mosaicoTemplate: "basic-versafix-1",
        mosaicoMetadata: "{...}",
        mosaicoContent: "{...yummy apple...}"
      },
      {
        subject: "Bananas are the best fruit",
        body_html: "<html>...yummy banana...</html>",
        mosaicoTemplate: "basic-versafix-1",
        mosaicoMetadata: "{...}",
        mosaicoContent: "{...yummy banana...}"
      }

    ]
  }
}

In comparing the examples, observe:

If the subject is tested, then we move subject into mailing.template_options.variants.[0,1].subject.
If the body_html (w/mosaicoContent, etc) is tested, then we move the body_html (w/mosaicoContent, etc) into mailing.template_options.variants.[0,1].body_html.

Most of the work in developing this PR has been about updating web UI to consistently populate+cleanup the data in template_options.variants (based on whether the user activates subject and/or body_html variations).

TODO

When you use editor, there are options to "Preview" (in browser) or "Send test" (via email). This is basically working for "Send test" (but not stress-tested yet). However, the "Preview" is not yet working.
~~Submit a mailing and try out the full flow of two blasts (followed by third/final blast).~~

When we add support for "A/B" testing of different message-bodies, there will be more ways to enable "A/B" testing. We may not know on the first screen whether "A/B" is active. This moves the "A/B" distribution options to the end -- at which point, we will know for sure whether "A/B" testing is active.

…on()

…get (`mailing`)

…tProp(), deleteProp()

…ne you are editing

Do you want or expect this button to send one message or two messages? It depends. * Heretofore, with subject-based A/B, the user intent was a bit ambiguous -- but the imperative was to send both. The user needs *some* way to send each A/B condition -- but there's only one button for sending. So that button sends both. * Now, with design-based A/B, you might reasonably infer user-intent. When editing "Design (A)", you send a test for "Design (A)". When editing "Design (B)", you send a test for "Design (B)". You can navigate back+forth to send whichever test you want. * But OTOH, the user now has a choice. They may do subject-based, design-based, or both. So: * If you always send the one being edited, then it's good for design-based. * If you always send both, then it's good for subject-based. * If you send one-or-both dynamically, then it's inconsistent. * IMHO, whether subject-based or design-based, it's still not bad to send both. When you go to read the messages in your email client, you can quickly compare the latest rendering of each. * The main thing is to be clear about what's happening.

totten · 2024-02-02T23:49:17Z

OK, I fixed the "Preview" issue and clarified the "Send test email" behavior. A trial-run with three deliveries (A+B+Winner) went fine.

It should be good for review/testing.

seamuslee001 · 2024-02-03T05:14:47Z

@johntwyman you were saying :)

kurund · 2024-02-23T10:07:41Z

@totten

Thank you for the work.

A/B version for the message seems be working as expected. However, I don't see an option for selecting A/B for From. Am I missing something?

mattwire · 2024-03-01T12:31:46Z

Merging based on discussion with @totten

totten · 2024-03-01T14:45:33Z

Thanks @kurund. I hadn't realized the From was also a target. It shouldn't be big a change (probably a few hours; a lot easier than the template body/metadata). So we'll go at that in a follow-up PR.

kurund · 2024-03-04T06:14:09Z

@totten

Ok thanks.

totten added 8 commits February 1, 2024 20:34

(NFC) bootstrap-wizard.html - Fix misaligned whitespace

b360910

(REF) Extract helper crmMosaicoVariants

592b2cb

crmMosaicoVariants - Handle mosaico-style fields

45f0361

crmMosaicoBlockDesign - Explicitly pass-through scope.attachments

4618315

(REF) Move mosaicoCtrl into crm-mailing-block-design

16390ce

(REF) crmMosaicoBlockDesign - Extract methods hasMarkup(), hasSelecti…

a27c3b4

…on()

(REF) crmMosaicoBlockDesign - Simplify signatures. Allow implicit tar…

11ccb44

…get (`mailing`)

totten force-pushed the 2.x-ab-body2 branch from fdee9ac to b957e11 Compare February 2, 2024 07:17

totten added 5 commits February 2, 2024 15:28

crmMosaicoBlockDesign - Accept variant. Extract methods getProp(), se…

0db9fdd

…tProp(), deleteProp()

crmMosaicoBlockMailing - Fix guard on help message

ed7bf0f

Update main layout to allow A/B testing for the "Design"

9a8f5c3

Preview - When you have two designs (A/B), show the preview for the o…

d861d12

…ne you are editing

totten force-pushed the 2.x-ab-body2 branch from b957e11 to 43febe0 Compare February 2, 2024 23:29

totten marked this pull request as ready for review February 2, 2024 23:45

BeccaTregenna mentioned this pull request Feb 5, 2024

Enhance A/B testing for "From" lines #566

Closed

mattwire merged commit 54db52e into veda-consulting-company:2.x Mar 1, 2024
1 check passed

totten deleted the 2.x-ab-body2 branch March 1, 2024 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add A/B testing for email design #633

Add A/B testing for email design #633

totten commented Feb 2, 2024 •

edited

Loading

totten commented Feb 2, 2024

seamuslee001 commented Feb 3, 2024

kurund commented Feb 23, 2024

mattwire commented Mar 1, 2024

totten commented Mar 1, 2024

kurund commented Mar 4, 2024

Add A/B testing for email design #633

Add A/B testing for email design #633

Conversation

totten commented Feb 2, 2024 • edited Loading

Overview

Before

After

Technical Details

TODO

totten commented Feb 2, 2024

seamuslee001 commented Feb 3, 2024

kurund commented Feb 23, 2024

mattwire commented Mar 1, 2024

totten commented Mar 1, 2024

kurund commented Mar 4, 2024

totten commented Feb 2, 2024 •

edited

Loading