Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Brave Ads split testing #14613

Closed
moritzhaller opened this issue Mar 10, 2021 · 3 comments · Fixed by brave/brave-core#8216
Closed

Implement Brave Ads split testing #14613

moritzhaller opened this issue Mar 10, 2021 · 3 comments · Fixed by brave/brave-core#8216
Assignees
@moritzhaller
Labels
enhancement feature/ads OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass-Linux QA/Yes release-notes/exclude
Milestone
1.23.x - Release

Comments

@moritzhaller
Copy link

No description provided.

@moritzhaller moritzhaller self-assigned this Mar 10, 2021
@moritzhaller moritzhaller added enhancement feature/ads OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality priority/P3 The next thing for us to work on. It'll ride the trains. QA/Yes labels Mar 10, 2021
@moritzhaller moritzhaller changed the title A/B Testing for Advertisers Split-Testing for Advertisers Mar 11, 2021
@moritzhaller moritzhaller mentioned this issue Mar 11, 2021
Merged
24 tasks
@tmancey tmancey changed the title Split-Testing for Advertisers Implement Brave ads split testing Mar 12, 2021
@tmancey tmancey removed OS/Android Fixes related to Android browser functionality OS/iOS Fixes related to iOS browser functionality labels Mar 12, 2021
@tmancey tmancey changed the title Implement Brave ads split testing Implement Brave Ads split testing Mar 12, 2021
@moritzhaller moritzhaller added this to the 1.23.x - Beta milestone Mar 17, 2021
@btlechowski btlechowski added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Mar 24, 2021
@btlechowski
Copy link

Set as QA/Blocked - waiting for a custom catalog

@btlechowski btlechowski added QA/Blocked QA/In-Progress Indicates that QA is currently in progress for that particular issue and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue QA/Blocked labels Mar 24, 2021
@btlechowski
Copy link

Got the needed catalog, changing to QA/In-Progress

This was referenced Apr 7, 2021
Closed
Closed
@btlechowski btlechowski mentioned this issue Apr 7, 2021
Closed
@btlechowski
Copy link

Verification passed on

Brave 1.23.63 Chromium: 89.0.4389.114 (Official Build) beta (64-bit)
Revision 1ea76e193b4fadb723bfea2a19a66c93a1bc0ca6-refs/branch-heads/4389@{#1616}
OS Ubuntu 18.04 LTS

Verified test plan from https://github.com/brave/internal/issues/762

Notification ad - AdvertiserSplitTestStudy is not defined in seed
FAIL - an ad is shown. Logged #15164

[22735:22735:0407/025832.503329:VERBOSE1:ad_notification_serving.cc(98)] Ad notification delivered:
  uuid: f59741bf-7883-45f3-a43a-7b6301ac2ad6
  creativeInstanceId: 54592ab6-5a8e-42e6-8e11-9e9c04e9eecb
  creativeSetId: c7e7a381-4c74-4dd9-9b6d-4edf9330454b
  campaignId: 0d01425b-c970-4a25-b2d0-ccf748cfaf6e
  segment: untargeted
  title: Ad - GroupB
  body: Ad - GroupB
  targetUrl: https://youtube.com

Notification ad - Group is not defined in AdvertiserSplitTestStudy in the seed
GroupA ads are filtered out

[26023:26023:0407/033100.308054:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group
[26023:26023:0407/033100.320437:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group

Notification ad - GroupA
GroupA ad is shown:

[21423:21423:0407/023736.005062:VERBOSE1:ad_notification_serving.cc(98)] Ad notification delivered:
  uuid: 35e1cdfb-86b9-4da5-81b6-eef46111dc10
  creativeInstanceId: a0369601-b46f-46aa-93e1-2c9bafda3c35
  creativeSetId: 5e96fecc-2926-4c72-bb76-4e745bf6b539
  campaignId: 0d01425b-c970-4a25-b2d0-ccf748cfaf6e
  segment: untargeted
  title: Ad - GroupA
  body: Ad - GroupA
  targetUrl: https://youtube.com

GroupB ad is filtered out:

[21423:21423:0407/023735.997289:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group

Logged #15162 for missing log message

Notification ad - GroupA - round robin
Ad is not shown after round robin. Logged #15166

[5503:5503:0407/051021.109751:VERBOSE1:ad_notification_serving.cc(158)] Serve ad for segments:
[5503:5503:0407/051021.109842:VERBOSE1:ad_notification_serving.cc(160)]   untargeted
[5503:5503:0407/051021.110644:VERBOSE1:eligible_ad_notifications.cc(70)] All advertisers have been shown, so round robin
[5503:5503:0407/051021.110748:VERBOSE1:client.cc(386)] Resetting seen advertisers
[5503:5503:0407/051021.113347:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group
[5503:5503:0407/051021.113495:VERBOSE1:ad_notification_serving.cc(174)] No eligible ads found for segments
[5503:5503:0407/051021.113617:VERBOSE1:ad_notification_serving.cc(189)] Serve ad for parent segments:
[5503:5503:0407/051021.113663:VERBOSE1:ad_notification_serving.cc(191)]   untargeted
[5503:5503:0407/051021.118112:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group
[5503:5503:0407/051021.118225:VERBOSE1:ad_notification_serving.cc(205)] No eligible ads found for parent segments
[5503:5503:0407/051021.118304:VERBOSE1:ad_notification_serving.cc(217)] Serve untargeted ad
[5503:5503:0407/051021.137394:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId c7e7a381-4c74-4dd9-9b6d-4edf9330454b excluded as not associated with advertiser split test group
[5503:5503:0407/051021.137533:VERBOSE1:ad_notification_serving.cc(232)] No eligible ads found for untargeted segment
[5503:5503:0407/051021.137659:VERBOSE1:ad_notification_serving.cc(233)] Ad notification not served: No eligible ads found
[5503:5503:0407/051021.137770:VERBOSE1:ad_notification_serving.cc(85)] Ad notification not delivered

Notification ad - GroupB
GroupB ad is shown:

[7111:7111:0407/043243.144858:VERBOSE1:ad_notification_serving.cc(98)] Ad notification delivered:
  uuid: 4e50095d-d864-4de2-8667-fd7706295d29
  creativeInstanceId: 54592ab6-5a8e-42e6-8e11-9e9c04e9eecb
  creativeSetId: c7e7a381-4c74-4dd9-9b6d-4edf9330454b
  campaignId: 0d01425b-c970-4a25-b2d0-ccf748cfaf6e
  segment: untargeted
  title: Ad - GroupB
  body: Ad - GroupB

GroupA ad is filtered out:

[7111:7111:0407/043243.140039:VERBOSE2:exclusion_rule_util.h(26)] creativeSetId 5e96fecc-2926-4c72-bb76-4e745bf6b539 excluded as not associated with advertiser split test group

Notification ad - Random group (50/50)
Verified GroupA and GroupB ads are shown

@btlechowski btlechowski added QA Pass-Linux and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Apr 7, 2021
@tmancey tmancey added this to Ads Jun 10, 2024
@tmancey tmancey moved this to Done in Ads Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@moritzhaller moritzhaller

Labels
enhancement feature/ads OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass-Linux QA/Yes release-notes/exclude
Projects
Ads
Archived in project
Milestone
1.23.x - Release
Development

Successfully merging a pull request may close this issue.

Add advertiser split testing for Brave ads brave/brave-core
3 participants
@tmancey @moritzhaller @btlechowski