Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ads enabled + installation time P3A metric #28109

Closed
DJAndries opened this issue Jan 27, 2023 · 2 comments · Fixed by brave/brave-core#16897
Closed

Add ads enabled + installation time P3A metric #28109

DJAndries opened this issue Jan 27, 2023 · 2 comments · Fixed by brave/brave-core#16897
Assignees
@DJAndries
Labels
feature/ads features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop QA Pass - Android ARM QA Pass-Win64 QA/Yes release-notes/exclude
Milestone
1.51.x - Release

Comments

@DJAndries
Copy link
Collaborator

DJAndries commented Jan 27, 2023 •
edited
Loading

Add a one-time P3A metric reporting the number of hours between the installation time and the time when ads are enabled.

Brave.Rewards.EnabledInstallationTime

The bucket values will be as follows:

  1. Less than an hour
  2. 1 hour or more, less than 12 hours
  3. 12 hours or more, less than 24 hours
  4. 24 hours or more, less than 72 hours
  5. 72 hours or more

The P3A metric will be sent only once, but will be reported via the daily cadence.

Support for ephemeral/one-time P3A metrics is also part of this issue. A new flat set in metric_names.h will be added to enumerate ephemeral metrics.
@DJAndries DJAndries added OS/Android Fixes related to Android browser functionality OS/Desktop labels Jan 27, 2023
@DJAndries DJAndries changed the title Add ads enabled + installation to Add ads enabled + installation time P3A metric Jan 27, 2023
@DJAndries DJAndries self-assigned this Jan 27, 2023
@DJAndries DJAndries mentioned this issue Jan 27, 2023
Merged
25 tasks
@brave-builds brave-builds added this to the 1.51.x - Nightly milestone Mar 11, 2023
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Apr 6, 2023 •
edited
Loading

Verification PASSED on

Brave | 1.51.79 Chromium: 112.0.5615.49 (Official Build) beta (64-bit)
-- | --
Revision | bd2a7bcb881c11e8cfe3078709382934e3916914-refs/branch-heads/5615@{#936}
OS | Windows 10 Version 21H2 (Build 19044.2728)
Case 1: Less than an hour = 0_FAILED_#29624
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Enable rewards/Ads
  5. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 0 as expected
  6. Open charles and wait until the metrics is sent over the network via Charles
  7. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 0 as expected
  8. Reload the brave:/local-state file and look for the metrics flag "sent": true ("true" indicates the metrics value is sent/uploaded to the server)
p3a-json.brave.com brave://local-state
image image
  1. Close the browser
  2. Set the sys date ahead by one month
  3. Relaunch the browser
  4. the metrics Brave.Rewards.EnabledInstallationTime response value is 0 which is NOT as expected as per the test plan step 3 Add support for ephemeral P3A metrics, add ads enabled installation time metric brave-core#16897 (comment). Logged and encountered Ads-enabled metrics is shown in the local state file even after a month of installation/ads enabled #29624
Case 2: 1 hour or more, less than 12 hours_PASSED
Case 1: Ads is enabled after one hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by one hour
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
image image
Case 2: Ads is enabled after 6 hours before 12 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 6 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
image image
Case 3: Ads is enabled one minute (11:59) before 12 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 11 hours 59 minutes
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
image image
Case 3: 12 hours or more, less than 24 hours_PASSED
Case 3.1: Ads is enabled extactly at 12th hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 12 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image image
Case 3.2: Ads is enabled after 12+ hour before 24 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 12+ hour
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image image
Case 3.3: Ads is enabled one minute before (23:59) 24 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 23 hours 59 minutes
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image image
Case 4: 24 hours or more, less than 72 hours_PASSED
Case 4.1: Ads is enabled exactly at 24th hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 24 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 3 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 3 as expected
p3a-json.brave.com brave://local-state
image image
Case 4.2: Ads is enabled after 24+ hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 24+ hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 3 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 3 as expected
p3a-json.brave.com brave://local-state
image image
Case 4.3: Ads is enabled before 72 hours_FAILED_#29653
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 48+ hours but before 72 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 4 instead of 3
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 4 instead of 3
p3a-json.brave.com brave://local-state
image image
Case 5: 72 hours or more_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 72+ hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 4 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 4 as expected
p3a-json.brave.com brave://local-state
image image

@GeetaSarvadnya GeetaSarvadnya added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Apr 6, 2023
@GeetaSarvadnya GeetaSarvadnya mentioned this issue Apr 11, 2023
Open
@GeetaSarvadnya GeetaSarvadnya added QA Pass-Win64 and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Apr 11, 2023
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Apr 11, 2023 •
edited
Loading

Verification PASSED on Samsung Galaxy version 12 running Bravemonoarm64.apk_1.51.79

Case 1: Less than an hour = 0_FAILED_#29624
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Enable rewards/Ads
  5. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 0 as expected
  6. Open charles and wait until the metrics is sent over the network via Charles
  7. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 0 as expected
  8. Reload the brave:/local-state file and look for the metrics flag "sent": true ("true" indicates the metrics value is sent/uploaded to the server)
p3a-json.brave.com brave://local-state
image Screenshot_20230412-122615_Brave - Beta
  1. Close the browser
  2. Set the sys date ahead by one month
  3. Relaunch the browser
  4. the metrics Brave.Rewards.EnabledInstallationTime response value is 0 as expected
Case 2: 1 hour or more, less than 12 hours_PASSED
Case 1: Ads is enabled after one hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by one hour
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
Screenshot_20230412-134032_Brave - Beta image
Case 2: Ads is enabled after 6 hours before 12 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 6 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230412-203617_Brave - Beta
Case 3: Ads is enabled one minute (11:59) before 12 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 11 hours 59 minutes
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 1 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 1 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230412-235834_Brave - Beta
Case 3: 12 hours or more, less than 24 hours_PASSED
Case 3.1: Ads is enabled extactly at 12th hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 12 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230413-144730_Brave - Beta
Case 3.2: Ads is enabled after 12+ hour before 24 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 12+ hour
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230413-144730_Brave - Beta
Case 3.3: Ads is enabled one minute before (23:59) 24 hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 23 hours 59 minutes
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 2 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 2 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230413-144730_Brave - Beta
Case 4: 24 hours or more, less than 72 hours_PASSED
Case 4.1: Ads is enabled exactly at 24th hour_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 24 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 3 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 3 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230414-163004_Brave - Beta
Case 4.2: Ads is enabled after 24+ hours_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 24+ hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 3 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 3 as expected
p3a-json.brave.com brave://local-state
image Screenshot_20230414-163004_Brave - Beta
Case 4.3: Ads is enabled before 72 hours_FAILED_#29653
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 48 hours but before 72 hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 4 instead of 3
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 4 instead of 3
p3a-json.brave.com brave://local-state
image image
Case 5: 72 hours or more_PASSED
  1. Clean profile 1.51.x
  2. Open brave://local-state file
  3. Confirm that the metrics Brave.Rewards.EnabledInstallationTime is not shown in the local state file
  4. Close the browser, set the sys clock ahead by 72+ hours
  5. Relaunch browser and enable rewards/Ads
  6. Reload brave://local-state file and ensure that the metrics Brave.Rewards.EnabledInstallationTime response value is 4 as expected
  7. Open charles and wait until the metrics is sent over the network via Charles
  8. Confirmed that the metrics is listed under the host p3a-json.brave.com over the network and the metrics_value is 4 as expected
p3a-json.brave.com brave://local-state
Screenshot_20230512-144100_Brave - Beta image

@GeetaSarvadnya GeetaSarvadnya mentioned this issue Apr 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DJAndries DJAndries

Labels
feature/ads features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop QA Pass - Android ARM QA Pass-Win64 QA/Yes release-notes/exclude
Projects
None yet
Milestone
1.51.x - Release
Development

Successfully merging a pull request may close this issue.

Add support for ephemeral P3A metrics, add ads enabled installation time metric brave/brave-core
3 participants
@DJAndries @GeetaSarvadnya @brave-builds