VPN-5547: Implement subscription management telemetry #8838

MattLichtenstein · 2023-12-19T05:02:44Z

Description

Add impression telemetry for the subscription management view
Replace old manage account and manage subscription telemetry with new interaction format
Add new telemetry for navbar button presses (that records source view as extra key)
- Source views need to be added to the screen via Navigator::addView
- Source views need a telemetryScreenId property
- Left old navbar button press telemetry in tact as we transition to the new telemetry for more views
Add functional tests for subscription management view telemetry
Add unit tests for navbar telemetry

Reference

VPN-5547: Implement Telemetry
[Telemetry] Miro board

- Also add functional tests

MattLichtenstein · 2023-12-19T05:19:36Z

Request for data collection review form

What questions will you answer with this data?

As a Product Manager … I want to understand how often users on Monthly plans display interest in upgrading to an Annual plan by virtue of using the in-app Profile screen CTA.
As a Product Manager … I want to understand how often users who come from the in-app Profile screen CTA end up upgrading to an Annual Plan.

Why does Mozilla need to answer these questions? Are there benefits for users?
Do we need this information to address product or business requirements?

This data is necessary for the Mozilla VPN product team to understand the behavior of the subscription management and subscription upgrading experience and how to make that flow better for users.

What alternative methods did you consider to answer these questions?
Why were they not sufficient?

None

Can current instrumentation answer these questions?

No

List all proposed measurements and indicate the category of data collection for each
measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Name	Measurement Description	Data Collection Category	Tracking Bug
`impression.account_screen`	The user has viewed the subscription management screen	impression	https://mozilla-hub.atlassian.net/browse/VPN-5547
`interaction.edit_selected`	The user has clicked the "edit account" button	interaction	https://mozilla-hub.atlassian.net/browse/VPN-5547
`interaction.manage_subscription_selected`	The user has clicked the “manage subscription” button from the subscription management view	interaction	https://mozilla-hub.atlassian.net/browse/VPN-5547
`interaction.change_plan_selected`	The user has clicked the “change plan” button from the subscription management view (monthly web plan)	interaction	https://mozilla-hub.atlassian.net/browse/VPN-5547

Please provide a link to the documentation for this data collection which
describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

How long will this data be collected?

This collection will be collected permanently.
[email protected], [email protected] will be responsible for the permanent collections.

What populations will you measure?

All channels, countries, and locales. No filters.

If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences.

Please provide a general description of how you will analyze this data.

Dashboards that product managers and others will consult on a regular basis.

Where do you intend to share the results of your analysis?

Within the Mozilla VPN team and Security and Privacy Products team.

Is there a third-party tool (i.e. not Glean or Telemetry) that you
are proposing to use for this data collection?

No.

MattLichtenstein · 2023-12-19T05:24:31Z

tests/functional/testSubscription.js

+        const isMonthlyPlanTest = testCase === "monthly plan"
+
+        //Check if we are testing the monthly plan so we can mock the guardian response
+        beforeEach(async () => {


@brizental do you know of any good way to do this before each item in testCases instead of before each test (it)?

Maybe you want to use before instead of beforeEach ? That will be applied before all test cases inside a describe block. See: https://mochajs.org/#hooks

MattLichtenstein · 2023-12-19T05:25:50Z

tests/functional/testSubscription.js

+          assert.equal(subscriptionManagementScreenTelemetryId, manageSubscriptionSelectedEventsExtras.screen);
+        });
+
+        it('homeSelected interaction event is recorded with correct screen', async () => {


@brizental do you think it is worth testing the navbar telemetry with the appropriate screen extra key like I did here for every different potential screen? Or should we just test it once somewhere else?

I think since that is happening in the C++ code you can just test it in a unit test and skip this telemetry piece in functional tests completely. We can pair on that if you'd like to see how it would go.

Otherwise, I think it's also not a big deal to just leave it like this and no need to test for each screen. The main logic is tested here. Testing for each screen would just be testing that the telemetryScreenId of some screen was set correctly, but for that there are other test cases already.

brizental

Hey Matt, this is looking great! Question, I see there is instrumentation on the "Sign out" button spec'd in the Miro board but I don't see it here. Is it already there or mabe coming in a follow up PR?

src/frontend/navigator.cpp

tests/functional/testAddons.js

brizental · 2023-12-19T12:53:00Z

tests/functional/testSubscription.js

+        const isMonthlyPlanTest = testCase === "monthly plan"
+
+        //Check if we are testing the monthly plan so we can mock the guardian response
+        beforeEach(async () => {


Maybe you want to use before instead of beforeEach ? That will be applied before all test cases inside a describe block. See: https://mochajs.org/#hooks

tests/functional/testSubscription.js

brizental · 2023-12-19T12:54:35Z

tests/functional/testSubscription.js

+       });
+
+        it('accountScreen impression event is recorded', async () => {
+


nit: The newline at the start of each test is bothering me 😅

brizental · 2023-12-19T12:58:59Z

tests/functional/testSubscription.js

+          assert.equal(subscriptionManagementScreenTelemetryId, manageSubscriptionSelectedEventsExtras.screen);
+        });
+
+        it('homeSelected interaction event is recorded with correct screen', async () => {


I think since that is happening in the C++ code you can just test it in a unit test and skip this telemetry piece in functional tests completely. We can pair on that if you'd like to see how it would go.

Otherwise, I think it's also not a big deal to just leave it like this and no need to test for each screen. The main logic is tested here. Testing for each screen would just be testing that the telemetryScreenId of some screen was set correctly, but for that there are other test cases already.

brizental · 2023-12-19T15:15:21Z

src/telemetry/impression_metrics.yaml

+      - https://mozilla-hub.atlassian.net/browse/VPN-5547
+    data_reviews:
+      - https://github.com/mozilla-mobile/mozilla-vpn-client/pull/8838#issuecomment-1862135964


This needs to point to the data-review response. Same applies to all other places where this link is used.

Ahh good call. Will fix after it has been reviewed

travis79

Request for data collection review form
1. What questions will you answer with this data?


* As a Product Manager … I want to understand how often users on Monthly plans display interest in upgrading to an Annual plan by virtue of using the in-app Profile screen CTA.

* As a Product Manager … I want to understand how often users who come from the in-app Profile screen CTA end up upgrading to an Annual Plan.


2. Why does Mozilla need to answer these questions? Are there benefits for users?
   Do we need this information to address product or business requirements?


* This data is necessary for the Mozilla VPN product team to understand the behavior of the subscription management and subscription upgrading experience and how to make that flow better for users.


3. What alternative methods did you consider to answer these questions?
   Why were they not sufficient?


* None


4. Can current instrumentation answer these questions?


* No


5. List all proposed measurements and indicate the category of data collection for each
   measurement, using the Firefox data collection categories found on the Mozilla wiki.
Measurement Name Measurement Description Data Collection Category Tracking Bug
impression.account_screen The user has viewed the subscription management screen impression mozilla-hub.atlassian.net/browse/VPN-5547
interaction.edit_selected The user has clicked the "edit account" button interaction mozilla-hub.atlassian.net/browse/VPN-5547
interaction.manage_subscription_selected The user has clicked the “manage subscription” button from the subscription management view interaction mozilla-hub.atlassian.net/browse/VPN-5547
interaction.change_plan_selected The user has clicked the “change plan” button from the subscription management view (monthly web plan) interaction mozilla-hub.atlassian.net/browse/VPN-5547
6. Please provide a link to the documentation for this data collection which
   describes the ultimate data set in a public, complete, and accurate way.
This collection is Glean so is documented in the Glean Dictionary.
7. How long will this data be collected?
This collection will be collected permanently. [email protected], [email protected] will be responsible for the permanent collections.
8. What populations will you measure?
All channels, countries, and locales. No filters.
9. If this data collection is default on, what is the opt-out mechanism for users?
These collections are Glean. The opt-out can be found in the product's preferences.
10. Please provide a general description of how you will analyze this data.


* Dashboards that product managers and others will consult on a regular basis.


11. Where do you intend to share the results of your analysis?


* Within the Mozilla VPN team and Security and Privacy Products team.


12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
    are proposing to use for this data collection?
No.

Data Review

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the data preferences in the application settings.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Permanent collection to be monitored by [email protected] and vpn-team

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction data

Is the data collection request for default-on or default-off?

Default-on

Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

Is the data collection covered by the existing Firefox privacy notice?

Yes

Does the data collection use a third-party collection tool?

No

Result

data-review+

MattLichtenstein · 2023-12-19T20:26:48Z

I see there is instrumentation on the "Sign out" button spec'd in the Miro board but I don't see it here. Is it already there or mabe coming in a follow up PR?

@brizental Moving the "sign out" button from the settings menu to the subscription management view will come as part of VPN-5696: Implement new Settings menu & IA

brizental · 2023-12-21T15:37:20Z

src/frontend/navigator.cpp

+      // screens)
+      if (layers.length() < 1 || layers.last()
+                                     .m_layer->property("telemetryScreenId")
+                                     .toString()


Will this crash if the property is not there at all?

Just though of this, I think it's worth adding a test case for when the property is not there.

It will not. if the property doesn't exist, the QVariant will be invalid (aka it's userType will be QMetaType::UnknownType which is 0). performing QVariant::toString on a QVariant with an unknown meta type will return an empty string

brizental · 2023-12-21T15:38:24Z

src/frontend/navigator.cpp

+      }
+
+      QString telemetryScreenId =
+          layers.last().m_layer->property("telemetryScreenId").toString();


nit: Move this declaration before the conditional and use this variable instead of repeating layers.last().m_layer->property("telemetryScreenId").toString().

brizental · 2023-12-21T15:39:31Z

src/frontend/navigator.cpp

+              mozilla::glean::interaction::SettingsSelectedExtra{
+                  ._screen = telemetryScreenId,
+              });
+          break;


Should we add a default case here, just in case?

brizental · 2023-12-21T15:44:07Z

tests/unit_tests/testnavigator.h

+ private slots:
+  void init();
+
+  void navbar_button_telemetry();


nit:

Suggested change

void navbar_button_telemetry();

void testNavbarButtonTelemetry();

I know this is not standardized right now... So feel free to ignore.

MattLichtenstein added 5 commits December 18, 2023 16:32

add interaction metrics for navbar

d5efabc

copy paste typo

cc40fe6

Add telemetry for subscription management view (non-upgradeable)

922cab8

- Also add functional tests

fix addon functional test

4d0e329

add change plan telemetry and refactor telemetry tests

9758dac

MattLichtenstein requested review from lesleyjanenorton and brizental December 19, 2023 05:02

add data review link to new events

a663dee

MattLichtenstein commented Dec 19, 2023

View reviewed changes

brizental suggested changes Dec 19, 2023

View reviewed changes

brizental reviewed Dec 19, 2023

View reviewed changes

travis79 approved these changes Dec 19, 2023

View reviewed changes

fix data review links

0d6f3d9

MattLichtenstein added 4 commits December 20, 2023 10:31

formatting

d7ae211

replace navbar telemetry functional tests with unit tests

23c7997

fix functional tests

60f0424

fix testSubscription

2840280

MattLichtenstein requested a review from brizental December 21, 2023 04:52

brizental approved these changes Dec 21, 2023

View reviewed changes

add more unit tests

9f06d8a

MattLichtenstein merged commit eef4311 into main Dec 21, 2023
125 checks passed

MattLichtenstein deleted the VPN-5547 branch December 21, 2023 22:10

data-sync-user mentioned this pull request Jan 3, 2024

[Intermittent] [macOS] Mozilla VPN is crashing without clear steps while the VPN is OFF #8893

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VPN-5547: Implement subscription management telemetry #8838

VPN-5547: Implement subscription management telemetry #8838

MattLichtenstein commented Dec 19, 2023 •

edited

Loading

MattLichtenstein commented Dec 19, 2023

MattLichtenstein Dec 19, 2023

brizental Dec 19, 2023

MattLichtenstein Dec 19, 2023

brizental Dec 19, 2023

brizental left a comment

brizental Dec 19, 2023

brizental Dec 19, 2023

brizental Dec 19, 2023

brizental Dec 19, 2023 •

edited

Loading

MattLichtenstein Dec 19, 2023

travis79 left a comment

MattLichtenstein commented Dec 19, 2023 •

edited

Loading

brizental Dec 21, 2023

brizental Dec 21, 2023

MattLichtenstein Dec 21, 2023

brizental Dec 21, 2023

brizental Dec 21, 2023

brizental Dec 21, 2023

		});

		it('accountScreen impression event is recorded', async () => {

	void navbar_button_telemetry();
	void testNavbarButtonTelemetry();

VPN-5547: Implement subscription management telemetry #8838

VPN-5547: Implement subscription management telemetry #8838

Conversation

MattLichtenstein commented Dec 19, 2023 • edited Loading

Description

Reference

MattLichtenstein commented Dec 19, 2023

Request for data collection review form

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brizental left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brizental Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

travis79 left a comment

Choose a reason for hiding this comment

Request for data collection review form

Data Review

Result

MattLichtenstein commented Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MattLichtenstein commented Dec 19, 2023 •

edited

Loading

brizental Dec 19, 2023 •

edited

Loading

MattLichtenstein commented Dec 19, 2023 •

edited

Loading