Nav Unification: Decision on incremental rollout target user groups.

[cpapazoglou]

After Nav Unification: Research incremental rollout mechanism and outline options let's make sure that we have a clear plan on the rollout target user groups.

[cpapazoglou]

... and then roll changes out to 5%, then 20%, then 50%, then 100% of customers

@davemart-in other than selecting customers by eg their user id only there was an idea of initially launching to single site users. Does this still makes sense? Are there any other targets / segments we could make use of?

[davemart-in]

@cpapazoglou I think that still makes sense. Otherwise customers with multiple sites may see the updated nav for one site and the older nave for another. I'm not aware of any other constraints that we'd want to consider. I'll cc @obenland here though in case he has additional thoughts.

[cpapazoglou]

@davemart-in actually, let's say we want to target 5% of our customers. In all scenarios, we will be using the UserId rather than the BlogId which will result affected users having the same experience across their sites.

Scenario 1

• We will get them by filtering with UserId%100 < 5 this will return us all customers with UserIds ending in 00,01,02,03,04.

Scenario 2

• We will firstly filter down by single site customers.
• At this point will get the 5% by filtering with UserId%100 < 5 this will return us all customers with UserIds ending in 00,01,02,03,04.

Scenario 3

• We will firstly filter down by single site customers who have certain Locale set or / and are from a certain group of Countries.
• At this point will get the 5% by filtering with UserId%100 < 5 this will return us all customers with UserIds ending in 00,01,02,03,04.

Scenario 4

• We will firstly filter down by users who don't have any plan - free users.
• At this point will get the 5% by filtering with UserId%100 < 5 this will return us all customers with UserIds ending in 00,01,02,03,04.

Scenario 5

• We will firstly filter down by users who have at least one atomic site.
• At this point will get the 5% by filtering with UserId%100 < 5 this will return us all customers with UserIds ending in 00,01,02,03,04.

Notes:

• In all Scenarios except (1), the affected customers will be less than 5% of our customers dataset. In case we don't need to segment them more, Scenario (1) would be the most straightforward in terms of implementation and roll out.

• Scenario (4) could be a good case for a/b testing the Free domain with a plan nudge. This would also mean that we are targeting the least risky audience in terms of retention rate. Also, probably the least probable to give feedback / contact support.

• Scenario (5) could be a good case for getting back feedback from our most experienced customers who may have the most profit from this unification but also the most problems (eg plugins, opt-out links). This segment is risky but better sooner than later. Also this segment will result to the most narrow audience (5% of atomic users ;-)).

• In scenario (2) we are possibly getting the most broad audience. They may either have a simple or atomic site. They will certainly not have the Switch Sites component. They will be less affected from colour-schemes unification.

• I don't see any value in scenario (3) - just noted it down for visualising our options.

[davemart-in]

Scenario 2 sounds good to me. @obenland thoughts?

[obenland]

Only that Scenario 2 is my favorite, too :)

[cpapazoglou]

Scenario (2) it is then! With the exception that it won't matter for a12s. After internal testing where a12s can opt in / out, during rollout all a12s will be presented the new experience.

so the initial segment would be

isA11n OR UserId%100 < 5 && singleSiteUser

we will just need to decide in the future when we are removing the singleSiteUser filter or if we are creating a separate rollout target of non single site users eg UserId%100 < 5 && !singleSiteUser. In the latter case we can incrementally rollout both single site users and non single site users and better control edge cases (bugs).

[getdave]

Reopening as we want to re-affirm our understanding once more.

• exact process and timing for ramping up - do we still follow this?
• Do we still need to restrict to Scenario 2 “Single Site”.
• Ramp slowly?
• Do we enable for all new users? or just existing?

[obenland]

Thanks for re-opening the discussion. I think today I'd be leaning towards scenario 1 with the addition of all new users.

UserId < latestUserId && ( UserId % 100 ) < 5

In terms of ramp up speed and increments, I wouldn't mind making those decisions as it's happening and we have a better understanding of support load and issues.

[cpapazoglou]

@obenland we have been discussing enabling nav-unification for all users with getCurrentUserDate > launchDate + 24h, so that deployment related code changes are 100% in sync among codebases. +24h is used to account for potential timezone differences. What's your take on this?

[obenland]

Sure, sounds good