From 549cf76628d1a87a4b9af825055785cc6177bdf7 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:20:24 -0800 Subject: [PATCH 1/8] docs: add a guide about distributing to QA --- src/content/docs/guides/testing-patches.mdx | 74 +++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 src/content/docs/guides/testing-patches.mdx diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx new file mode 100644 index 00000000..c91dddac --- /dev/null +++ b/src/content/docs/guides/testing-patches.mdx @@ -0,0 +1,74 @@ +--- +title: Testing Patches +description: Distribute patches to groups of users via Tracks +sidebar: + order: 3 +--- + +We recommend that you test your patches with a subset of users before +distributing to all users. + +Shorebird’s mechanism to release to a subset of users is “tracks”. By +convention the “stable” track is distributed to all users of your app by +default. When you make a patch “live” it goes to “stable” and all users will +update to that patch. + +Today Shorebird supports “staging”, “beta” and “stable” tracks: +https://pub.dev/documentation/shorebird_code_push/latest/shorebird_code_push/UpdateTrack.html +We have plans to [allow additional track names in the future](https://github.com/shorebirdtech/updater/issues/255). + +Typically we se customers use "staging" for internal testing (on developers +machines) and "beta" for wider group testing (e.g. QA teams where `shorebird` +commands are not available). + +We recommend 3 ways to distribute patches to a subset of +users (e.g. your QA team) for testing without affecting your public users in +production. + +All of these options require disabling auto_update support, and using +[package:shorebird_code_push](https://pub.dev/packages/shorebird_code_push) for +more advanced control over the update process. + +### Option 1: Hidden UI to switch tracks + +If your app does not have a login/user-account mechanism, this is your best +option. If it does, we recommend option 2. + +Some stores discourage hidden UIs, however our customers tell us it is common +practice. For example how Android enables developer mode when [clicking 7 times +on the version of +Android](https://developer.android.com/studio/debug/dev-options), some apps may +enable a “QA” mode in their app after a certain series of clicks or special +gestures, etc. From a QA mode you could allow testers to switch to the “beta” +patch track even from your production app. They could similarly switch back to +“stable” to move back to what production users see today. + +### Option 2: Control Shorebird track based on the current user’s account. + +If your app has a login mechanism, this is your best option. If it does not, we +recommend option 1. + +Once your user has logged in, you can mark a user as a tester in your user +information and then change your Shorebird update code to use that information +when updating, e.g. + +```dart +final track = user.isTester ? ShorebirdTrack.beta : ShorebirdTrack.stable; +shorebirdUpdater.update(track); +``` + +### Option 3: Control Shorebird Track based on how the app was installed. + +Other companies provide existing mechanisms for testing mobile apps before +production. Apple TestFlight, Google Release Tracks, Firebase App Distribution +are all examples of this. Shorebird does not have built-in detection of these +distribution mechanisms, but it is typically possible to detect the mechanism +via which an App was installed on the user’s device. + +It is then possible to key the Shorebird track off of this install mechanism, +allowing you to thus distribute patches *only* to your TestFlight users, etc. + +Because detection of install mechanisms can be unreliable, we recommend option 1 +or 2 for most teams, however include this for completeness. Most teams who +distribute via TestFlight, for example, also have QA-specific accounts and thus +Option 2 is available to them and generally works strictly better. \ No newline at end of file From 8c112e85a77cd9ff7e27ceb929b6bb10e0f08b70 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:20:48 -0800 Subject: [PATCH 2/8] chore: npm run format --- src/content/docs/guides/testing-patches.mdx | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index c91dddac..61f4c6d2 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -8,9 +8,9 @@ sidebar: We recommend that you test your patches with a subset of users before distributing to all users. -Shorebird’s mechanism to release to a subset of users is “tracks”. By +Shorebird’s mechanism to release to a subset of users is “tracks”. By convention the “stable” track is distributed to all users of your app by -default. When you make a patch “live” it goes to “stable” and all users will +default. When you make a patch “live” it goes to “stable” and all users will update to that patch. Today Shorebird supports “staging”, “beta” and “stable” tracks: @@ -32,20 +32,20 @@ more advanced control over the update process. ### Option 1: Hidden UI to switch tracks If your app does not have a login/user-account mechanism, this is your best -option. If it does, we recommend option 2. +option. If it does, we recommend option 2. Some stores discourage hidden UIs, however our customers tell us it is common practice. For example how Android enables developer mode when [clicking 7 times on the version of Android](https://developer.android.com/studio/debug/dev-options), some apps may enable a “QA” mode in their app after a certain series of clicks or special -gestures, etc. From a QA mode you could allow testers to switch to the “beta” -patch track even from your production app. They could similarly switch back to +gestures, etc. From a QA mode you could allow testers to switch to the “beta” +patch track even from your production app. They could similarly switch back to “stable” to move back to what production users see today. ### Option 2: Control Shorebird track based on the current user’s account. -If your app has a login mechanism, this is your best option. If it does not, we +If your app has a login mechanism, this is your best option. If it does not, we recommend option 1. Once your user has logged in, you can mark a user as a tester in your user @@ -61,14 +61,14 @@ shorebirdUpdater.update(track); Other companies provide existing mechanisms for testing mobile apps before production. Apple TestFlight, Google Release Tracks, Firebase App Distribution -are all examples of this. Shorebird does not have built-in detection of these +are all examples of this. Shorebird does not have built-in detection of these distribution mechanisms, but it is typically possible to detect the mechanism via which an App was installed on the user’s device. It is then possible to key the Shorebird track off of this install mechanism, -allowing you to thus distribute patches *only* to your TestFlight users, etc. +allowing you to thus distribute patches _only_ to your TestFlight users, etc. Because detection of install mechanisms can be unreliable, we recommend option 1 -or 2 for most teams, however include this for completeness. Most teams who +or 2 for most teams, however include this for completeness. Most teams who distribute via TestFlight, for example, also have QA-specific accounts and thus -Option 2 is available to them and generally works strictly better. \ No newline at end of file +Option 2 is available to them and generally works strictly better. From 590f53ba6afc9bcd8b26f6a5bfcd5e751c235530 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:24:02 -0800 Subject: [PATCH 3/8] docs: reorder sidebar --- astro.config.mjs | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/astro.config.mjs b/astro.config.mjs index 358d2bae..aca4a545 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -64,26 +64,30 @@ export default defineConfig({ { label: 'Guides', items: [ + { + label: 'Development Workflow', + link: '/guides/development-workflow', + }, { label: 'Staging Patches', link: '/guides/staging-patches', }, { - label: 'Development Workflow', - link: '/guides/development-workflow', + label: 'Testing Patches', + link: '/guides/testing-patches', }, { - label: 'Patch Signing (beta)', + label: 'Signing Patches', link: '/guides/patch-signing', }, - { - label: 'Crash Reporting', - link: '/guides/crash-reporting', - }, { label: 'Percentage-Based Rollouts', link: '/guides/percentage-based-rollouts', }, + { + label: 'Crash Reporting', + link: '/guides/crash-reporting', + }, { label: 'Flavors', autogenerate: { From a6f0af7110171eed6857e8cf5fc7dd2d54756261 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:27:30 -0800 Subject: [PATCH 4/8] fix: make dart actually plausibly compile --- src/content/docs/guides/testing-patches.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index 61f4c6d2..11bb0440 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -53,8 +53,8 @@ information and then change your Shorebird update code to use that information when updating, e.g. ```dart -final track = user.isTester ? ShorebirdTrack.beta : ShorebirdTrack.stable; -shorebirdUpdater.update(track); +final track = user.isTester ? UpdateTrack.beta : UpdateTrack.stable; +shorebirdUpdater.update(track: track); ``` ### Option 3: Control Shorebird Track based on how the app was installed. From 25f3a3471f803946ead328e8b7c03a875420c979 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:29:55 -0800 Subject: [PATCH 5/8] feat: add a note about privacy --- src/content/docs/guides/testing-patches.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index 11bb0440..7714918f 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -21,6 +21,13 @@ Typically we se customers use "staging" for internal testing (on developers machines) and "beta" for wider group testing (e.g. QA teams where `shorebird` commands are not available). +As part of our privacy stance, we _never_ know anything about your users, so +we have no way to tell users apart or to specify which users should receive +a patch from within our product. However we do provide you mechanisms for +controlling updates via Shorebird with user information you have access to +from within your company and your application. See this guide as well as +[Percentage Based Rollouts](/guides/percentage-based-rollouts) for examples. + We recommend 3 ways to distribute patches to a subset of users (e.g. your QA team) for testing without affecting your public users in production. From 6fc406281f311ace081322303311becbda320a21 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:38:44 -0800 Subject: [PATCH 6/8] Apply suggestions from code review Co-authored-by: Bryan Oltman --- src/content/docs/guides/testing-patches.mdx | 32 ++++++++++----------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index 7714918f..e8a0c270 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -17,18 +17,18 @@ Today Shorebird supports “staging”, “beta” and “stable” tracks: https://pub.dev/documentation/shorebird_code_push/latest/shorebird_code_push/UpdateTrack.html We have plans to [allow additional track names in the future](https://github.com/shorebirdtech/updater/issues/255). -Typically we se customers use "staging" for internal testing (on developers +Typically we se customers use "staging" for internal testing (on developers' machines) and "beta" for wider group testing (e.g. QA teams where `shorebird` commands are not available). As part of our privacy stance, we _never_ know anything about your users, so we have no way to tell users apart or to specify which users should receive -a patch from within our product. However we do provide you mechanisms for +a patch from within our product. However, we do provide you mechanisms for controlling updates via Shorebird with user information you have access to -from within your company and your application. See this guide as well as -[Percentage Based Rollouts](/guides/percentage-based-rollouts) for examples. +from within your company and your application. See the +[Percentage Based Rollouts guide](/guides/percentage-based-rollouts) for examples. -We recommend 3 ways to distribute patches to a subset of +We recommend using one of the following three approaches to distribute patches to a subset of users (e.g. your QA team) for testing without affecting your public users in production. @@ -41,13 +41,13 @@ more advanced control over the update process. If your app does not have a login/user-account mechanism, this is your best option. If it does, we recommend option 2. -Some stores discourage hidden UIs, however our customers tell us it is common -practice. For example how Android enables developer mode when [clicking 7 times -on the version of -Android](https://developer.android.com/studio/debug/dev-options), some apps may +Some stores discourage hidden UIs; however our customers tell us it is common +practice. For example, some apps may enable a “QA” mode in their app after a certain series of clicks or special -gestures, etc. From a QA mode you could allow testers to switch to the “beta” -patch track even from your production app. They could similarly switch back to +gestures, etc., similar to how Android enables developer mode when [tapping 7 times +on the Android version text in the settings app](https://developer.android.com/studio/debug/dev-options) +From a QA mode, you could allow testers to switch to the “beta” +patch track, even from your production app. They could similarly switch back to “stable” to move back to what production users see today. ### Option 2: Control Shorebird track based on the current user’s account. @@ -55,8 +55,8 @@ patch track even from your production app. They could similarly switch back to If your app has a login mechanism, this is your best option. If it does not, we recommend option 1. -Once your user has logged in, you can mark a user as a tester in your user -information and then change your Shorebird update code to use that information +Once your user has logged in, you determine whether the user is a tester +and then change your Shorebird update code to use that information when updating, e.g. ```dart @@ -72,10 +72,10 @@ are all examples of this. Shorebird does not have built-in detection of these distribution mechanisms, but it is typically possible to detect the mechanism via which an App was installed on the user’s device. -It is then possible to key the Shorebird track off of this install mechanism, +It is then possible to pick the Shorebird track based on the detected install mechanism, allowing you to thus distribute patches _only_ to your TestFlight users, etc. Because detection of install mechanisms can be unreliable, we recommend option 1 -or 2 for most teams, however include this for completeness. Most teams who +or 2 for most teams, but include this option for completeness. Most teams who distribute via TestFlight, for example, also have QA-specific accounts and thus -Option 2 is available to them and generally works strictly better. +option 2 is strictly better. From 64ae01facf3073efdfb54d0b4472e87d02abbdaf Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:39:01 -0800 Subject: [PATCH 7/8] Apply suggestions from code review Co-authored-by: Bryan Oltman --- src/content/docs/guides/testing-patches.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index e8a0c270..1e85980a 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -32,7 +32,7 @@ We recommend using one of the following three approaches to distribute patches t users (e.g. your QA team) for testing without affecting your public users in production. -All of these options require disabling auto_update support, and using +All of these options require disabling `auto_update` support, and using [package:shorebird_code_push](https://pub.dev/packages/shorebird_code_push) for more advanced control over the update process. From 33f540f01d3e5a47eadba9bd3f3429b23d5fdb76 Mon Sep 17 00:00:00 2001 From: Eric Seidel Date: Fri, 24 Jan 2025 14:44:43 -0800 Subject: [PATCH 8/8] fix: swap option 1 and option 2 --- src/content/docs/guides/testing-patches.mdx | 32 ++++++++++----------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/src/content/docs/guides/testing-patches.mdx b/src/content/docs/guides/testing-patches.mdx index 1e85980a..9df04c9f 100644 --- a/src/content/docs/guides/testing-patches.mdx +++ b/src/content/docs/guides/testing-patches.mdx @@ -36,24 +36,10 @@ All of these options require disabling `auto_update` support, and using [package:shorebird_code_push](https://pub.dev/packages/shorebird_code_push) for more advanced control over the update process. -### Option 1: Hidden UI to switch tracks - -If your app does not have a login/user-account mechanism, this is your best -option. If it does, we recommend option 2. - -Some stores discourage hidden UIs; however our customers tell us it is common -practice. For example, some apps may -enable a “QA” mode in their app after a certain series of clicks or special -gestures, etc., similar to how Android enables developer mode when [tapping 7 times -on the Android version text in the settings app](https://developer.android.com/studio/debug/dev-options) -From a QA mode, you could allow testers to switch to the “beta” -patch track, even from your production app. They could similarly switch back to -“stable” to move back to what production users see today. - -### Option 2: Control Shorebird track based on the current user’s account. +### Option 1: Control Shorebird track based on the current user’s account. If your app has a login mechanism, this is your best option. If it does not, we -recommend option 1. +recommend option 2. Once your user has logged in, you determine whether the user is a tester and then change your Shorebird update code to use that information @@ -64,6 +50,20 @@ final track = user.isTester ? UpdateTrack.beta : UpdateTrack.stable; shorebirdUpdater.update(track: track); ``` +### Option 2: Control Shorebird track with a hidden UI. + +If your app does not have a login/user-account mechanism, this is your best +option. If it does, we recommend option 1. + +Some stores discourage hidden UIs; however our customers tell us it is common +practice. For example, some apps may +enable a “QA” mode in their app after a certain series of clicks or special +gestures, etc., similar to how Android enables developer mode when [tapping 7 times +on the Android version text in the settings app](https://developer.android.com/studio/debug/dev-options) +From a QA mode, you could allow testers to switch to the “beta” +patch track, even from your production app. They could similarly switch back to +“stable” to move back to what production users see today. + ### Option 3: Control Shorebird Track based on how the app was installed. Other companies provide existing mechanisms for testing mobile apps before