Post deployment test runbook

Don't panic!

This document is here to help you figure out if the tests have failed because of a problem with the tests or a problem with the website.

If there's a problem with the website, you'll want to reproduce the issue and decide if we can rollback or if we need to fix forward.

If there's a problem with the tests, you can take a breath and find out if the issue is permanent or transient, and if it happened with the automated run, the build or the actual tests.

1. Let others know!

You might have gotten a message on your PR or noticed the message on google chat. Reply to the automated message on google chat to let others know you're looking into it!

2. Check the post deploy test run workflow

The E2E Github Action runs on every pull_request to main that also has the Seen-on-prod label attached to it (from prout). The action runs a suite of Playwright tests against the live site in Browserstack via the npm script pw-browserstack-test.

If there's been a failure, the first place to look is as the output of this command in the failing build's corresponding workflow run. To see the failing build, Open the playwright workflow. Click into the details and check the build log to see what failed. You might need to scroll down to the bottom.

You have the option to re-run the failed job.

3. Check BrowserStack

You can view the Builds/Sessions in BrowserStack that correspond to a failing worklow run by clicking the "Visit next to see related sessions" link in the Google Chat notification. This will take you directly to a view in BrowserStack such as the example below.

If all tests passed on their first attempt you'd see 1 BrowserStack build/session. However because Playwright is configured to give a failing test 1 opportunity to retry a failed workflow run will have multiple corresponding Browsertack builds/sessions. The reason for this is that when a build/session encounters a failed test it will terminate and be marked as "Failed". A new build/session will begin starting with a retry of the failed test from the previous build/session. If the retry fails it will terminate the build/session marking it as "Failed" and resume testing in a new build/session. If the retry passes all remaining tests will be run within the same build/session, as long as no other test failures occur.

In the example above:

• playwright-build-2adf698-14 Mar 2024 14:18:14 encountered a failing test (failed-test-1) and terminated and was marked as "Failed".
• playwright-build-2adf698-14 Mar 2024 14:21:08 retried failed-test-1, which passed, testing resumed within the same build/session, however a 2nd failing test (failed-test-2) was encountered so the build/session terminated and was marked as "Failed".
• playwright-build-2adf698-14 Mar 2024 14:27:05 retried failed-test-2, which failed again, so the build/session terminated and was marked as "Failed".
• playwright-build-2adf698-14 Mar 2024 14:29:06 the remaining tests were executed within this build/session and all passed, so this was marked as "Passes"

Because 1 a test failed twice (including a retry) the action flagged this as failed workflow run, and the team were notified via Google Chat.

FAQ

I've clicked the link but can't see the BrowserStack test results??

The department has multiple BrowserStack accounts, and chances are you've logged in to the wrong one! This account is only used to run these tests. Asks the leads (or anyone in the Digital/Reader Revenue chat room really) in either subscriptions or contributions to send you an invite.