Postmortem: broken site

[julieg18]

This ticket contains a postmortem about dvc.org breaking yesterday.

High Level Summary

dvc.org broke for a couple hours yesterday. None of the buttons and links were working correctly. The first sign of the site breaking was a sentry alert, and a team member noticed about two hours later after accessing the site. @shcheklein found out that #2779 was most likely what broke the site. After we reverted the pull request, the site started working correctly again.

Timeline

All times in UTC time

• 2021-09-06 14:49 yarn upgrade && yarn format-all #2779 was merged and build was deployed
• 2021-09-06 21:03 First sentry error went out(The result of this StaticQuery could not be fetched.) We received a couple more after that.
• 2021-09-06 22:47 @jorgeorpinel messaged in slack about the site not working
• 2021-09-06 22:51 @shcheklein found out that yarn upgrade && yarn format-all #2779 is what was most likely causing the site to break
• 2021-09-06 22:53 pull request(Revert "yarn upgrade && yarn format-all" #2799) for reverting yarn upgrade && yarn format-all #2779 was created and merged
• 2021-09-06 23:03 build was deployed and site was working correctly again

Perf indicators:

• Time to notice: about two hours
• Fix Developed After: 4 minutes
• Resolved After: 17 minutes

Impact

• No users were able to use dvc.org correctly

Root cause analysis

• Pull request Restyle live: initial docs draft #2279 was about upgrading some packages, fixing some typescript/formatting errors. Most likely one or more of the upgraded packages is what caused the site to break. Though further testing/research will be needed to find out which one and why exactly.

Prevention and next steps

• Maybe keep a closer eye on Sentry errors, though I don't think any members of the websites team were online at the time.

[jorgeorpinel]

2021-4-20

I think all the dates were 2021-09-06 🙂

33:03

Interesting time of day 😬

As for prevention, I think #2779 should've been 2 PRs: one for yarn update, and another one for formatting files. The former could be a task left for the website team to do periodically, while the docs team can do the formattings when/if needed.

[shcheklein]

Thanks @julieg18 for the postmortem. A great summary. One thing we need to discuss better is to

Maybe keep a closer eye on Sentry errors, though I don't think any members of the websites team were online at the time.

I think that one the items that we should really prioritize is Sentry. We need to denoise the channel. The that the prevented us from paying attention to the problem until we personally hit it.

I think I was online and @jorgeorpinel was around.

And for critical issues we should all be paying more attention and agree that someone keeps (or all of doing our best - that would the easiest) an eye on that channel even when we are on vacation.

[julieg18]

I think that one the items that we should really prioritize is Sentry. We need to denoise the channel. The that the prevented us from paying attention to the problem until we personally hit it.

Makes sense! We could update Sentry to only send alerts to Slack if we have a certain amount of users that are getting the error. Looking at our current dvc.org errors that should lower the amount since over half of our dvc.org issues have 1 or 2 users.

[shcheklein]

@julieg18 yep, sounds good