Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: set up uptime monitoring #623

Merged
merged 10 commits into from
Jun 10, 2022
Merged

Feature: set up uptime monitoring #623

merged 10 commits into from
Jun 10, 2022

Conversation

afeld
Copy link
Contributor

@afeld afeld commented May 24, 2022 •
edited
Loading

Closes #449.

This is a proof of concept to show two things:

About half A lot of the additions in the pull request are just Terraform boilerplate. Uptime-specific stuff is in terraform/uptime.tf. Note this doesn't bring any existing Azure resources into management by Terraform.

If people are 👍 on the approach, I would want to do the following from here:

  • Terraform stuff
    • Set up a non-local Terraform backend
    • Add documentation around use of Terraform
    • Get buy-in from @cal-itp/cdt-devsecops that we will all start to work in Terraform
    • Look into how to tag Terraform-managed resources as such
  • Monitoring stuff
    • Decide on the URLs we want to monitor, at least as URL ping tests
    • Decide on how we want to be alerted — options

Feedback welcome!

@afeld afeld requested a review from a team as a code owner May 24, 2022 20:16
@afeld afeld marked this pull request as draft May 24, 2022 20:16
afeld
afeld commented May 24, 2022
View reviewed changes
terraform/uptime.tf Outdated
<Request Method="GET" Version="1.1" Url="https://dev-benefits.calitp.org/healthcheck" ThinkTime="0" Timeout="300" ParseDependentRequests="True" FollowRedirects="True" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="200" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" />
</Items>
</WebTest>
XML
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annoyingly, this is a bit more complex than most Terraform resources. Thankfully, was mostly able to copy-and-paste from an example.

@afeld
Copy link
Contributor Author

afeld commented May 24, 2022 •
edited
Loading

Screenshot of Azure Application Insights WebTest, showing a scatter plot with a bunch of response times throughout the day

live version

thekaveman
thekaveman reviewed May 24, 2022
View reviewed changes
terraform/main.tf Outdated Show resolved Hide resolved
@afeld
Copy link
Contributor Author

afeld commented May 31, 2022

Example alert email. The resolution message looks almost identical, except it says "alert deactivated."

@afeld afeld force-pushed the feat/uptime-monitor branch 2 times, most recently from 9ee9a6e to 9c927e5 Compare June 3, 2022 22:48
@afeld afeld marked this pull request as ready for review June 3, 2022 22:50
@afeld
Copy link
Contributor Author

afeld commented Jun 3, 2022 •
edited
Loading

Ready for review! If someone feels strongly, I can split out the minimal Terraform setup from adding the uptime monitor; this pull request currently adds both. This configuration has been applied, i.e. the uptime monitors are live.

afeld
afeld commented Jun 3, 2022
View reviewed changes
docs/deployment/azure.md

[dev-benefits.calitp.org](https://dev-benefits.calitp.org) is currently deployed into a Microsoft Azure account provided by [California Department of Technology (CDT)'s Office of Enterprise Technology (OET)](https://techblog.cdt.ca.gov/2020/06/cdt-taking-the-lead-in-digital-transformation/), a.k.a. the "DevSecOps" team. More specifically, it uses [custom containers](https://docs.microsoft.com/en-us/azure/app-service/configure-custom-container) on [Azure App Service](https://docs.microsoft.com/en-us/azure/app-service/overview).

The infrastructure is configured as code via [Terraform](https://www.terraform.io/), for [various reasons](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/the-benefits-of-infrastructure-as-code/ba-p/2069350). We are adding existing resources to the configuration progressively. In other words, not _all_ our resources in Azure show up under [`terraform/`][terraform-dir], but we are [moving that direction](https://github.com/cal-itp/benefits/issues/618).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this caveat.

@afeld afeld requested a review from thekaveman June 3, 2022 22:53
@afeld afeld force-pushed the feat/uptime-monitor branch from 9c927e5 to 9226150 Compare June 3, 2022 23:03
angela-tran
angela-tran previously approved these changes Jun 3, 2022
View reviewed changes
Copy link
Member

@angela-tran angela-tran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm new to Terraform and still learning, but based on your explanation/demo yesterday, these changes look good to me. The documentation added here is really helpful too 👍

thekaveman
thekaveman requested changes Jun 4, 2022
View reviewed changes
Copy link
Member

@thekaveman thekaveman left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the walkthrough yesterday, and agreed with @angela-tran this looks good!

I do think we should just use the Production subscription values everywhere though - the https://dev-benefits.calitp.org domain is already pointing there and think it will be easier to focus on one subscription w/slots going forward.

terraform/main.tf Outdated Show resolved Hide resolved
terraform/monitor.tf Outdated Show resolved Hide resolved
terraform/main.tf Outdated Show resolved Hide resolved
terraform/uptime/main.tf Outdated Show resolved Hide resolved
@afeld afeld force-pushed the feat/uptime-monitor branch from 9226150 to f297e14 Compare June 8, 2022 02:37
@afeld afeld marked this pull request as draft June 8, 2022 02:37
@afeld afeld force-pushed the feat/uptime-monitor branch from f297e14 to 93bde90 Compare June 8, 2022 02:40
@afeld afeld requested a review from thekaveman June 9, 2022 21:18
@afeld afeld marked this pull request as ready for review June 9, 2022 21:18
@afeld afeld force-pushed the feat/uptime-monitor branch from 1178112 to 81b275c Compare June 9, 2022 21:19
thekaveman
thekaveman approved these changes Jun 9, 2022
View reviewed changes
Copy link
Member

@thekaveman thekaveman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

We should combine the common Azure-related docs from deployment/README.md and deployment/azure.md somewhere.

@afeld
Copy link
Contributor Author

afeld commented Jun 10, 2022

Will do that as a follow-up.

@afeld afeld merged commit e193198 into dev Jun 10, 2022
@afeld afeld deleted the feat/uptime-monitor branch June 10, 2022 00:01
@afeld afeld mentioned this pull request Jun 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thekaveman thekaveman thekaveman approved these changes

@angela-tran angela-tran angela-tran left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Set up uptime monitoring
3 participants
@afeld @thekaveman @angela-tran