Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(Azure): suppress downtime alerts when data is being loaded #241

Merged
merged 2 commits into from
Feb 17, 2023

Conversation

afeld
Copy link
Contributor

@afeld afeld commented Feb 16, 2023

Fixes #220.

Uses an alert processing rule to disable notifications during that specific window of time every night.

terraform/uptime.tf Outdated Show resolved Hide resolved
@@ -8,3 +8,27 @@ module "healthcheck" {
name = "mst-courtesy-cards-eligibility-server-${local.env_name}-healthcheck"
url = "https://${azurerm_linux_web_app.main.default_hostname}/healthcheck"
}

# ignore when app restarts as data is being reloaded
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only configured on the Velocity side, or somewhere on the Eligibility Server side as well?

@afeld afeld marked this pull request as ready for review February 16, 2023 23:46
@afeld afeld requested a review from a team as a code owner February 16, 2023 23:46
@afeld
Copy link
Contributor Author

afeld commented Feb 16, 2023

Unfortunately we won't really know if this works until we merge it and see if an alert comes through overnight 🤷

# ignore when app restarts as data is being reloaded
# https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-processing-rules
# the Terraform resource doesn't support time windows, so need to drop down to an ARM template instead
# https://github.com/hashicorp/terraform-provider-azurerm/issues/16726
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See this note re: the JSON above.

@afeld afeld force-pushed the fix/suppress-downtime-alerts branch from e582236 to 0c8000d Compare February 16, 2023 23:52
}
],
"schedule": {
"timeZone": "Eastern Standard Time",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created based on grabbing JSON from the Portal, hence it being in my time zone. Didn't bother to change.

Copy link
Member

@angela-tran angela-tran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the walkthrough @afeld

@afeld afeld merged commit 6689570 into dev Feb 17, 2023
@afeld afeld deleted the fix/suppress-downtime-alerts branch February 17, 2023 00:07
@afeld
Copy link
Contributor Author

afeld commented Feb 27, 2023

Confirmed we're not longer getting the alerts in dev.

Screenshot 2023-02-27 at 2 47 57 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

resolve nightly downtime alerts
2 participants