[Synthetics] copy alert state to alert context and implement alert recovery

[dominiqueclarke]

Summary

Resolves #128760
Resolves #128761

This PR takes the first step towards transitioning Uptime alerting towards preferring context over state. For more information about why this change is important, please visit the ticket.

For more information on the overall stages of the transition, please see #126280

Testing that State has been copied over to Context

1. Connect to ES via oblt-cli. This ensures you will have at least one monitor that is failing and one with an aging or expired certificate.
2. Navigate to Uptime overview and create a monitor status rule from the alert flyout, making sure to give the rule a name.
3. Select an alert connector. The easiest connector to configure is a server log. Click save.
4. Ensure the alert comes to your server log with the appropriate alert message
5. Navigate to Uptime overview and create a tls rule from the alert flyout, making sure to give the rule a name.
6. Select the server log connector. Click save
7. Ensure the tls alert comes to your server log with the appropriate alert message

Note
You'll likely want to set up control rules to ensure that the alert messages match our original implementation and still work for legacy users using state instead of context.
To do so, create two more rules, one for monitor status and one for tls. Adjust the alert message to the following. Ensure that the control (state) matches the test (context).

Monitor Status

Monitor {{state.monitorName}} with url {{{state.monitorUrl}}} from {{state.observerLocation}} {{{state.statusMessage}}} The latest error message is {{{state.latestErrorMessage}}}

TLS

Detected TLS certificate {{state.commonName}} from issuer {{state.issuer}} is {{state.status}}. Certificate {{state.summary}}

Testing alert resolution

Monitor Status

1. Start your own failing monitor via the synthetics service or running heartbeat from source. This ensures you're able to control your monitor's up and down state.
2. Create a monitor status alert. To make things simple, I recommend creating an alert for when the monitor has failed more than once within the last 3 minutes, and removing the availability check.
3. Before saving your alert, ensure that you add an action for the Recovery action group. First select a connector type (I like to use Server logs). Then add default action for Run When: Uptime Monitor Down. Click Add action again. This type change Run When to Recovered. Confirm that the default content for recovery matches the AC defined in [Uptime] Specify alert recovery context #128761.

4. Wait for the monitor alert to trigger. 5. Force the monitor to resolve by either A.) stopping heartbeat from running or deleting the monitor or B.) Changing the monitor config to a value that would trigger an UP status. 6. Confirm that you receive the alert recovery message to your specified action connector

TLS

1. Run your own monitors, or connect to ES via oblt-cli.
2. Navigate to Uptime Settings
3. Change age limit for certificates to 1 day

4. Create a TLS status alert. Make sure you add an action for the when the alert is triggered as well as for when the alert is recovered. Confirm that the default content for recovery matches the AC defined in https://github.com//issues/128761. 5. Wait for the alert to trigger 6. Navigate back to Uptime Settings. Change age limit back to 730 days. 7. Confirm that you receive the alert recovery message to your specified action connector

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[shahzad31]

does this need to be set for other alert types?

[shahzad31]

Tested e2e and everything looks fine !!

Great work on this !!

[dominiqueclarke]

@elasticmachine merge upstream

[dominiqueclarke]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 7d2daad
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
synthetics	792.4KB	792.4KB	+16.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
synthetics	22.0KB	23.2KB	+1.3KB

History

• 💔 Build #43318 failed 7784f15
• 💚 Build #42804 succeeded a102eb6
• 💚 Build #41656 succeeded 0fbdbee
• 💚 Build #41452 succeeded b25cb9b
• 💔 Build #40999 failed a1e4abe
• 💔 Build #40534 failed 75bd15c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream