Add upgrade rule smoke tests

[liza-mae]

Adding an upgrade test for issue #135740.

[liza-mae]

@elasticmachine merge upstream

[liza-mae]

@kobelb can I get a review for this please?

[kobelb]

How is the "Upgrade Rule" being created? I don't see where we load the problematic alerting rule that would cause the rule to fail to run.

Also, this test is only checking the last response status. Depending on how the alerting rule is created, this might miss an upgrade failure if the alerting rule didn't have the chance to run yet.

[liza-mae]

The loading of the rule is done in separate repository as part of the upgrade setup. It uses the rest API to create it, I will put up that PR and link it here so you can review. I was able to reproduce the bug with the rule I created, however I did not verify the last response in the setup, maybe it is needed there. I will make the updates and let you know.

[liza-mae]

@elasticmachine merge upstream

[liza-mae]

@kobelb the rule setup is linked to this is PR if you would like to review again.

[kobelb]

Thanks @liza-mae. Is the rule setup done on an old version of Kibana prior to upgrading to a new version and then the smoke tests are done?

If so, that relegates my concerns to the fact that this alerting rule might not have had a chance to run prior to the smoke test verifying that the last response status is "OK". This is potentially unlikely, and the only way I can think of detecting that is by comparing the last time it ran prior to the upgrade to post upgrade, or having the smoke test wait for the last response to update.

[liza-mae]

@kobelb the setup is done an older version of kibana. I added a rest api call to create the rule and it appeared to work fine with the testing I did on 7.13.x, 7.17.x, 8.2.x, 8.3.x and they all had status Ok except for the bug originating from 8.2.x where it had Error and this test caught it. This was done in local testing, I will need to merge the PR to try it on all the paths in CI though, if there is a concern the rule does not run and I can adjust the setup. In what cases would the rule not run after it is created?

[liza-mae]

I am using https://www.elastic.co/guide/en/kibana/current/create-rule-api.html

[liza-mae]

When I run the API setup, I get this in the UI:

I have not seen it be anything other than Ok in the setup when testing of the versions I listed above. It appears the rule is executed when created and then at each interval listed after that. We can have a quick meeting so I can get clarification if I am missing something.

[liza-mae]

@elasticmachine merge upstream

[liza-mae]

@elasticmachine merge upstream

[liza-mae]

All tests pass for this feature on main:
https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-upgrade-snapshot-tests+main/96/testReport/

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 21c0c5f
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #58822 succeeded 3175b09
• 💚 Build #58347 succeeded bdb8b47
• 💚 Build #57828 succeeded 70cc0d2
• 💚 Build #56814 succeeded 3c2412b
• 💚 Build #56456 succeeded 14e828a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @liza-mae

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details