[Uptime] Unskip certs func tests #73086

justinkambic · 2020-07-23T16:02:32Z

Summary

Fixes #70493.

We've had these tests skipped for some time. This patch attempts to update the tests to prevent flaky failures.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

justinkambic · 2020-07-23T18:46:03Z

Flaky test run passed, but an unrelated test had one failure: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/654/.

justinkambic · 2020-07-23T21:54:27Z

Second round of flaky test runner passed: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/658/

elasticmachine · 2020-07-23T21:55:24Z

Pinging @elastic/uptime (Team:uptime)

justinkambic · 2020-07-23T21:55:33Z

@elasticmachine merge upstream

justinkambic · 2020-07-26T13:25:34Z

Third run of flaky test runner passed x50, there were failures but none of the them were related to these tests: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/661/.

justinkambic · 2020-07-26T13:25:39Z

@elasticmachine merge upstream

shahzad31

@justinkambic i don't see any change beside un-skipping the test?

justinkambic · 2020-07-27T14:44:21Z

@shahzad31 this one was flaky because we sometimes ended up with two items when we were only expecting one. Could it be possible that we have updated the behavior of the list on the cert page at some point in the last month? I am having a really hard time reproducing this issue.

spalger

We just had a failure occur in 7.x https://kibana-ci.elastic.co/job/elastic+kibana+7.x/6833/

I don't think the fact that this is hard to reproduce means we should just re-enable the test as it's causing a real problem for other devs when it fails in PRs.

justinkambic · 2020-07-27T19:42:02Z

I don't think the fact that this is hard to reproduce means we should just re-enable the test as it's causing a real problem for other devs when it fails in PRs.

Yeah I wasn't aware of the recent failure when I made that comment. I'm going to need to delve deeper into this one.

…s that can lead to test flakiness.

justinkambic · 2020-07-27T21:23:37Z

Flaky test run with additional patch that should fix this issue: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/680/

justinkambic · 2020-07-28T16:08:57Z

@spalger I ran this flaky test build and it seems to have only run the SSL tests in x-pack ci-group 6, and I can't figure out why I'm not seeing the certs test run. It's in that group.

Parameters for the job are here: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/680/parameters/

justinkambic · 2020-07-28T16:13:50Z

@elasticmachine merge upstream

spalger · 2020-07-28T19:38:57Z

@spalger I ran this flaky test build and it seems to have only run the SSL tests in x-pack ci-group 6, and I can't figure out why I'm not seeing the certs test run. It's in that group.

Parameters for the job are here: kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/680/parameters

That doesn't seem to be the case to me, but it does look like the job timed out which is going to cause erratic failure reporting. You should probably lower the iteration count so that it can complete within the time limit.

justinkambic · 2020-07-28T20:13:36Z

@elasticmachine merge upstream

shahzad31

LGTM, seems ok now, flkay runner are passing as well https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/709/

justinkambic · 2020-07-29T16:42:15Z

@spalger I think we're at a point now where it's worth a try unskipping. @shahzad31 added an improvement and I've updated the code that generates the data used by the test. The failures before were happening because we were expecting one item but received two; the field the objects are grouped by was different between two objects. I think now that field should be the same for any generated data.

spalger

Sounds good, thanks for working on this and I hope it works!

spalger · 2020-07-30T20:55:22Z

@elasticmachine merge upstream

kibanamachine · 2020-07-30T22:40:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: cdbc47e

Build metrics

✅ unchanged

History

💚 Build #65213 succeeded 87a51ba
💔 Build #65188 failed a6e4f71
💚 Build #64904 succeeded c4c1e9a
💔 Build #64773 failed a10ad61
💚 Build #64520 succeeded b5b9ee9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Temporarily unload all other functional tests. * Unskip certificates test. * Uncomment skipped test files. * Add explicit fields to check generator call to prevent grouping issues that can lead to test flakiness. * added wait for loading * update missing func Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Shahzad <[email protected]>

kibanamachine · 2020-08-04T15:08:45Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-08-05T15:18:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine · 2020-08-06T15:28:44Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

* Temporarily unload all other functional tests. * Unskip certificates test. * Uncomment skipped test files. * Add explicit fields to check generator call to prevent grouping issues that can lead to test flakiness. * added wait for loading * update missing func Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Shahzad <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Shahzad <[email protected]>

justinkambic · 2020-08-06T18:54:54Z

Backported to:

7.9/7.9.0 78419d0
[7.9] [Uptime] Unskip certs func tests (#73086) #73957
7.x/7.10.0 74bae73
[7.x] [Uptime] Unskip certs func tests (#73086) #73956

justinkambic added bug Fixes for quality problems that affect the customer experience blocker test_xpack_functional v8.0.0 Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v7.10.0 v7.9.0 labels Jul 23, 2020

justinkambic self-assigned this Jul 23, 2020

justinkambic added 3 commits July 23, 2020 14:46

Temporarily unload all other functional tests.

88035d4

Unskip certificates test.

3fe4580

Uncomment skipped test files.

7daa422

justinkambic force-pushed the uptime_unskip-certs-func-tests branch from f063f67 to 7daa422 Compare July 23, 2020 18:58

justinkambic marked this pull request as ready for review July 23, 2020 21:55

justinkambic added the release_note:skip Skip the PR/issue when compiling release notes label Jul 23, 2020

justinkambic requested review from spalger and shahzad31 July 26, 2020 15:33

shahzad31 reviewed Jul 27, 2020

View reviewed changes

spalger suggested changes Jul 27, 2020

View reviewed changes

justinkambic added 2 commits July 27, 2020 15:42

Merge branch 'master' into uptime_unskip-certs-func-tests

575ff5d

Add explicit fields to check generator call to prevent grouping issue…

b5b9ee9

…s that can lead to test flakiness.

justinkambic force-pushed the uptime_unskip-certs-func-tests branch from 3d83dd7 to b5b9ee9 Compare July 27, 2020 21:17

Merge branch 'master' into uptime_unskip-certs-func-tests

a10ad61

elasticmachine and others added 3 commits July 28, 2020 14:13

Merge branch 'master' into uptime_unskip-certs-func-tests

c4c1e9a

added wait for loading

a6e4f71

update missing func

87a51ba

shahzad31 approved these changes Jul 29, 2020

View reviewed changes

justinkambic requested a review from spalger July 30, 2020 20:37

spalger approved these changes Jul 30, 2020

View reviewed changes

Merge branch 'master' into uptime_unskip-certs-func-tests

cdbc47e

justinkambic merged commit 61194b6 into elastic:master Jul 31, 2020

justinkambic mentioned this pull request Jul 31, 2020

[7.x] [Uptime] Unskip certs func tests (#73086) #73956

Merged

justinkambic mentioned this pull request Jul 31, 2020

[7.9] [Uptime] Unskip certs func tests (#73086) #73957

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 4, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 6, 2020

justinkambic deleted the uptime_unskip-certs-func-tests branch August 6, 2020 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Uptime] Unskip certs func tests #73086

[Uptime] Unskip certs func tests #73086

justinkambic commented Jul 23, 2020

justinkambic commented Jul 23, 2020

justinkambic commented Jul 23, 2020

elasticmachine commented Jul 23, 2020

justinkambic commented Jul 23, 2020

justinkambic commented Jul 26, 2020

justinkambic commented Jul 26, 2020

shahzad31 left a comment

justinkambic commented Jul 27, 2020

spalger left a comment

justinkambic commented Jul 27, 2020

justinkambic commented Jul 27, 2020 •

edited

Loading

justinkambic commented Jul 28, 2020

justinkambic commented Jul 28, 2020

spalger commented Jul 28, 2020

justinkambic commented Jul 28, 2020

shahzad31 left a comment

justinkambic commented Jul 29, 2020

spalger left a comment

spalger commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

kibanamachine commented Aug 4, 2020

kibanamachine commented Aug 5, 2020

kibanamachine commented Aug 6, 2020

justinkambic commented Aug 6, 2020

[Uptime] Unskip certs func tests #73086

[Uptime] Unskip certs func tests #73086

Conversation

justinkambic commented Jul 23, 2020

Summary

Checklist

For maintainers

justinkambic commented Jul 23, 2020

justinkambic commented Jul 23, 2020

elasticmachine commented Jul 23, 2020

justinkambic commented Jul 23, 2020

justinkambic commented Jul 26, 2020

justinkambic commented Jul 26, 2020

shahzad31 left a comment

Choose a reason for hiding this comment

justinkambic commented Jul 27, 2020

spalger left a comment

Choose a reason for hiding this comment

justinkambic commented Jul 27, 2020

justinkambic commented Jul 27, 2020 • edited Loading

justinkambic commented Jul 28, 2020

justinkambic commented Jul 28, 2020

spalger commented Jul 28, 2020

justinkambic commented Jul 28, 2020

shahzad31 left a comment

Choose a reason for hiding this comment

justinkambic commented Jul 29, 2020

spalger left a comment

Choose a reason for hiding this comment

spalger commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

💚 Build Succeeded

Build metrics

History

kibanamachine commented Aug 4, 2020

kibanamachine commented Aug 5, 2020

kibanamachine commented Aug 6, 2020

justinkambic commented Aug 6, 2020

justinkambic commented Jul 27, 2020 •

edited

Loading