Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve bugfix 7198 (null tests in discover) test stability #71250

Merged
merged 6 commits into from
Jul 13, 2020
Merged

improve bugfix 7198 (null tests in discover) test stability #71250

merged 6 commits into from
Jul 13, 2020

Conversation

lizozom
Copy link
Contributor

@lizozom lizozom commented Jul 9, 2020 •
edited
Loading

Summary

Improve test stability

Backported here #71211

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@lizozom lizozom added v8.0.0 Team:AppArch release_note:skip Skip the PR/issue when compiling release notes v7.9.0 labels Jul 9, 2020
@lizozom lizozom self-assigned this Jul 9, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@lizozom lizozom requested a review from kertal July 9, 2020 14:51
@timroes
Copy link
Contributor

timroes commented Jul 9, 2020

Since the test is now skipped on master, this needs to merge upstream and unskip it again.

@lizozom
Copy link
Contributor Author

lizozom commented Jul 9, 2020

@elasticmachine merge upstream

@lizozom
Copy link
Contributor Author

lizozom commented Jul 11, 2020

@elasticmachine merge upstream

kertal
kertal approved these changes Jul 13, 2020
View reviewed changes
Copy link
Member

@kertal kertal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix looks good to flaky test runner: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/582/testReport/ 👍
Could you modify the title and description of this PR to be a bit more descriptive? Many thx. I bet it's the long weekend but I totally forgot what bug fix 7198 was 🧠

kertal
kertal reviewed Jul 13, 2020
View reviewed changes
test/functional/apps/discover/_doc_navigation.js
@@ -29,7 +29,7 @@ export default function ({ getService, getPageObjects }) {
const retry = getService('retry');

// Flaky: https://github.com/elastic/kibana/issues/71216
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove this comment now

@lizozom lizozom merged commit 79df8e0 into elastic:master Jul 13, 2020
@lizozom lizozom changed the title improve bugfix 7198 test stability improve bugfix 7198 (null tests in discover) test stability Jul 13, 2020
@lizozom lizozom removed the v7.9.0 label Jul 13, 2020
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / kibana-xpack-agent / Kibana Embedded in iframe with X-Pack Security.x-pack/test/functional_embedded/tests/iframe_embedded·ts.Kibana embedded in iframe should open Kibana for logged-in user

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/70928

[00:00:00]       │
[00:00:00]         └-: Kibana embedded
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: in iframe
[00:00:00]             └-> "before all" hook
[00:00:00]             └-> should open Kibana for logged-in user
[00:00:00]               └-> "before each" hook: global before each
[00:00:00]               │ debg TestSubjects.exists(headerGlobalNav)
[00:00:00]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="headerGlobalNav"]') with timeout=2500
[00:00:02]               │ debg --- retry.tryForTime error: [data-test-subj="headerGlobalNav"] is not displayed
[00:00:03]               │ debg navigating to login url: https://localhost:6131/login
[00:00:03]               │ debg navigate to: https://localhost:6131/login
[00:00:03]               │ debg browser[INFO] https://localhost:6131/login?_t=1594626075031 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:03]               │
[00:00:03]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:03]               │ debg ... sleep(700) start
[00:00:04]               │ debg ... sleep(700) end
[00:00:04]               │ debg returned from get, calling refresh
[00:00:04]               │ debg browser[INFO] https://localhost:6131/login?_t=1594626075031 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:04]               │
[00:00:04]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:04]               │ debg currentUrl = https://localhost:6131/login
[00:00:04]               │          appUrl = https://localhost:6131/login
[00:00:04]               │ debg TestSubjects.find(kibanaChrome)
[00:00:04]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:08]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:18Z
[00:00:08]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:08]               │
[00:00:08]               │      "
[00:00:08]               │ debg ... sleep(501) start
[00:00:08]               │ debg ... sleep(501) end
[00:00:08]               │ debg in navigateTo url = https://localhost:6131/login
[00:00:08]               │ debg TestSubjects.exists(statusPageContainer)
[00:00:08]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:00:11]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:00:12]               │ debg Waiting for Login Form to appear.
[00:00:12]               │ debg Waiting up to 100000ms for login form...
[00:00:12]               │ debg TestSubjects.exists(loginForm)
[00:00:12]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:00:12]               │ debg TestSubjects.setValue(loginUsername, elastic)
[00:00:12]               │ debg TestSubjects.click(loginUsername)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:12]               │ debg TestSubjects.setValue(loginPassword, changeme)
[00:00:12]               │ debg TestSubjects.click(loginPassword)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:12]               │ debg TestSubjects.click(loginSubmit)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:12]               │ debg Waiting for login result, expected: undefined.
[00:00:12]               │ debg Waiting up to 20000ms for logout button visible...
[00:00:12]               │ debg TestSubjects.exists(userMenuButton)
[00:00:12]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:00:15]               │ debg browser[INFO] https://localhost:6131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:15]               │
[00:00:15]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:15]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:26Z
[00:00:15]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:15]               │
[00:00:15]               │      "
[00:00:15]               │ debg --- retry.tryForTime error: [data-test-subj="userMenuButton"] is not displayed
[00:00:16]               │ proc [kibana]  error  [07:41:28.065] [error][client][connection] Error: 140600031983488:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
[00:00:16]               │ proc [kibana] 
[00:00:16]               │ERROR browser[SEVERE] http://localhost:6131/api/_newsfeed-FTS-external-service-simulators/kibana/v8.0.0-SNAPSHOT.json - Failed to load resource: net::ERR_EMPTY_RESPONSE
[00:00:16]               │ERROR browser[SEVERE] https://localhost:6131/34549/bundles/plugin/newsfeed/newsfeed.plugin.js 0:19107 TypeError: Failed to fetch
[00:00:16]               │          at Fetch._callee3$ (https://localhost:6131/34549/bundles/core/core.entry.js:34:105174)
[00:00:16]               │          at l (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155138)
[00:00:16]               │          at Generator._invoke (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:154891)
[00:00:16]               │          at Generator.forEach.e.<computed> [as throw] (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155495)
[00:00:16]               │          at fetch_asyncGeneratorStep (https://localhost:6131/34549/bundles/core/core.entry.js:34:99267)
[00:00:16]               │          at _throw (https://localhost:6131/34549/bundles/core/core.entry.js:34:99675)
[00:00:16]               │ debg TestSubjects.exists(userMenuButton)
[00:00:16]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:00:16]               │ debg TestSubjects.exists(userMenu)
[00:00:16]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:00:19]               │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:00:19]               │ debg TestSubjects.click(userMenuButton)
[00:00:19]               │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:00:19]               │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:00:19]               │ debg Waiting up to 20000ms for user menu opened...
[00:00:19]               │ debg TestSubjects.exists(userMenu)
[00:00:19]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:00:19]               │ debg TestSubjects.exists(userMenu > logoutLink)
[00:00:19]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:00:20]               │ debg TestSubjects.find(iframe_embedded)
[00:00:20]               │ debg Find.findByCssSelector('[data-test-subj="iframe_embedded"]') with timeout=10000
[00:00:20]               │ debg browser[INFO] https://localhost:6131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:20]               │
[00:00:20]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:20]               │ debg TestSubjects.exists(headerGlobalNav)
[00:00:20]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="headerGlobalNav"]') with timeout=2500
[00:00:22]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:33Z
[00:00:22]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:22]               │
[00:00:22]               │      "
[00:00:22]               │ debg --- retry.tryForTime error: [data-test-subj="headerGlobalNav"] is not displayed
[00:00:23]               │ proc [kibana]  error  [07:41:35.080] [error][client][connection] Error: 140600031983488:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
[00:00:23]               │ proc [kibana] 
[00:00:23]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional_embedded/screenshots/failure/Kibana embedded in iframe should open Kibana for logged-in user.png"
[00:00:23]               │ERROR browser[SEVERE] http://localhost:6131/api/_newsfeed-FTS-external-service-simulators/kibana/v8.0.0-SNAPSHOT.json - Failed to load resource: net::ERR_EMPTY_RESPONSE
[00:00:23]               │ERROR browser[SEVERE] https://localhost:6131/34549/bundles/plugin/newsfeed/newsfeed.plugin.js 0:19107 TypeError: Failed to fetch
[00:00:23]               │          at Fetch._callee3$ (https://localhost:6131/34549/bundles/core/core.entry.js:34:105174)
[00:00:23]               │          at l (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155138)
[00:00:23]               │          at Generator._invoke (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:154891)
[00:00:23]               │          at Generator.forEach.e.<computed> [as throw] (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155495)
[00:00:23]               │          at fetch_asyncGeneratorStep (https://localhost:6131/34549/bundles/core/core.entry.js:34:99267)
[00:00:23]               │          at _throw (https://localhost:6131/34549/bundles/core/core.entry.js:34:99675)
[00:00:23]               │ info Current URL is: https://localhost:6131/iframe_embedded
[00:00:23]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional_embedded/failure_debug/html/Kibana embedded in iframe should open Kibana for logged-in user.html
[00:00:23]               └- ✖ fail: Kibana embedded in iframe should open Kibana for logged-in user
[00:00:23]               │      Error: expected true to equal false
[00:00:23]               │       at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
[00:00:23]               │       at Assertion.be.Assertion.equal (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:227:8)
[00:00:23]               │       at Assertion.be (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:69:22)
[00:00:23]               │       at Context.it (test/functional_embedded/tests/iframe_embedded.ts:39:33)
[00:00:23]               │ 
[00:00:23]               │

Stack Trace

Error: expected true to equal false
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:69:22)
    at Context.it (test/functional_embedded/tests/iframe_embedded.ts:39:33)

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

gmmorris added a commit to gmmorris/kibana that referenced this pull request Jul 13, 2020
@gmmorris
Merge branch 'master' into alerting/consumer-based-rbac
048e769 
* master: (78 commits)
  Bump lodash package version (elastic#71392)
  refactor: 💡 use allow-list in AppArch codebase (elastic#71400)
  improve bugfix 7198 test stability (elastic#71250)
  [Security Solution][Ingest Manager][Endpoint] Optional ingest manager (elastic#71198)
  [Metrics UI] Round metric threshold time buckets to nearest unit (elastic#71172)
  [Security Solution][Endpoint] Policy creation callback fixes + Improved error handling in user manifest loop (elastic#71269)
  [Security Solution] Allow to configure Event Renderers settings (elastic#69693)
  Fix a11y keyboard overlay (elastic#71214)
  [APM] UI text updates (elastic#71333)
  [Logs UI] Limit `extendDatemath` to valid ranges (elastic#71113)
  [SIEM] fix tooltip of notes (elastic#71342)
  address index templates feedback (elastic#71353)
  Upgrade EUI to v26.3.1 (elastic#70243)
  [build] Creates Linux aarch64 archive (elastic#69165)
  [SIEM][Detection Engine] Fixes skipped tests (elastic#71347)
  [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items
  [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911)
  [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886)
  [ML] DF Analytics:  stop status polling when job stopped (elastic#71159)
  [SIEM][CASE] IBM Resilient Connector (elastic#66385)
  ...
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 15, 2020
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kertal kertal mentioned this pull request Jul 16, 2020
Closed
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

2 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

18 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@lizozom lizozom added the backport:skip This commit does not require backporting label Aug 17, 2020
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kertal kertal kertal approved these changes

Assignees

@lizozom lizozom

Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lizozom @elasticmachine @timroes @kibanamachine @kertal