Restore webdriver-based functional test runner driver #31446

spalger · 2019-02-19T01:46:53Z

Reopening #26477, we'll take another shot and this time run the tests even more often to make sure we address even more sources of flakiness before merging.

~~Note: we're running all of the functional tests 6 times per build. Hopefully this will help us see flakiness before we merge into master.~~

(originally opened as #31388, wanted branch to be on elastic/kibana so we could have separate PRs for each flaky ciGroup)

Closes #27905

…driver

spalger · 2019-02-19T01:47:23Z

test/functional/services/find.js

+      log.debug(`Find.existsByDisplayedByCssSelector('${selector}') with timeout=${timeout}`);
+      return await this.exists(async (driver) => {
+        const elements = wrapAll(await driver.findElements(By.css(selector)));
+        return await this.filterElementIsDisplayed(elements);


Filtering after the findElements() call like this means that the timeout isn't used if the findElements() call returns elements but none of them are displayed, I'm pretty sure this is the reason we're seeing flakiness like https://kibana-ci.elastic.co/job/elastic+kibana+master/66/JOB=x-pack-ciGroup6,node=immutable/testReport/junit/X-Pack%20Chromium%20Functional%20Reporting%20Tests/x-pack_test_reporting_functional_reporting%C2%B7js/reporting_app_Reporting_Dashboard_Print_PDF_button_becomes_available_when_saved/

I tried to fix this like so https://github.com/elastic/kibana/pull/31368/files#diff-6238cdd3f41d65e4fddb017713976292R209, but it caused other timeout issues, since calls to exist() were waiting for longer, and subsequent timeout calls were missing temporary elements like the success toast here:

[00:20:37] │ debg Waiting up to 20000ms for flyout closed... [00:20:37] │ debg TestSubjects.exists(dashboardAddPanel) [00:20:37] │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="dashboardAddPanel"]') with timeout=2500 [00:20:48] │ debg Find.findByCssSelector('.euiToast') with timeout=10000 ... [00:20:58] └- ✖ fail: "dashboard app using current data dashboard snapshots compare area chart snapshot" [00:20:58] │ TimeoutError: Waiting for element to be located By(css selector, .euiToast) [00:20:58] │ Wait timed out after 10033ms [00:20:58] │ at node_modules/selenium-webdriver/lib/webdriver.js:834:17 [00:20:58] │ at process._tickCallback (internal/process/next_tick.js:68:7)

https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/1118/JOB=kibana-ciGroup3,node=immutable/testReport/junit/UI%20Functional%20Tests/test_functional_apps_dashboard__dashboard_snapshots%C2%B7js/dashboard_app_using_current_data_dashboard_snapshots_compare_area_chart_snapshot/

spalger · 2019-02-19T01:47:52Z

(copied from #31388 (comment))

💔 Build Failed

continuous-integration/kibana-ci/pull-request

[00:04:06]                   │ debg --- retry.try error: unknown error: Element <button class="kuiLocalMenuItem" ng-repeat="menuItem in kbnTopNav.menuItems" aria-label="... Visualization" aria-haspopup="false" aria-expanded="false" aria-disabled="false" ng-class="{'kuiLocalMenuItem-isSelected': kbnTopNav.isCurrent(menuItem.key), 'kuiLocalMenuItem-isDisabled': menuItem.disableButton()}" ng-click="kbnTopNav.handleClick(menuItem, $event)" ng-bind="menuItem.label" tooltip="" tooltip-placement="bottom" tooltip-popup-delay="400" tooltip-append-to-body="1" data-test-subj="visualizeSaveButton">Save</button> is not clickable at point (73, 67). Other element would receive the click: <div class="euiOverlayMask">...</div>

https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/1149/JOB=x-pack-ciGroup6-2,node=immutable/testReport/junit/X-Pack%20Chromium%20Functional%20Reporting%20Tests/x-pack_test_reporting_functional_reporting%C2%B7js/reporting_app_Reporting_Visualize_Print_PDF_button_becomes_available_when_saved/

spalger · 2019-02-19T01:48:10Z

(copied from #31388 (comment))

💔 Build Failed

continuous-integration/kibana-ci/pull-request

1 failure in x-pack-ciGroup7

[00:09:41]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="globalLoadingIndicator"]') with timeout=2500
[00:09:44]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:44]                   │ debg Find.existsByCssSelector('[data-test-subj~="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:44]                   │ debg TestSubjects.click(mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time)
[00:09:44]                   │ debg Find.clickByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
[00:09:44]                   │ debg Find.findByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
[00:09:54]                   │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:09:54]                   │      Wait timed out after 10026ms
[00:09:54]                   │ debg Find.findByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
...
[00:11:50]                   │ debg --- retry.try error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:11:50]                   │      Wait timed out after 10043ms
[00:11:50]                   │          at /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/node_modules/selenium-webdriver/lib/webdriver.js:834:17
[00:11:50]                   │          at process._tickCallback (internal/process/next_tick.js:68:7)
...
[00:11:51]                   └- ✖ fail: "maps app  maps loaded from sample data flights "before all" hook"
[00:11:51]                   │      Error: retry.try timeout: Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:11:51]                   │ Wait timed out after 10043ms
[00:11:51]                   │     at /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/node_modules/selenium-webdriver/lib/webdriver.js:834:17
[00:11:51]                   │     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:11:51]                   │     at lastError (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:29:9)
[00:11:51]                   │     at retryForSuccess (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:63:13)
[00:11:51]                   │       at lastError (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:29:9)
[00:11:51]                   │       at retryForSuccess (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:63:13)

spalger · 2019-02-19T01:48:28Z

(copied from #31388 (comment))

💔 Build Failed

continuous-integration/kibana-ci/pull-request

1 Failure in x-pack-ciGroup6

This one appears to be related to the issue described in #31388 (comment)

# note times, the timeout was basically ignored, it failed "instantly"
[00:01:36]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="saveDashboardSuccess"]') with timeout=120000
...
[00:01:37]                   └- ✖ fail: "reporting app Reporting Dashboard Print PDF button becomes available when saved"
[00:01:37]                   │      Error: expected testSubject(saveDashboardSuccess) to exist
[00:01:37]                   │       at TestSubjects.existOrFail (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup6-2/node/immutable/kibana/test/functional/services/test_subjects.js:53:15)
[00:01:37]                   │       at process._tickCallback (internal/process/next_tick.js:68:7)

elasticmachine · 2019-02-19T02:32:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-02-19T07:31:53Z

retest

elasticmachine · 2019-02-19T08:15:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-02-19T08:18:46Z

Retest

elasticmachine · 2019-02-19T08:59:39Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-02-19T11:30:15Z

retest

elasticmachine · 2019-02-19T12:14:10Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

dmlemeshko · 2019-02-26T13:53:56Z

retest

elasticmachine · 2019-02-26T14:41:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-02-26T15:53:39Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

dmlemeshko · 2019-02-26T16:04:10Z

1 failure which is expected:

X-Pack Chromium Functional Reporting Tests.x-pack/test/reporting/functional/reporting·js.reporting app Reporting Dashboard Print PNG button becomes available when saved

PR #31447 has a fix for it.

* rerun x-pack-ciGroup6 40 times * improve reporting tests stability (#31454)

cuff-links

I went through this pretty finely. I think since we merged a bunch of smaller PRs into this branch, that helped this branch to be relatively easy to review. I also pulled this locally and ran the tests x 3 (using TEST_THROTTLE_NETWORK and got 3 green builds. That's amazing! I approve these changes and all they stand for!

elasticmachine · 2019-03-01T21:35:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-03-01T22:00:25Z

retest

elasticmachine · 2019-03-01T22:51:55Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-03-01T23:15:40Z

retest

spalger · 2019-03-02T00:09:17Z

retest

dmlemeshko · 2019-03-02T00:27:45Z

1 job failed due to CI (probably )overload.

00:22:15 $ node ./preinstall_check
00:22:15 [1/5] Validating package.json...
00:22:15 [2/5] Resolving packages...
00:22:16 [3/5] Fetching packages...
00:22:20 FATAL: command execution failed
00:22:20 java.nio.channels.ClosedChannelException
00:22:20 	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
00:22:20 	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
00:22:20 	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
00:22:20 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
00:22:20 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
00:22:20 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
00:22:20 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
00:22:20 	at java.lang.Thread.run(Thread.java:748)
00:22:20 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from kibana-ci-immutable-ubuntu-1551480492822590185.c.elastic-ci-prod.internal/10.224.1.167:38462' is disconnected.

spalger · 2019-03-02T01:04:31Z

Come on Jenkins, last one! https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/3281/

elasticmachine · 2019-03-02T01:04:48Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…flakiness" This reverts commit 4770993.

elasticmachine · 2019-03-02T01:51:11Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

dmlemeshko · 2019-03-09T18:47:56Z

Backported
7.0 53ecc0b
7.x 34123b0

spalger and others added 4 commits February 17, 2019 23:18

Restore webdriver-based functional test runner driver

7127c40

run all function test suites 6 times per build, to help spot flakiness

4770993

[tests/reporting] fix flaky click on report generation

dbdbab2

Merge remote-tracking branch 'upstream/master' into implement/ftr-web…

e92784a

…driver

spalger added WIP Work in progress test_ui_functional test_xpack_functional labels Feb 19, 2019

spalger commented Feb 19, 2019

View reviewed changes

This was referenced Feb 19, 2019

Restore webdriver-based functional test runner driver #31388

Closed

[#31446] stabilize x-pack-ciGroup6 #31447

Merged

[#31446] stabilize x-pack-ciGroup7 #31448

Closed

Merge branch 'master' into implement/ftr-webdriver

256a4f1

[#31446] stabilize x-pack-ciGroup6 (#31447)

a823c9b

* rerun x-pack-ciGroup6 40 times * improve reporting tests stability (#31454)

cuff-links approved these changes Mar 1, 2019

View reviewed changes

cuff-links mentioned this pull request Mar 1, 2019

Create index pattern sometimes the asterisk is not placed at the end #16793

Open

This comment has been minimized.

Sign in to view

Revert "run all function test suites 6 times per build, to help spot …

1d34d9f

…flakiness" This reverts commit 4770993.

spalger removed the WIP Work in progress label Mar 2, 2019

spalger merged commit a6afcc5 into master Mar 2, 2019

cuff-links mentioned this pull request Mar 2, 2019

WebDriver is stuck without logs and fails on mocha timeout #29550

Closed

dmlemeshko mentioned this pull request Mar 4, 2019

WebDriver FTR flaky tests #32385

Closed

3 tasks

dmlemeshko added a commit to dmlemeshko/kibana that referenced this pull request Mar 4, 2019

Restore webdriver-based functional test runner driver (elastic#31446)

0253feb

spalger deleted the implement/ftr-webdriver branch March 5, 2019 15:53

dmlemeshko pushed a commit to dmlemeshko/kibana that referenced this pull request Mar 6, 2019

Restore webdriver-based functional test runner driver (elastic#31446)

dd10494

dmlemeshko mentioned this pull request Mar 6, 2019

[7.x] Restore webdriver-based functional test runner driver (#31446) #32554

Merged

dmlemeshko added a commit that referenced this pull request Mar 6, 2019

Restore webdriver-based functional test runner driver (#31446) (#32554)

34123b0

spalger mentioned this pull request Mar 6, 2019

Typescript-ify FTR Remote #32447

Merged

dmlemeshko pushed a commit to dmlemeshko/kibana that referenced this pull request Mar 9, 2019

Restore webdriver-based functional test runner driver (elastic#31446)

d98d3e0

dmlemeshko mentioned this pull request Mar 9, 2019

[7.0] Restore webdriver-based functional test runner driver (#31446) #32846

Merged

dmlemeshko added v8.0.0 v7.2.0 v7.0.0 labels Mar 9, 2019

dmlemeshko added a commit that referenced this pull request Mar 9, 2019

Restore webdriver-based functional test runner driver (#31446) (#32846)

53ecc0b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore webdriver-based functional test runner driver #31446

Restore webdriver-based functional test runner driver #31446

spalger commented Feb 19, 2019 •

edited

Loading

spalger Feb 19, 2019

spalger commented Feb 19, 2019

spalger commented Feb 19, 2019 •

edited

Loading

spalger commented Feb 19, 2019 •

edited

Loading

elasticmachine commented Feb 19, 2019

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

dmlemeshko commented Feb 26, 2019

elasticmachine commented Feb 26, 2019

elasticmachine commented Feb 26, 2019

dmlemeshko commented Feb 26, 2019

cuff-links left a comment

elasticmachine commented Mar 1, 2019

spalger commented Mar 1, 2019

elasticmachine commented Mar 1, 2019

spalger commented Mar 1, 2019

spalger commented Mar 2, 2019

This comment has been minimized.

dmlemeshko commented Mar 2, 2019

spalger commented Mar 2, 2019

elasticmachine commented Mar 2, 2019

elasticmachine commented Mar 2, 2019

dmlemeshko commented Mar 9, 2019

Restore webdriver-based functional test runner driver #31446

Restore webdriver-based functional test runner driver #31446

Conversation

spalger commented Feb 19, 2019 • edited Loading

spalger Feb 19, 2019

Choose a reason for hiding this comment

spalger commented Feb 19, 2019

💔 Build Failed

spalger commented Feb 19, 2019 • edited Loading

💔 Build Failed

spalger commented Feb 19, 2019 • edited Loading

💔 Build Failed

elasticmachine commented Feb 19, 2019

💚 Build Succeeded

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

💚 Build Succeeded

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

💚 Build Succeeded

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

💚 Build Succeeded

dmlemeshko commented Feb 26, 2019

elasticmachine commented Feb 26, 2019

💚 Build Succeeded

elasticmachine commented Feb 26, 2019

💔 Build Failed

dmlemeshko commented Feb 26, 2019

cuff-links left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 1, 2019

💚 Build Succeeded

spalger commented Mar 1, 2019

elasticmachine commented Mar 1, 2019

💚 Build Succeeded

spalger commented Mar 1, 2019

spalger commented Mar 2, 2019

This comment has been minimized.

dmlemeshko commented Mar 2, 2019

spalger commented Mar 2, 2019

elasticmachine commented Mar 2, 2019

💚 Build Succeeded

elasticmachine commented Mar 2, 2019

💚 Build Succeeded

dmlemeshko commented Mar 9, 2019

spalger commented Feb 19, 2019 •

edited

Loading

spalger commented Feb 19, 2019 •

edited

Loading

spalger commented Feb 19, 2019 •

edited

Loading