Run Kibana tests on Cloud meta issue

[LeeDr]

Describe the feature: We need to be able to run Kibana tests against Elastic Cloud deployments.

• Need an API to create an Elasticsearch and Kibana instance on Cloud https://github.com/elastic/cloud/issues/12821
• Support 1 Kibana and 1 Elasticsearch URL as input params Support 1 Kibana and 1 Elasticsearch URL as input params #9760
• Kibana tests can fail on Cloud if deleting index during snapshotting Kibana tests can fail on Cloud if deleting index during snapshotting #17416
• Need a way to exclude some tests that require settings we can't get on Cloud. For example, we set '--server.maxPayloadBytes=1648576', //default is 1048576 in tasks/config/run.js but I don't think we can tweak that kibana setting on Elastic Cloud so we should skip test test\functional\apps\management\_test_huge_fields.js Add option to exclude tests from cloud run #19228
• Create kibana jenkins cloud shell script Add a way to run Kibana tests from Jenkins for cloud  #19229
• Bad selectors causing failures Update test selectors to improve test flakiness  #19679
• Loading data and other parts of tests are slower on Cloud so we need to increase the timeout. Think this setting which is currently 3 minutes needs to increase to 4 or 5 minutes;
  https://github.com/elastic/kibana/blob/master/src/functional_test_runner/lib/config/schema.js#L66
• There is an existing Kibana issue where the play button doesn't appear in the Dev Tools > Console. We could work-around it by either re-arranging the order of tests in config.js so console isn't first, or by adding a page refresh Dev Tools Console sometimes doesn't show the Play arrow button #13232

[tylersmalley]

It would be great if we could have ECE available on CI with the nightly snapshots of ES. This was we could always use it to spin up clusters. I assume it would be faster/cleaner than running from source, which is what we do now.

[LeeDr]

Snapshots are being deployed nightly now on staging Elastic Cloud. But we don't have the API yet to automate creating an Elasticsearch node/cluster. But they're working on it.

Using an already started Cloud Elasticsearch node would certainly save time starting up Elasticsearch as part of our CI jobs. But we couldn't let multiple CI jobs hit the same cluster at the same time because our tests delete the .kibana index all the time.

• The Kibana CI jobs could use the API to spin up an elasticsearch node just for this job and then destroy it. But I'm not sure if that would be any faster than what we do now.
• We could have one Cloud Elasticsearch node and have our tests use a unique .kibana index for each job. But we'd still have a problem when one test loads logstash-* data and another test unloads it.
• We could have a pool of Cloud Elasticsearch nodes and have our tests somehow select a "free" one.

[suyograo]

The best approach is to use the API to spin up an ES node and tear it down after the test is run (success or failure). We'll need to capture logs/state when a test fails to debug why the test failed. For now, its even ok to leave the cluster running if the tests failed.

[liza-mae]

So far issues observed when trying to run kibana tests on a cloud instance:

1- Selectors are not being found. It is unclear yet if it is timing and/or just bad selectors. I did notice some of the selectors are using nth-child, which is fragile, we should look into adding data-test-subj instead for these tests.
2- Elements are not being clicked even though they are visible.
3- Tests have sleeps which cause some flaky, should have either retries or event based

(Note: The kibana tests are dependent on everything contained in the file, so tests may not run individually from a file if they depend on the previous test for data)

[liza-mae]

I don't believe issue #13232 listed in the description is related to cloud testing, so I am going to remove it from this meta issue.

[liza-mae]

To address the issue of selectors not being found, I have opened #19679, these deal with failures not with the use of .ng-scope which should be removed. We need to move towards data-test-subj.

[liza-mae]

Addressing #19679 has eliminated all but 12 or so failures for me on cloud testing. So I think that is a good start for 6.2.4 baseline.

[liza-mae]

Here are my results and failure analysis:

6.2.4 Baseline Results

Summary
297 passing (40m)
3 pending
12 failing

Failure Analysis

• dashboard app dashboard queries Pie chart attached to saved search filters data as expected
• no such element: Unable to locate element: {"method":"css selector","selector":"[group-name="buckets"] vis-editor-agg-params:not(.ng-hide) .agg-select"
• dashboard app dashboard queries filters "before all" hook for "Pie chart attached to saved - search filters shows no data with conflicting dashboard query
• Unable to locate element: {"method":"css selector","selector":"a[href="#/dashboards"]"}
• dashboard app dashboard snapshots compare area chart snapshot
• Error: expected 0.1490841968911917 to be below 0.05
• management index result field sort sort by heading - name should sort ascending
• no such element: Unable to locate element: {"method":"css selector","selector":"div.agg-table-paginated table.table.table-condensed tbody tr:nth-child(1) td:nth-child(1) span.ng-binding"}
• management index result field sort sort by heading - name should sort descending
• no such element: Unable to locate element: {"method":"css selector","selector":"div.agg-table-paginated table.table.table-condensed tbody tr:nth-child(1) td:nth-child(1) span.ng-binding"}
• management index result field sort sort by heading - type should sort ascending
• no such element: Unable to locate element: {"method":"css selector","selector":"div.agg-table-paginated table.table.table-condensed tbody tr:nth-child(1) td:nth-child(2) span.ng-binding"}
• management index result field sort sort by heading - type should sort descending
• no such element: Unable to locate element: {"method":"css selector","selector":"div.agg-table-paginated table.table.table-condensed tbody tr:nth-child(1) td:nth-child(2) span.ng-binding"}
• timelion app expression typeahead dynamic suggestions for argument values .es() should show field suggestions for split argument when index pattern set
• Error: expected 1 to sort of equal 52
• timelion app expression typeahead dynamic suggestions for argument values .es() should show field suggestions for metric argument when index pattern set
• TypeError: Cannot read property 'click' of undefined
• visualize app visualize app Visual Builder markdown should allow printing raw value of data
• Error: expected '61442901600000' to equal '6'
• visualize app visualize app Visual Builder markdown allow time offsets allow positive time offsets
• Error: expected '331442901600000' to equal '3'
• visualize app visualize app Visual Builder markdown allow time offsets allow negative time offsets
• Error: expected '23231442901600000' to equal '23'

[liza-mae]

The visualize app failures are mac platform specific, so we only have 9 more to go

[liza-mae]

The Kibana parts of this are now complete.

1. I baselined version 6.3.2, running locally and debugging tests.
2. Latest results, all tests passed locally after debugging and fixing may test failures: https://drive.google.com/open?id=1pAvR6cN6ohlmnxf7ozWv-7i-I-hjYulT
3. A Jenkins script has been created to run the kibana tests
   The remaining done through infrastructure work in ESTF project.