[Data Discovery] Run example plugin functional tests in Serverless #163411

davismcphee · 2023-08-08T13:43:51Z

Summary

Important

These tests currently do not run in "real" Serverless and only run in "local" Serverless (what we use in CI). Additional work will need to be done to allow these tests to be run in an actual Serverless environment, since they required config.ts changes which are only loaded when running locally.

This PR copies the Data Discovery example plugin functional tests to test_serverless and adds support for running them against the Search project.

It also adds support for running functional tests against example plugins in general in Serverless ("local" Serverless only currently).

In the future these should run as deployment-agnostic tests, but support does not yet exist (see #161574), so in the meantime they've been duplicated and modified in place. I've left TODO comments where test files have been modified so we know what needs to be addressed once they are converted to deployment-agnostic tests.

Part of #162347.

Checklist

~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)~~
~~Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)~~
~~If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list~~
~~This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)~~
~~This was checked for cross-browser compatibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately

x-pack/test_serverless/functional/config.base.ts

davismcphee · 2023-08-09T19:29:09Z

x-pack/test_serverless/functional/es_archives/pre_calculated_histogram/data.json

+  "type": "doc",
+  "value": {
+    "id": "index-pattern:histogram-test",
+    "index": ".kibana_analytics_1",


When running Serverless functional tests, the index used for data view SOs is .kibana_analytics_1 instead of .kibana_analytics as in stateful, so I copied and modified this file to reflect that.

We try to avoid the usage of system indices in esArchives where possible (outside of specific SO migration tests, this often brings a lot of trouble and has been root cause of race conditions in the past). Have you considered using the kbnArchiver to save and restore this data view? Example usage here: https://github.com/elastic/kibana/blob/main/x-pack/test/functional/apps/canvas/custom_elements.ts#L28

so it seems this test should be rewritten, but this could als be a follow up PR IMO

I agree, this can be done as a follow-up.

Thanks for the info! I agree, let's do this work as a followup and fix it in both stateful and serverless.

x-pack/test_serverless/functional/page_objects/index.ts

x-pack/test_serverless/functional/test_suites/search/config.ts

elasticmachine · 2023-08-09T20:16:12Z

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

dmlemeshko

Great work. The only problem I see is that we won't be able to run these tests against projects created on MKI, meaning we won't verify functionality in the real customer environment.

x-pack/test_serverless/functional/test_suites/search/config.ts

x-pack/test_serverless/functional/test_suites/examples/config.ts

.buildkite/pipelines/pull_request/base.yml

jbudz · 2023-08-23T19:14:00Z

Can we make sure there's a followup issue or note in #162347 for de-duplicating these tests when #161574 is available?

x-pack/test_serverless/functional/test_suites/common/examples/search_examples/index.ts

davismcphee

Can we make sure there's a followup issue or note in #162347 for de-duplicating these tests when #161574 is available?

Definitely, added a separate issue for it here: #164911.

.buildkite/pipelines/pull_request/base.yml

x-pack/test_serverless/functional/config.base.ts

x-pack/test_serverless/functional/test_suites/common/examples/search_examples/index.ts

jbudz

CI changes lgtm

kertal

DataDiscovery.lgtm Code review, in buildkites testing abilities we trust 👍

dmlemeshko · 2023-08-30T07:11:05Z

I believe we should hold on with this PR until #162673 is merged. Then pull the latest main with actual serverless ES and make sure these tests are still passing.

…ins in Serverless

…fix'

…iles

…ardList

dmlemeshko

LGTM

pheyos

LGTM

kibana-ci · 2023-08-31T08:29:41Z

💚 Build Succeeded

Buildkite Build
Commit: 4686417

Metrics [docs]

✅ unchanged

History

💚 Build #155021 succeeded 88679b4
💛 Build #154679 was flaky 8d653f546059691325abb9a37dca267a6ce838d2
💚 Build #153778 succeeded a03cf4161c7445f83e6ea29d33c2310c8e317c47
💔 Build #153766 failed 53569a2009c9fbb0da4ca7bcf9eaf11308e3313d
💔 Build #153716 failed a1e5b61c21b7c46d8d65615d96fd00c8453a878c
💔 Build #152604 failed f9e5062628004a506a00b6a2f5db4ca9cbff5b29

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @davismcphee

davismcphee added release_note:skip Skip the PR/issue when compiling release notes Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. labels Aug 8, 2023

davismcphee self-assigned this Aug 8, 2023

davismcphee force-pushed the serverless-examples-tests branch 2 times, most recently from 26184a2 to 0ffdb5e Compare August 9, 2023 18:12

davismcphee commented Aug 9, 2023

View reviewed changes

davismcphee changed the title ~~[Data Discovery] Run tests against example plugins in Serverless~~ [Data Discovery] Run example plugin functional tests in Serverless Aug 9, 2023

davismcphee marked this pull request as ready for review August 9, 2023 20:16

davismcphee requested review from a team August 9, 2023 20:24

dmlemeshko reviewed Aug 11, 2023

View reviewed changes

x-pack/test_serverless/functional/test_suites/search/config.ts Outdated Show resolved Hide resolved

davismcphee force-pushed the serverless-examples-tests branch from 0ffdb5e to 5849d44 Compare August 21, 2023 23:07

davismcphee requested a review from a team as a code owner August 21, 2023 23:42

pheyos reviewed Aug 22, 2023

View reviewed changes

x-pack/test_serverless/functional/test_suites/examples/config.ts Outdated Show resolved Hide resolved

jbudz reviewed Aug 23, 2023

View reviewed changes

.buildkite/pipelines/pull_request/base.yml Show resolved Hide resolved

davismcphee mentioned this pull request Aug 23, 2023

[Data Discovery] Add Serverless functional test coverage #162347

Closed

3 tasks

kertal reviewed Aug 24, 2023

View reviewed changes

x-pack/test_serverless/functional/test_suites/common/examples/search_examples/index.ts Outdated Show resolved Hide resolved

davismcphee force-pushed the serverless-examples-tests branch from f9e5062 to a1e5b61 Compare August 25, 2023 20:13

davismcphee mentioned this pull request Aug 26, 2023

[Data Discovery] Deduplicate Serverless functional tests #164911

Open

3 tasks

davismcphee commented Aug 26, 2023

View reviewed changes

.buildkite/pipelines/pull_request/base.yml Show resolved Hide resolved

x-pack/test_serverless/functional/config.base.ts Outdated Show resolved Hide resolved

x-pack/test_serverless/functional/test_suites/common/examples/search_examples/index.ts Outdated Show resolved Hide resolved

jbudz approved these changes Aug 28, 2023

View reviewed changes

kertal approved these changes Aug 28, 2023

View reviewed changes

davismcphee force-pushed the serverless-examples-tests branch from a03cf41 to ddb9057 Compare August 29, 2023 20:17

davismcphee and others added 4 commits August 30, 2023 19:09

Initial pass at adding support for running tests against example plug…

43b3744

…ins in Serverless

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

d06495c

…fix'

Clean up Serverless example tests

d9f9a83

Add comments for differences in Serverless tests

f90b3db

davismcphee added 13 commits August 30, 2023 19:09

Add test_serverless example tests to CODEOWNERS

b3e15b6

Run search_examples example tests in test_serverless

41a1ee6

Cleanup after rebase

af4aaa2

Add 'examples' serverless test suite

483a840

Fix serverless examples tests imports

b864ce3

Fix failing tests

4fc14da

Separate Serverless examples tests into separate config.examples.ts f…

cb00844

…iles

Fix typo

5773607

Run serverless example tests on merge

222b9cf

Migrate from serverArgs to kbnServerArgs

186428e

Drop SQL and search sessions tests

165107d

Fix failing tests after rebase

5fa3265

Migrate from emptyKibanaIndex to kibanaServer.savedObjects.cleanStand…

88679b4

…ardList

davismcphee force-pushed the serverless-examples-tests branch from 8d653f5 to 88679b4 Compare August 30, 2023 22:09

Merge branch 'main' into serverless-examples-tests

4686417

dmlemeshko approved these changes Aug 31, 2023

View reviewed changes

pheyos approved these changes Aug 31, 2023

View reviewed changes

davismcphee merged commit b4bfb2a into elastic:main Aug 31, 2023

davismcphee deleted the serverless-examples-tests branch August 31, 2023 13:23

kibanamachine added v8.11.0 backport:skip This commit does not require backporting labels Aug 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data Discovery] Run example plugin functional tests in Serverless #163411

[Data Discovery] Run example plugin functional tests in Serverless #163411

davismcphee commented Aug 8, 2023 •

edited

Loading

davismcphee Aug 9, 2023

pheyos Aug 23, 2023

kertal Aug 24, 2023

pheyos Aug 24, 2023

davismcphee Aug 26, 2023

elasticmachine commented Aug 9, 2023

dmlemeshko left a comment

jbudz commented Aug 23, 2023

davismcphee left a comment

jbudz left a comment

kertal left a comment

dmlemeshko commented Aug 30, 2023

dmlemeshko left a comment

pheyos left a comment

kibana-ci commented Aug 31, 2023

[Data Discovery] Run example plugin functional tests in Serverless #163411

[Data Discovery] Run example plugin functional tests in Serverless #163411

Conversation

davismcphee commented Aug 8, 2023 • edited Loading

Summary

Checklist

For maintainers

davismcphee Aug 9, 2023

Choose a reason for hiding this comment

pheyos Aug 23, 2023

Choose a reason for hiding this comment

kertal Aug 24, 2023

Choose a reason for hiding this comment

pheyos Aug 24, 2023

Choose a reason for hiding this comment

davismcphee Aug 26, 2023

Choose a reason for hiding this comment

elasticmachine commented Aug 9, 2023

dmlemeshko left a comment

Choose a reason for hiding this comment

jbudz commented Aug 23, 2023

davismcphee left a comment

Choose a reason for hiding this comment

jbudz left a comment

Choose a reason for hiding this comment

kertal left a comment

Choose a reason for hiding this comment

dmlemeshko commented Aug 30, 2023

dmlemeshko left a comment

Choose a reason for hiding this comment

pheyos left a comment

Choose a reason for hiding this comment

kibana-ci commented Aug 31, 2023

💚 Build Succeeded

Metrics [docs]

History

davismcphee commented Aug 8, 2023 •

edited

Loading