[FTR] Support for deployment agnostic e2e tests #161574
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@pheyos thanks for working on this! Would it be possible to use the serverless config in a way that allows us to keep the test suite where it currently is? |
This was referenced Aug 9, 2023
tonyghiani
added a commit
that referenced
this pull request
Aug 11, 2023
## 📓 Summary Closes #160627 This implementation adds the majority of the tests listed down here for the Log Explorer current implementation. ✅ [**Flaky Test Runner - x50 executions**](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2844#_) ``` ↳ Discover Log-Explorer profile ↳ Columns selection initialization and update ↳ when the log explorer profile loads ↳ should initialize the table columns to logs' default selection ↳ should restore the table columns from the URL state if exists ↳ Customizations ↳ when Discover is loaded with the log-explorer profile ↳ DatasetSelector should replace the DataViewPicker ↳ the TopNav bar should hide the New, Open and Save options ↳ should add a searchable deep link to the profile page ↳ should render a filter controls section as part of the unified search bar ↳ DatasetSelection initialization and update ↳ when the "index" query param does not exist ↳ should initialize the "All log datasets" selection ↳ when the "index" query param exists ↳ should decode and restore the selection from a valid encoded index ↳ should fallback to the "All log datasets" selection and notify the user of an invalid encoded index ↳ when navigating back and forth on the page history ↳ should decode and restore the selection for the current index ↳ Dataset Selector ↳ without installed integrations or uncategorized data streams ↳ when open on the first navigation level ↳ should always display the "All log datasets" entry as the first item ↳ should always display the unmanaged datasets entry as the second item ↳ should display an error prompt if could not retrieve the integrations ↳ should display an empty prompt for no integrations ↳ when navigating into Uncategorized data streams ↳ should display a loading skeleton while loading ↳ should display an error prompt if could not retrieve the data streams ↳ should display an empty prompt for no data streams ↳ with installed integrations and uncategorized data streams ↳ when open on the first navigation level ↳ should always display the "All log datasets" entry as the first item ↳ should always display the unmanaged datasets entry as the second item ↳ should display a list of installed integrations ↳ should sort the integrations list by the clicked sorting option ↳ should filter the integrations list by the typed integration name ↳ should display an empty prompt when the search does not match any result ↳ should load more integrations by scrolling to the end of the list ↳ when clicking on integration and moving into the second navigation level ↳ should display a list of available datasets ↳ should sort the datasets list by the clicked sorting option ↳ should filter the datasets list by the typed dataset name ↳ should update the current selection with the clicked dataset ↳ when navigating into Uncategorized data streams ↳ should display a list of available datasets ↳ should sort the datasets list by the clicked sorting option ↳ should filter the datasets list by the typed dataset name ↳ should update the current selection with the clicked dataset ↳ when open/close the selector ↳ should restore the latest navigation panel ↳ should restore the latest search results ↳ when switching between integration panels ↳ should remember the latest search and restore its results for each integration ``` ## Note on serverless tests suite For testing the feature in a serverless environment, we are copying all the tests into the `x-pack/test_serverless` folder until #161574 is merged, which will provide a new space to write tests independently from the deployment type, avoiding then tests duplication. ## New `browser` service utils for Network conditions simulation To properly test that this feature works correctly under poor network conditions or offline scenarios, the `browser` service now exposes some new methods for altering network conditions on demand. Also, network profiles to match the [network profiles provided by Chrome debugger](https://github.com/ChromeDevTools/devtools-frontend/blob/da276a3faec9769cb55e442f0db77ebdce5cd178/front_end/core/sdk/NetworkManager.ts#L363-L393) have been created. In case the browser is not of `chromium` type and the driver does not support the network simulation, these methods throw an error that can be caught for skipping the affected test. --------- Co-authored-by: Marco Antonio Ghiani <[email protected]> Co-authored-by: kibanamachine <[email protected]>
3 tasks
davismcphee
added a commit
that referenced
this pull request
Aug 29, 2023
## Summary This PR copies the Data Discovery API integration tests to `test_serverless`. They are currently in the common folder and will run once for each project type in CI, but we could instead move them to a specific project if we don't want to run them three times each. In the future these should run as deployment-agnostic tests, but support does not yet exist (see #161574), so in the meantime they've been duplicated and modified in place. I've left `TODO` comments where test files have been modified so we know what needs to be addressed once they are converted to deployment-agnostic tests. Part of #162347. ### Checklist - [ ] ~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)~ - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] ~Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))~ - [ ] ~Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))~ - [ ] ~If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)~ - [ ] ~This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))~ - [ ] ~This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)~ ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
davismcphee
added a commit
that referenced
this pull request
Aug 31, 2023
…163411) ## Summary > [!IMPORTANT] > These tests currently do not run in "real" Serverless and only run in "local" Serverless (what we use in CI). Additional work will need to be done to allow these tests to be run in an actual Serverless environment, since they required `config.ts` changes which are only loaded when running locally. This PR copies the Data Discovery example plugin functional tests to `test_serverless` and adds support for running them against the Search project. It also adds support for running functional tests against example plugins in general in Serverless ("local" Serverless only currently). In the future these should run as deployment-agnostic tests, but support does not yet exist (see #161574), so in the meantime they've been duplicated and modified in place. I've left `TODO` comments where test files have been modified so we know what needs to be addressed once they are converted to deployment-agnostic tests. Part of #162347. ### Checklist - [ ] ~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)~ - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] ~Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))~ - [ ] ~Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))~ - [ ] ~If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)~ - [ ] ~This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))~ - [ ] ~This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)~ ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Dzmitry Lemechko <[email protected]>
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
To update your PR or re-run it, just comment with: cc @pheyos |
|
@pheyos is there any update on this? Is the current recommendation to still copy/paste tests to run in both stateful and serverless envs? |
|
@jasonrhodes no update on this and the recommendation is still to copy the tests that should run in stateful and serverless. There is planning in progress around test framework enhancements and this topic is definitely on the list of requirements. |
13 tasks
|
Implemented in #189853 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the initial boilerplate code for e2e tests (functional UI + API integration) that are deployment agnostic, so they can run the same on local, ESS and serverless.
Details
This is WIP, more details to add over time.