[Security Solutions] Fixes a cypress flake by adding pipe, click, and should #92762
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FrankHassanabad
added
v8.0.0
v7.13.0
v7.12.0
Team:Detections and Resp
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
FrankHassanabad
added
release_note:skip
peluja1012
approved these changes
Feb 25, 2021
MadameSheema
approved these changes
Feb 25, 2021
💚 Build SucceededMetrics [docs]
To update your PR or re-run it, just comment with: |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Feb 25, 2021
## Summary We were seeing flake tests with the error messages: ```ts expected '<button.euiButton.euiButton--primary>' to have text 'Install 2 Elastic prebuilt rules ', but the text was 'Install 1 Elastic prebuilt rule ' ``` Running this locally several times I was able to reproduce the flake. I was able to see that the click handler when clicked on the check boxes is not always taking effect for the first check box. A common issue with Cypress is that a lot of page loads and JS activities on pages can add/remove click handlers quickly such as when we are frosting and un-frosting a loading screen. When we try and click on the click handlers for a test and the click handlers are not added yet, we miss one or more and end up with a flake test. Instead we can click as fast as possible using `pipe` and then checking that the check box is clicked before continuing using a `should` Even though the loading screen is done frosting in these tests, the click handler can take one or two milliseconds before they end up being added to the checkboxes. **Analysis** When you have the flake and fail you can scroll in Cypress and pin a point in time and see that indeed we did not get the first checkbox checked even though we had a click. Which makes sense as within a few milliseconds the click handlers are added and we do check the second checkbox: <img width="2083" alt="analysis" src="https://user-images.githubusercontent.com/1151048/109119992-a19d1b80-7702-11eb-882a-c035eba97455.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Feb 25, 2021
## Summary We were seeing flake tests with the error messages: ```ts expected '<button.euiButton.euiButton--primary>' to have text 'Install 2 Elastic prebuilt rules ', but the text was 'Install 1 Elastic prebuilt rule ' ``` Running this locally several times I was able to reproduce the flake. I was able to see that the click handler when clicked on the check boxes is not always taking effect for the first check box. A common issue with Cypress is that a lot of page loads and JS activities on pages can add/remove click handlers quickly such as when we are frosting and un-frosting a loading screen. When we try and click on the click handlers for a test and the click handlers are not added yet, we miss one or more and end up with a flake test. Instead we can click as fast as possible using `pipe` and then checking that the check box is clicked before continuing using a `should` Even though the loading screen is done frosting in these tests, the click handler can take one or two milliseconds before they end up being added to the checkboxes. **Analysis** When you have the flake and fail you can scroll in Cypress and pin a point in time and see that indeed we did not get the first checkbox checked even though we had a click. Which makes sense as within a few milliseconds the click handlers are added and we do check the second checkbox: <img width="2083" alt="analysis" src="https://user-images.githubusercontent.com/1151048/109119992-a19d1b80-7702-11eb-882a-c035eba97455.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Feb 25, 2021
* master: (38 commits) Fixes Cypress flake by adding pipe, click, and should (elastic#92762) [Discover] Fix filtering selected sidebar fields (elastic#91828) [ML] Fixes positions of calendar arrow buttons in start datafeed modal (elastic#92625) [dev/build_ts_refs] check that commit in outDirs matches mergeBase (elastic#92513) add dep on `@kbn/config` so it is built first [Expressions] [Lens] Add id and copyMetaFrom arg to mapColumn fn + add configurable onError argument to math fn (elastic#90481) [Lens] Fix Workspace hidden when using Safari (elastic#92616) [Lens] Fixes vertical alignment validation messages (elastic#91878) forbid x-elastic-product-origin header in elasticsearch configuration (elastic#92359) [Security Solution][Detections] Set default indicator path to reduce friction with new filebeat modules (elastic#92081) [ILM][Accessibility] Added A11y test for ILM new policy form. (elastic#92570) [Security Solution][Exceptions] - Fixes exceptions builder UI where invalid values can cause overwrites of other values (elastic#90634) Automatically generated Api documentation (elastic#86232) Increase index pattern select limit to 1000 (elastic#92093) [core.logging] Add RewriteAppender for filtering LogMeta. (elastic#91492) [Security Solution][Detection Rules] Update prebuilt rule threats to match schema (elastic#92281) [Security Solutions][Detection Engine] Fixes bug with not being able to duplicate indicator matches (elastic#92565) [Dashboard] Export appropriate references from byValue panels (elastic#91567) [Upgrade Assistant] Align code between branches (elastic#91862) [Security Solution][Case] Fix alerts push (elastic#91638) ...
kibanamachine
added a commit
that referenced
this pull request
Feb 25, 2021
## Summary We were seeing flake tests with the error messages: ```ts expected '<button.euiButton.euiButton--primary>' to have text 'Install 2 Elastic prebuilt rules ', but the text was 'Install 1 Elastic prebuilt rule ' ``` Running this locally several times I was able to reproduce the flake. I was able to see that the click handler when clicked on the check boxes is not always taking effect for the first check box. A common issue with Cypress is that a lot of page loads and JS activities on pages can add/remove click handlers quickly such as when we are frosting and un-frosting a loading screen. When we try and click on the click handlers for a test and the click handlers are not added yet, we miss one or more and end up with a flake test. Instead we can click as fast as possible using `pipe` and then checking that the check box is clicked before continuing using a `should` Even though the loading screen is done frosting in these tests, the click handler can take one or two milliseconds before they end up being added to the checkboxes. **Analysis** When you have the flake and fail you can scroll in Cypress and pin a point in time and see that indeed we did not get the first checkbox checked even though we had a click. Which makes sense as within a few milliseconds the click handlers are added and we do check the second checkbox: <img width="2083" alt="analysis" src="https://user-images.githubusercontent.com/1151048/109119992-a19d1b80-7702-11eb-882a-c035eba97455.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Feb 25, 2021
## Summary We were seeing flake tests with the error messages: ```ts expected '<button.euiButton.euiButton--primary>' to have text 'Install 2 Elastic prebuilt rules ', but the text was 'Install 1 Elastic prebuilt rule ' ``` Running this locally several times I was able to reproduce the flake. I was able to see that the click handler when clicked on the check boxes is not always taking effect for the first check box. A common issue with Cypress is that a lot of page loads and JS activities on pages can add/remove click handlers quickly such as when we are frosting and un-frosting a loading screen. When we try and click on the click handlers for a test and the click handlers are not added yet, we miss one or more and end up with a flake test. Instead we can click as fast as possible using `pipe` and then checking that the check box is clicked before continuing using a `should` Even though the loading screen is done frosting in these tests, the click handler can take one or two milliseconds before they end up being added to the checkboxes. **Analysis** When you have the flake and fail you can scroll in Cypress and pin a point in time and see that indeed we did not get the first checkbox checked even though we had a click. Which makes sense as within a few milliseconds the click handlers are added and we do check the second checkbox: <img width="2083" alt="analysis" src="https://user-images.githubusercontent.com/1151048/109119992-a19d1b80-7702-11eb-882a-c035eba97455.png"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Frank Hassanabad <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
We were seeing flake tests with the error messages:
Running this locally several times I was able to reproduce the flake. I was able to see that the click handler when clicked on the check boxes is not always taking effect for the first check box. A common issue with Cypress is that a lot of page loads and JS activities on pages can add/remove click handlers quickly such as when we are frosting and un-frosting a loading screen.
When we try and click on the click handlers for a test and the click handlers are not added yet, we miss one or more and end up with a flake test. Instead we can click as fast as possible using
pipeand then checking that the check box is clicked before continuing using ashouldEven though the loading screen is done frosting in these tests, the click handler can take one or two milliseconds before they end up being added to the checkboxes.
Analysis
When you have the flake and fail you can scroll in Cypress and pin a point in time and see that indeed we did not get the first checkbox checked even though we had a click. Which makes sense as within a few milliseconds the click handlers are added and we do check the second checkbox:
Checklist