Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM] Cypress e2e: Flaky test read_only_user/service_overview/time_comparison.spec.ts #109205

Closed
sorenlouv opened this issue Aug 19, 2021 · 1 comment · Fixed by #114831
Closed

[APM] Cypress e2e: Flaky test read_only_user/service_overview/time_comparison.spec.ts #109205

sorenlouv opened this issue Aug 19, 2021 · 1 comment · Fixed by #114831
Assignees
@smith
Labels
Team:APM All issues that need APM UI Team support technical debt Improvement of the software architecture and operational architecture

Comments

@sorenlouv
Copy link
Member

The test read_only_user/service_overview/time_comparison.spec.ts fails intermittently (2 out of 100 runs) with this error:

  1) Service overview: Time Comparison
       changes comparison type when a new time range is selected:

      Timed out retrying after 30000ms
      + expected - actual

      -'week'
      +'day'

Reproduce
Since it only happens rarely it is necessary to run the tests many times:

for i in {1..100}; do node scripts/build_kibana_platform_plugins && node x-pack/plugins/apm/scripts/ftr_e2e/cypress_run.js && sleep 60; done
@sorenlouv sorenlouv added [zube]: Inbox Team:APM All issues that need APM UI Team support labels Aug 19, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:apm)

@sorenlouv sorenlouv changed the title [APM] Flaky test [APM] Cypress e2e: Flaky test read_only_user/service_overview/time_comparison.spec.ts Aug 19, 2021
@sorenlouv sorenlouv added [zube]: 7.16 technical debt Improvement of the software architecture and operational architecture and removed [zube]: Inbox labels Aug 24, 2021
@smith smith mentioned this issue Oct 12, 2021
Closed
@smith smith self-assigned this Oct 13, 2021
smith added a commit to smith/kibana that referenced this issue Oct 13, 2021
@smith
Re-enable and fix APM E2E tests
45ecc74 
* Run previously disabled APM E2E tests on all PRs (we were previously only running them when APM files had changes.)
* Remove `precise: true` from `getComparisonTypes` call which caused intermittent failures in date comparison tests.
* Simplify error count alert tests to test the "happy path" (elastic#79284 exists in order to expand to more tests for rule editing and creation)
* Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI.

I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed.

Fixes elastic#114419.
Fixes elastic#109205.
@smith smith mentioned this issue Oct 13, 2021
Merged
smith added a commit that referenced this issue Oct 15, 2021
@smith
Re-enable and fix APM E2E tests (#114831)
07777b9 
* Re-enable previously disabled APM E2E tests.
* Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown.
* Simplify error count alert tests to test the "happy path" (#79284 exists in order to expand to more tests for rule editing and creation.)
* Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI.

I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed.

Fixes #114419.
Fixes #109205.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 15, 2021
@smith @kibanamachine
Re-enable and fix APM E2E tests (elastic#114831)
41ce557 
* Re-enable previously disabled APM E2E tests.
* Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown.
* Simplify error count alert tests to test the "happy path" (elastic#79284 exists in order to expand to more tests for rule editing and creation.)
* Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI.

I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed.

Fixes elastic#114419.
Fixes elastic#109205.
kibanamachine added a commit that referenced this issue Oct 15, 2021
@kibanamachine @smith
Re-enable and fix APM E2E tests (#114831) (#115251)
ec9140b 
* Re-enable previously disabled APM E2E tests.
* Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown.
* Simplify error count alert tests to test the "happy path" (#79284 exists in order to expand to more tests for rule editing and creation.)
* Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI.

I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed.

Fixes #114419.
Fixes #109205.

Co-authored-by: Nathan L Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smith smith

Labels
Team:APM All issues that need APM UI Team support technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enable and fix APM E2E tests smith/kibana
5 participants
@smith @sorenlouv @ogupte @elasticmachine @cauemarcondes