-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[APM] Run APM e2e test uncondtionally #114419
Comments
Pinging @elastic/apm-ui (Team:apm) |
APM E2E was disabled again in #114544. |
The latest failures don't seem to be from the issue with Hapi from before, but a failure on the error count test. Looking into this. |
This also fails intermittently: #109205. Will try to fix in the PR to fix this issue. |
* Run previously disabled APM E2E tests on all PRs (we were previously only running them when APM files had changes.) * Remove `precise: true` from `getComparisonTypes` call which caused intermittent failures in date comparison tests. * Simplify error count alert tests to test the "happy path" (elastic#79284 exists in order to expand to more tests for rule editing and creation) * Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI. I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed. Fixes elastic#114419. Fixes elastic#109205.
I'm re-enabling them but not running them unconditionally. @spalger wrote a comment on the PR:
In a follow-up conversation in Slack:
Since running Cypress tests unconditionally is not supported, we are still open to the failure this issue was created to address, so our options are either:
For now we're continuing to use Cypress and are aware of the risk. Closing this issue. |
* Re-enable previously disabled APM E2E tests. * Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown. * Simplify error count alert tests to test the "happy path" (#79284 exists in order to expand to more tests for rule editing and creation.) * Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI. I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed. Fixes #114419. Fixes #109205.
* Re-enable previously disabled APM E2E tests. * Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown. * Simplify error count alert tests to test the "happy path" (elastic#79284 exists in order to expand to more tests for rule editing and creation.) * Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI. I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed. Fixes elastic#114419. Fixes elastic#109205.
* Re-enable previously disabled APM E2E tests. * Round to the nearest second in `getComparisonTypes` to avoid cases where a millisecond difference can change which results get shown. * Simplify error count alert tests to test the "happy path" (#79284 exists in order to expand to more tests for rule editing and creation.) * Wait for alert list API request to complete before clicking "Create rule" button when running the test to create a rule from the Stack Management UI. I ran the e2e tests 100 times locally with no failures so I'm confident the flakiness has been addressed. Fixes #114419. Fixes #109205. Co-authored-by: Nathan L Smith <[email protected]>
In #113618 we had to disable APM's e2e test because they had started to fail. The cause was an update of hapi was upgraded to 20.2.0 which included a bug.
In #114414 hapi will be upgraded again with a patch that should fix APM's e2e's.
To avoid getting in this situation again we should run the e2e's unconditionally - and not only when APM code is changed.
This can be done by removing the
whenChanged
check:kibana/vars/tasks.groovy
Lines 149 to 155 in 8a80c85
The text was updated successfully, but these errors were encountered: