Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Defend Workflows] Fix artifact entries list FTR #189961

Merged
merged 20 commits into from
Aug 27, 2024
Merged

[Defend Workflows] Fix artifact entries list FTR #189961

merged 20 commits into from
Aug 27, 2024

Conversation

gergoabraham
Copy link
Contributor

@gergoabraham gergoabraham commented Aug 6, 2024 •
edited
Loading

Summary

tl;wr;

  • instead of selecting from dropdown, values are entered for Event Filters
  • artifacts for update/delete test cases are created via API call instead of UI interaction

Note

FTRs needs to be re-enabled before merge, revert this e0858a5 - done ✅

This PR attempts to decrease flakiness in artifact_entries_list FTR. Looks like the reason for flakiness was the dropdown for the field to be filtered. After spending a bit of a time looking at the code, couldn't find any quick solution why it can be flaky, so instead just tried to improve the tests. For that, added a 'special' (aka hack) test case to just try to set event filter 100 times, and also a failing test case just to receive logs (77455d7) - that's why all flaky runners are red, you need to see the logs.

It didn't work to add some waits to let React re-render and let the API call for suggestions be finished, didn't work to select from dropdown by text instead of test id, didn't work to add even more wait in between multiple user actions. See logs here and search for results are in inside the logs: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6744#019145e0-445d-4978-89c8-701908e14e3f
image

In the end, looks like the solution is simply input the values instead of selecting them from the dropdown. In the last hacky-flaky run, it passed all 100 times: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6756
image

And a last flaky run without the hack, with the official flaky runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6772 100/100 ✅

Also, entries for the update/delete test cases are created via API call instead of UI interaction, so those cases overlap less with the create test case.

@gergoabraham gergoabraham added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team:Defend Workflows “EDR Workflows” sub-team of Security Solution labels Aug 6, 2024
@gergoabraham gergoabraham self-assigned this Aug 6, 2024
@gergoabraham
Copy link
Contributor Author

/ci

@gergoabraham gergoabraham force-pushed the fix-artifact-entries-list-ftr branch from b4212a1 to 7a12578 Compare August 9, 2024 13:38
@gergoabraham
Copy link
Contributor Author

/ci

1 similar comment
@gergoabraham
Copy link
Contributor Author

/ci

@gergoabraham
Copy link
Contributor Author

special run: runs 3 times, but in the modified test, 3 solutions are performed 100 times each
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6741

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6741

[❌] x-pack/test/security_solution_endpoint/configs/integrations.config.ts: 0/3 tests passed.
[❌] x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts: 0/3 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6744

[❌] x-pack/test/security_solution_endpoint/configs/integrations.config.ts: 0/3 tests passed.
[❌] x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts: 0/3 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6747

[❌] x-pack/test/security_solution_endpoint/configs/integrations.config.ts: 0/1 tests passed.
[❌] x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts: 0/1 tests passed.

see run history

@gergoabraham gergoabraham force-pushed the fix-artifact-entries-list-ftr branch from 08310d3 to d39dbbf Compare August 12, 2024 14:52
@gergoabraham
Copy link
Contributor Author

/ci

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#6748

[❌] x-pack/test/security_solution_endpoint/configs/integrations.config.ts: 0/1 tests passed.
[❌] x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts: 0/1 tests passed.

see run history

@gergoabraham
Copy link
Contributor Author

/ci

@gergoabraham
Copy link
Contributor Author

/ci

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6772

[✅] x-pack/test/security_solution_endpoint/configs/integrations.config.ts: 50/50 tests passed.
[✅] x-pack/test/security_solution_endpoint/configs/serverless.integrations.config.ts: 50/50 tests passed.

see run history

@gergoabraham gergoabraham marked this pull request as ready for review August 15, 2024 08:45
@gergoabraham gergoabraham requested a review from a team as a code owner August 15, 2024 08:45
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@gergoabraham gergoabraham mentioned this pull request Aug 15, 2024
Merged
@gergoabraham
Copy link
Contributor Author

@elasticmachine merge upstream

@gergoabraham
Copy link
Contributor Author

@elasticmachine merge upstream

gergoabraham added a commit that referenced this pull request Aug 19, 2024
@gergoabraham
[EDR Workflows] Improve event filters related cy tests (#190610)
e7aabcd 
## Summary

reducing potential flakyness in cypress tests handling event filters, by
applying same change as #189961: enter text instead of selecting from
dropdown

> [!note]
> revert da247c5 before merge - done ✅ 

flaky runner:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6782
✅
@gergoabraham
Copy link
Contributor Author

@elasticmachine merge upstream

@gergoabraham
Copy link
Contributor Author

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @gergoabraham

tomsonpl
tomsonpl approved these changes Aug 27, 2024
View reviewed changes
Copy link
Contributor

@tomsonpl tomsonpl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@gergoabraham gergoabraham merged commit 8e9b827 into elastic:main Aug 27, 2024
21 checks passed
@gergoabraham gergoabraham deleted the fix-artifact-entries-list-ftr branch August 27, 2024 14:50
This was referenced Aug 27, 2024
Closed
Closed
Closed
Closed
Closed
Closed
@dasansol92 dasansol92 mentioned this pull request Sep 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tomsonpl tomsonpl tomsonpl approved these changes

@parkiino parkiino Awaiting requested review from parkiino parkiino is a code owner automatically assigned from elastic/security-defend-workflows

Assignees

@gergoabraham gergoabraham

Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.16.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gergoabraham @kibanamachine @elasticmachine @kibana-ci @tomsonpl