Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] [Tests] Determine true flakes / failures #161531

Closed
38 of 40 tasks
WafaaNasr opened this issue Jul 10, 2023 · 1 comment
Closed
38 of 40 tasks

[Security Solution] [Tests] Determine true flakes / failures #161531

WafaaNasr opened this issue Jul 10, 2023 · 1 comment
Assignees
@e40pud
Labels
Team:Detection Engine Security Solution Detection Engine Area test-failure-flaky

Comments

@WafaaNasr
Copy link
Contributor

WafaaNasr commented Jul 10, 2023 •
edited
Loading

Steps:

  1. Going through these tickets

  2. For each ticket determine whether it is a true flaky or failing test by running the Flaky runner tool 100 times

  3. Follow the Flaky test runner for Cypress

  4. If the test wasn't flaky, close it and link the respective flaky test runner link

Note: To run the flaky runner for a specific test, a PR should be present, for example in order to run a test against version 8.8 this PR has been created 8.8 pr
Main PR

FTR groups

Group2
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_endpoint_exceptions·ts - detection engine api security and spaces enabled Rule exception operators for endpoints operating system types (os_types) agent should filter 1 operating system types (os_type) if it is set as part of an endpoint exception #111855
    Team:Detections and Resp failed-test
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_endpoint_exceptions·ts - detection engine api security and spaces enabled Rule exception operators for endpoints "is one of" operator should filter 1 single value if it is set as an exception and the os_type is set to only 1 value #116231
    Team:Detections and Resp failed-test

Group4
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "threshold" rule type should show "legacy_notifications_disabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "disabled"/"in-active" #160453
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "ml" rule type should show "notifications_enabled", "notifications_disabled" "legacy_notifications_enabled", "legacy_notifications_disabled", all to be "0" for "enabled"/"active" rule that does not have any actions #160201
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "indicator_match/threat_match" rule type should show "legacy_notifications_enabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "enabled"/"active" #159825
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "ml" rule type should show "notifications_disabled" to be "1" for rule that has at least "1" action(s) and the alert is "disabled"/"in-active" #159498
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  5. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "ml" rule type should show "legacy_notifications_enabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "enabled"/"active" #159388
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  6. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "pre-packaged"/"immutable" rules should show "legacy_notifications_enabled" to be "1", "has_notification" to be "false, "has_legacy_notification" to be "true" for rule that has at least "1" action(s) and the alert is "enabled"/"active" #160373
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    yctercero
  7. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "kql" rule type should show "notifications_enabled", "notifications_disabled" "legacy_notifications_enabled", "legacy_notifications_disabled", all to be "0" for "disabled"/"in-active" rule that does not have any actions #159499
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    e40pud yctercero
  8. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "kql" rule type should show "legacy_notifications_disabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "disabled"/"in-active" #158905
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    e40pud yctercero
  9. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/usage_collector/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "eql" rule type should show "legacy_notifications_disabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "disabled"/"in-active" #156088
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test skipped-test
    e40pud
  10. Failing ES Promotion: FTR Configs #22 / detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule telemetry "kql" rule type should show "legacy_notifications_disabled" to be "1" for rule that has at least "1" legacy action(s) and the alert is "disabled"/"in-active" #160297
    Team:Detection Alerts Team:Detections and Resp failed-es-promotion
    e40pud

Group5
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/keyword_family/keyword·ts - detection engine api security and spaces enabled Detection keyword family data types Rule detects against a keyword of event.dataset "kql" rule type should detect the "dataset_name_1" from "event.dataset" #121795
    e40pud

Group7
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group7/exception_operators_data_types/double·ts - detection engine api security and spaces enabled - Group 7 Detection exceptions data types and operators Rule exception operators for data type double "is in list" operator working against string values in the data set will return 3 results if we have a list that includes 1 double #155122
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test

Group8
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/text·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type text "is not" operator will return 0 results if it cannot find what it is excluding #107911
    Team:Detections and Resp failed-test
    e40pud
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/long·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type long "is not in list" operator working against string values in the data set will return 4 results if we have a list that excludes all long #108282
    Team:Detections and Resp failed-test
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/long·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type long "is not in list" operator working against long values in the data set will return 4 results if we have a list that excludes all long #116387
    Team:Detection Engine failed-test

Group9
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group9/exception_operators_data_types/text_array·ts - detection engine api security and spaces enabled - Group 9 Detection exceptions data types and operators Rule exception operators for data type text "is in list" operator will return 4 results if we have two lists with an AND contradiction text === "word one" AND text === "word five" #155412
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/text_array·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type text "is in list" operator will return only the empty array for results if we have a list that includes all text #113418
    Team:Detections and Resp failed-test
    e40pud
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/ip·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type ip "is" operator should filter a CIDR range of "127.0.0.1/30" #117160
    Team:Detection Engine failed-test

Group10
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index migrate pre-8.0 action connector ids "before each" hook for "should import a non-default-space 7.16 rule with a connector made in the non-default space into the default space successfully" #156090
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/legacy_actions_migrations·ts - detection engine api security and spaces enabled - Group 10 migrate_legacy_actions "before all" hook for "migrates legacy actions for rule with no actions" #156096
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_export_rules·ts - detection engine api security and spaces enabled - Group 10 import_export_rules_flow should be able to reimport a rule referencing an exception list with existing comments #156095
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index importing with exceptions should resolve exception references that include comments #156094
    Team: SecuritySolution Team:Detections and Resp Team:Platform failed-test
  5. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index importing with exceptions should resolve exception references when importing into a clean slate #156093
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  6. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index importing with exceptions should be able to import a rule and an exception list #156091
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  7. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/import_rules·ts - detection engine api security and spaces enabled - Group 10 import_rules importing rules with an index importing with exceptions should only remove non existent exception list references from rule #156092
    Team: SecuritySolution failed-test

Rule Execution
Beta Give feedback
  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/threat_match·ts - detection engine api security and spaces enabled - rule execution logic Threat match type rules terms and match should have the same alerts with pagination #155304
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
    nkhristinin
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/threat_match·ts - detection engine api security and spaces enabled - rule execution logic Threat match type rules indicator enrichment: event-first search generates multiple signals with multiple matches #143978
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
  3. Failing ES Promotion: FTR Configs #22 / detection engine api security and spaces enabled - rule execution logic Non ECS fields in alert document source should fail creating alert when ECS field mapping is geo_point #154277
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-es-promotion
    e40pud
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_threat_matching·ts - detection engine api security and spaces enabled create_threat_matching tests with auditbeat data should be able to execute and get 10 signals when doing a specific query #122502
    Team: CTI Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test

Cypress

Exception
Beta Give feedback
  2. [Security Solution][Flaky] /x-pack/plugins/security_solution/cypress/e2e/exceptions/shared_exception_lists_management/manage_shared_exception_list.cy.ts/Manage shared exception list -- Delete exception list without rule reference #153781
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test test-failure-flaky
  5. [Security Solution] [Flaky] /x-pack/plugins/security_solution/cypress/e2e/rule_details_flow/add_edit_exception_data_view.cy.ts/Add exception using data views from rule details -- Creates an exception item from alert actions overflow menu #153758
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test test-failure-flaky
  6. Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/lists/create_exception_list_item·ts - Lists plugin Lists API "before all" hook for "should return a 400 if an endpoint exception item with a list-based entry is provided" #151636
    Feature:Rule Exceptions Team: SecuritySolution Team:Detection Engine Team:Detections and Resp blocker consider-next failed-test legit-flake skipped-test v8.15.0
    yctercero
  7. Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/lists/create_exception_list_item·ts - Lists plugin Lists API "after all" hook for "should return a 400 if endpoint exception entry has disallowed field" #151637
    Team:Detections and Resp failed-test

Unit testing

Tasks
Beta Give feedback
  1. Failing test: Jest Tests.x-pack/plugins/security_solution/public/detections/components/rules/rule_preview - query_preview/helpers isNoisy returns false if timeframe selection is "Last day" and average hits per hour is equal to one execution duration #153168
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp failed-test
@WafaaNasr WafaaNasr self-assigned this Jul 10, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Jul 10, 2023
@WafaaNasr WafaaNasr changed the title [Security Solution] [Flaky tests] Determine true flakes [Security Solution] [Flaky tests] Determine true flakes / failures Jul 10, 2023
@WafaaNasr WafaaNasr mentioned this issue Jul 10, 2023
Closed
@WafaaNasr WafaaNasr added the Team:Detection Engine Security Solution Detection Engine Area label Jul 10, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Jul 10, 2023
@WafaaNasr WafaaNasr changed the title [Security Solution] [Flaky tests] Determine true flakes / failures [Security Solution] [Flaky/ Failing tests] Determine true flakes / failures Jul 10, 2023
@WafaaNasr WafaaNasr changed the title [Security Solution] [Flaky/ Failing tests] Determine true flakes / failures [Security Solution] [Tests] Determine true flakes / failures Jul 10, 2023
e40pud added a commit to e40pud/kibana that referenced this issue Jul 10, 2023
@e40pud
Flaky tests: Group 4
541e69a 
elastic#161531
@e40pud e40pud mentioned this issue Jul 10, 2023
Merged
@WafaaNasr
Copy link
Contributor Author

Flaky test summary https://docs.google.com/spreadsheets/d/1bhcHV8kqUYQyjFMgV5WYR-LpEYcyPDPVRY3nXsYPG9g/edit?pli=1#gid=0

e40pud added a commit that referenced this issue Jul 12, 2023
@e40pud
Security/tests/flaky group 4 (#161571)
3827dde 
## Summary

Unskipping after running through flaky test runner 100 times:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2599

Main ticket: #161531

Resolving #159499
Resolving #158905
Resolving #156088
Resolving #160297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@e40pud e40pud

Labels
Team:Detection Engine Security Solution Detection Engine Area test-failure-flaky
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@e40pud @yctercero @WafaaNasr