[Security Solution] Unskip perform bulk action ESS FTR tests

[maximpn]

Closes: #196470
Closes: #196462

Summary

This PR unskips perform_bulk_action_ess.ts functional test.

Details

perform_bulk_action_ess.ts includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes perform_bulk_action_ess.ts refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally @kbn/expect was replaced with expect to make asserting more transparent and avoid unclear error messages.

Flaky test results

100 runs 🟢

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7259

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/configs/ess.config.ts: 100/100 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7272

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/configs/ess.config.ts: 100/100 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7273

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/configs/ess.config.ts: 100/100 tests passed.

see run history

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7274

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/configs/ess.config.ts: 100/100 tests passed.

see run history

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[xcrzx]

Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected.

@maximpn Could you expand on the details of the race condition? If the rule SO doesn't get updated properly in the test environment, is there a chance that this could also happen in production?

[maximpn]

@xcrzx I observed quite stable flakiness with the following scenario where a test does the following

• creates an enabled rule
• creates a legacy notification action
• disable the rule via bulk action

ER: Rule's legacy notification action gets migrated.
AR: Around ~5-10% cases had the test failed since the rule doesn't have any actions at the end where it's expected.

For example should disable rules and migrate actions demonstrated such flakiness in Flaky Test Runner and locally.

Creation of only disabled rules fixed the flakiness. It looks like the task manager tries to run the rule the same time a bulk operation is performed.

---

I was about to create a ticket for DE team but after rebasing on the latest commit in main I don't observe such flakiness anymore.

I checked the legacy notification actions migration code together with bulk actions rules client code on potential issues like refresh: false for ES client but everything looks fine.

[xcrzx]

I was about to create a ticket for DE team but after rebasing on the latest commit in main I don't observe such flakiness anymore.

So the flakiness resolved itself without any changes to the tests? In that case, do we still need the refactoring introduced in this PR?

[maximpn]

@xcrzx indeed it's just enough unskip the test since the problem seems to be fixed on itself. Though refactoring improves error messages. It should simplify understanding on what part failed if the test suddenly fails. For example there are two error messages below where the first one before the refactoring and the second is

before:

└- ✖ fail: Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions
--
  | │      TypeError: Cannot read properties of undefined (reading 'uuid')

after:

└- ✖ fail: Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic edit action should migrate legacy actions on edit
--
  | │      Error: expect(received).toEqual(expected) // deep equality
  | │
  | │ - Expected  - 17
  | │ + Received  +  1
  | │
  | │ - Array [
  | │ -   Object {
  | │ -     "action_type_id": ".slack",
  | │ -     "frequency": Object {
  | │ -       "notifyWhen": "onThrottleInterval",
  | │ -       "summary": true,
  | │ -       "throttle": "1h",
  | │ -     },
  | │ -     "group": "default",
  | │ -     "id": "1b73eb9e-54fa-42ff-8bc3-a1dd686f8423",
  | │ -     "params": Object {
  | │ -       "message": "Hourly
  | │ - Rule {{context.rule.name}} generated {{state.signals_count}} alerts",
  | │ -     },
  | │ -     "uuid": Any<String>,
  | │ -   },
  | │ - ]
  | │ + Array []

[xcrzx]

Thanks for the additional info. I looked through the changes and the refactoring looks good 👍

[kibanamachine]

Starting backport for target branches: 8.15, 8.16, 8.x

https://github.com/elastic/kibana/actions/runs/11682191876

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: d947623

Failed CI Steps

• FTR Configs #45

Test Failures

• [job] [logs] FTR Configs #45 / discover/group1 discover test nested query should support querying on nested fields

Metrics [docs]

✅ unchanged

History

• 💚 Build #247321 succeeded 2a25e23

cc @maximpn

[kibanamachine]

💔 Some backports could not be created

Status	Branch	Result
❌	8.15	Backport failed because of merge conflicts
✅	8.16
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 197660

Questions ?

Please refer to the Backport tool documentation

[sharie90]

Ok

[sharie90]

[]()