[Security Solution][Detection Engine] fixes ES|QL ECS multifiefields issue

[vitaliidm]

Summary

• fixes https://github.com/elastic/security-team/issues/7741 by replacing ecsMap from hardcoded @kbn/rule-registry-plugin to actual mapping for alerts indices from @kbn/alerts-as-data-utils
• when converting ES|QL row table results to object, null values skipped, since its results consists of all existing mappings in searched indices, if fields in query are not filtered

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[rylnd]

@vitaliidm to explicitly validate that this will fix the issue described in https://github.com/elastic/sdh-security-team/issues/736, we can add a regression unit test:

diff --git a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/utils/strip_non_ecs_fields.test.ts b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/utils/strip_non_ecs_fields.test.ts
index 9c10a317ee1..fe66c695f63 100644
--- a/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/utils/strip_non_ecs_fields.test.ts
+++ b/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/factories/utils/strip_non_ecs_fields.test.ts
@@ -71,6 +71,26 @@ describe('stripNonEcsFields', () => {
     ]);
   });
 
+  describe('with an object field that exists in the alerts mapping but not our local ECS definition', () => {
+    it('strips that field if it is supplied as a keyword', () => {
+      const { result, removed } = stripNonEcsFields({
+        device: 'test',
+        message: 'test message',
+      });
+
+      expect(result).toEqual({
+        message: 'test message',
+      });
+
+      expect(removed).toEqual([
+        {
+          key: 'device',
+          value: 'test',
+        },
+      ]);
+    });
+  });
+
   describe('array fields', () => {
     it('should not strip arrays of objects when an object is expected', () => {
       const { result, removed } = stripNonEcsFields({

[rylnd]

Changes look good to me!

I had a few questions about unit tests changing, but the integration tests answered my questions. Thanks!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 91f3b15

Failed CI Steps

• Investigations - Security Solution Cypress Tests #7
• Defend Workflows Cypress Tests #3

Test Failures

• [job] [logs] Defend Workflows Cypress Tests #3 / Artifact pages Trusted applications should update Endpoint Policy on Endpoint when adding Trusted application name should update Endpoint Policy on Endpoint when adding Trusted application name
• [job] [logs] Investigations - Security Solution Cypress Tests #7 / Detections : Page Filters Impact of inputs "after each" hook for "should recover from invalid kql Query result" "after each" hook for "should recover from invalid kql Query result"
• [job] [logs] Investigations - Security Solution Cypress Tests #7 / Detections : Page Filters Impact of inputs should recover from invalid kql Query result should recover from invalid kql Query result

Metrics [docs]

✅ unchanged

History

• 💔 Build #165561 failed 0d94ecf
• 💔 Build #165523 failed f806180
• 💔 Build #165410 failed 3d5f37b
• 💛 Build #165214 was flaky ceed442
• 💔 Build #165197 failed 6dbf00b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation