[Rule Migration] Adding CIM to ECS mapping and ESQL validation #202331
+461
−24
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
P1llus
added
release_note:skip
P1llus
commented
Dec 2, 2024
...ib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/nodes/validation/esql_query.ts
Outdated
Show resolved
Hide resolved
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
P1llus
changed the title
semd
reviewed
Dec 2, 2024
x-pack/plugins/security_solution/server/lib/siem_migrations/rules/task/agent/graph.ts
Outdated
Show resolved
Hide resolved
semd
reviewed
Dec 2, 2024
x-pack/plugins/security_solution/server/lib/siem_migrations/rules/task/agent/graph.ts
Outdated
Show resolved
Hide resolved
semd
reviewed
Dec 2, 2024
...rity_solution/server/lib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/graph.ts
Outdated
Show resolved
Hide resolved
semd
reviewed
Dec 2, 2024
...rity_solution/server/lib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/graph.ts
Outdated
Show resolved
Hide resolved
semd
reviewed
Dec 2, 2024
x-pack/plugins/security_solution/server/lib/siem_migrations/rules/task/agent/graph.ts
Outdated
Show resolved
Hide resolved
…ter, add more AST checks for ESQL validation
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
|
e40pud
approved these changes
Dec 3, 2024
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/12138108520 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 3, 2024
…ic#202331) ## Summary This PR adds the initial context to map CIM fields to ECS and two new nodes validation and a node to handle esql validation issues, fixing itself. This is how the graph looks compared to its old one: <img width="646" alt="image" src="https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982"> Validation always runs last, and if validation returns any errors it will run the appropriate node depending on what validation failed. Once it is resolved it will validate again and then END when its successful. Currently 5 error iterations is max, which is just an arbitrary number. The default Langgraph configuration is 25 nodes executed in total for a specific graph before it errors with a recursion limit (main and sub graphs are not combined in that count). A few things are included in this PR: - Moved ESQL KB caller to util(any better place?), as it is now used in multiple nodes. - New Validation node, where any sort of validation takes place, usually the last step before ending the graph (on success). - New ESQL Error node, to resolve any ESQL validation errors and trigger a re-validation. - Fix a small bug in the main graph on the conditional edges, added a map for the allowed return values. (cherry picked from commit c1d976b)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Dec 3, 2024
…202331) (#202668) # Backport This will backport the following commits from `main` to `8.x`: - [[Rule Migration] Adding CIM to ECS mapping and ESQL validation (#202331)](#202331) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Marius Iversen","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-03T10:51:19Z","message":"[Rule Migration] Adding CIM to ECS mapping and ESQL validation (#202331)\n\n## Summary\r\n\r\nThis PR adds the initial context to map CIM fields to ECS and two new\r\nnodes validation and a node to handle esql validation issues, fixing\r\nitself.\r\n\r\nThis is how the graph looks compared to its old one:\r\n<img width=\"646\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982\">\r\n\r\n\r\nValidation always runs last, and if validation returns any errors it\r\nwill run the appropriate node depending on what validation failed. Once\r\nit is resolved it will validate again and then END when its successful.\r\n\r\nCurrently 5 error iterations is max, which is just an arbitrary number.\r\nThe default Langgraph configuration is 25 nodes executed in total for a\r\nspecific graph before it errors with a recursion limit (main and sub\r\ngraphs are not combined in that count).\r\n\r\nA few things are included in this PR:\r\n\r\n- Moved ESQL KB caller to util(any better place?), as it is now used in\r\nmultiple nodes.\r\n- New Validation node, where any sort of validation takes place, usually\r\nthe last step before ending the graph (on success).\r\n- New ESQL Error node, to resolve any ESQL validation errors and trigger\r\na re-validation.\r\n- Fix a small bug in the main graph on the conditional edges, added a\r\nmap for the allowed return values.","sha":"c1d976b470bf08d472165f7751a444d6968c48d2","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team: SecuritySolution","backport:prev-minor","v8.18.0"],"title":"[Rule Migration] Adding CIM to ECS mapping and ESQL validation","number":202331,"url":"https://github.com/elastic/kibana/pull/202331","mergeCommit":{"message":"[Rule Migration] Adding CIM to ECS mapping and ESQL validation (#202331)\n\n## Summary\r\n\r\nThis PR adds the initial context to map CIM fields to ECS and two new\r\nnodes validation and a node to handle esql validation issues, fixing\r\nitself.\r\n\r\nThis is how the graph looks compared to its old one:\r\n<img width=\"646\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982\">\r\n\r\n\r\nValidation always runs last, and if validation returns any errors it\r\nwill run the appropriate node depending on what validation failed. Once\r\nit is resolved it will validate again and then END when its successful.\r\n\r\nCurrently 5 error iterations is max, which is just an arbitrary number.\r\nThe default Langgraph configuration is 25 nodes executed in total for a\r\nspecific graph before it errors with a recursion limit (main and sub\r\ngraphs are not combined in that count).\r\n\r\nA few things are included in this PR:\r\n\r\n- Moved ESQL KB caller to util(any better place?), as it is now used in\r\nmultiple nodes.\r\n- New Validation node, where any sort of validation takes place, usually\r\nthe last step before ending the graph (on success).\r\n- New ESQL Error node, to resolve any ESQL validation errors and trigger\r\na re-validation.\r\n- Fix a small bug in the main graph on the conditional edges, added a\r\nmap for the allowed return values.","sha":"c1d976b470bf08d472165f7751a444d6968c48d2"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/202331","number":202331,"mergeCommit":{"message":"[Rule Migration] Adding CIM to ECS mapping and ESQL validation (#202331)\n\n## Summary\r\n\r\nThis PR adds the initial context to map CIM fields to ECS and two new\r\nnodes validation and a node to handle esql validation issues, fixing\r\nitself.\r\n\r\nThis is how the graph looks compared to its old one:\r\n<img width=\"646\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982\">\r\n\r\n\r\nValidation always runs last, and if validation returns any errors it\r\nwill run the appropriate node depending on what validation failed. Once\r\nit is resolved it will validate again and then END when its successful.\r\n\r\nCurrently 5 error iterations is max, which is just an arbitrary number.\r\nThe default Langgraph configuration is 25 nodes executed in total for a\r\nspecific graph before it errors with a recursion limit (main and sub\r\ngraphs are not combined in that count).\r\n\r\nA few things are included in this PR:\r\n\r\n- Moved ESQL KB caller to util(any better place?), as it is now used in\r\nmultiple nodes.\r\n- New Validation node, where any sort of validation takes place, usually\r\nthe last step before ending the graph (on success).\r\n- New ESQL Error node, to resolve any ESQL validation errors and trigger\r\na re-validation.\r\n- Fix a small bug in the main graph on the conditional edges, added a\r\nmap for the allowed return values.","sha":"c1d976b470bf08d472165f7751a444d6968c48d2"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Marius Iversen <[email protected]>
hop-dev
pushed a commit
to hop-dev/kibana
that referenced
this pull request
Dec 5, 2024
…ic#202331) ## Summary This PR adds the initial context to map CIM fields to ECS and two new nodes validation and a node to handle esql validation issues, fixing itself. This is how the graph looks compared to its old one: <img width="646" alt="image" src="https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982"> Validation always runs last, and if validation returns any errors it will run the appropriate node depending on what validation failed. Once it is resolved it will validate again and then END when its successful. Currently 5 error iterations is max, which is just an arbitrary number. The default Langgraph configuration is 25 nodes executed in total for a specific graph before it errors with a recursion limit (main and sub graphs are not combined in that count). A few things are included in this PR: - Moved ESQL KB caller to util(any better place?), as it is now used in multiple nodes. - New Validation node, where any sort of validation takes place, usually the last step before ending the graph (on success). - New ESQL Error node, to resolve any ESQL validation errors and trigger a re-validation. - Fix a small bug in the main graph on the conditional edges, added a map for the allowed return values.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 9, 2024
…ic#202331) ## Summary This PR adds the initial context to map CIM fields to ECS and two new nodes validation and a node to handle esql validation issues, fixing itself. This is how the graph looks compared to its old one: <img width="646" alt="image" src="https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982"> Validation always runs last, and if validation returns any errors it will run the appropriate node depending on what validation failed. Once it is resolved it will validate again and then END when its successful. Currently 5 error iterations is max, which is just an arbitrary number. The default Langgraph configuration is 25 nodes executed in total for a specific graph before it errors with a recursion limit (main and sub graphs are not combined in that count). A few things are included in this PR: - Moved ESQL KB caller to util(any better place?), as it is now used in multiple nodes. - New Validation node, where any sort of validation takes place, usually the last step before ending the graph (on success). - New ESQL Error node, to resolve any ESQL validation errors and trigger a re-validation. - Fix a small bug in the main graph on the conditional edges, added a map for the allowed return values.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
…ic#202331) ## Summary This PR adds the initial context to map CIM fields to ECS and two new nodes validation and a node to handle esql validation issues, fixing itself. This is how the graph looks compared to its old one: <img width="646" alt="image" src="https://github.com/user-attachments/assets/253e449c-ac6f-4913-8da4-eb36f4e7b982"> Validation always runs last, and if validation returns any errors it will run the appropriate node depending on what validation failed. Once it is resolved it will validate again and then END when its successful. Currently 5 error iterations is max, which is just an arbitrary number. The default Langgraph configuration is 25 nodes executed in total for a specific graph before it errors with a recursion limit (main and sub graphs are not combined in that count). A few things are included in this PR: - Moved ESQL KB caller to util(any better place?), as it is now used in multiple nodes. - New Validation node, where any sort of validation takes place, usually the last step before ending the graph (on success). - New ESQL Error node, to resolve any ESQL validation errors and trigger a re-validation. - Fix a small bug in the main graph on the conditional edges, added a map for the allowed return values.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the initial context to map CIM fields to ECS and two new nodes validation and a node to handle esql validation issues, fixing itself.
This is how the graph looks compared to its old one:
Validation always runs last, and if validation returns any errors it will run the appropriate node depending on what validation failed. Once it is resolved it will validate again and then END when its successful.
Currently 5 error iterations is max, which is just an arbitrary number. The default Langgraph configuration is 25 nodes executed in total for a specific graph before it errors with a recursion limit (main and sub graphs are not combined in that count).
A few things are included in this PR: