[SIEM][Detections Engine] Add note markdown field to backend

[yctercero]

Summary

This is part of #59176 - breaking up into backend and frontend PRs.

Problem to solve/Customer Benefit: Analysts need as much context as possible when investigating signals. If a richer format of information can be provided to them via their UX, then they may become more effective at completing investigations and cases.

This PR adds new note field for markdown and updates unit tests. One of the bigger considerations in implementing this new field was how it should deal with backwards compatibility (migration, schema requirements, etc). Decided to make the new field optional (and no default if not present) on the way in and optional on the way out.

Manual tests done:

• testing rule without note value
  • ./post_rule.sh
  • ./patch_rule.sh
  • ./update_rule.sh
• testing rule with note value
  • ./post_rule.sh ./rules/queries/query_with_note.json
  • ./patch_rule.sh ./rules/patches/update_note.json
  • ./update_rule.sh ./rules/updates/update_note.json
  • ./update_rule.sh ./rules/updates/update_note.json removing note value prior to running (resulting in updated note having no note value any longer)

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[FrankHassanabad]

You have one error from the build to fix with e2e tests:

[FrankHassanabad]

You will want to add a note to this file:
query_with_everything.json

Something like this within that file:

  "timeline_id": "timeline_id",
  "timeline_title": "timeline_title",
  "note": "Some note for you",
  "version": 1

Test it like so:

./post_rule.sh ./rules/queries/query_with_everything.json

That's the file we communicate with people such as documentation and front end tests and we use it to test things manually if the need comes up.

In the queries folder I would add something like this query file and contents:

query_with_note.json

{
  "name": "Query with a note",
  "description": "Query with a note",
  "rule_id": "query-with-note",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "user.name: root or user.name: admin",
  "note": "Hello I am a markdown note"
}

And then in the patches and update folder I would add something similar so it's easy for us to run manual tests when we need to and that would pretty much complete the feature.

You can run the update_rule.sh and the patch_rule.sh scripts to ensure it works as expected. Then you can update your comments section at the top of this to let everyone know what scripts they can run to test out the notes section.

[yctercero]

@elasticmachine merge upstream

[FrankHassanabad]

Thanks for all the updates for this and the tests. I checked it out and tested it and everything looks really good and the code reads really clean. LGTM! 👍

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: e551118

History

• 💔 Build #32462 failed c7876c2
• 💔 Build #32447 failed e429a5f
• 💚 Build #32391 succeeded 0025dca
• 💔 Build #32297 failed 1e28319
• 💔 Build #32168 failed f8513d3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream