[Security Solution] Populate alert status auditing fields #171589

marshallmain · 2023-11-20T22:49:07Z

This PR populates the existing kibana.alert.workflow_user field in the alerts-as-data mappings with the profile_uid of the last user to modify the status of the alert. It also adds a new field, kibana.alert.workflow_status_updated_at, to track the last time the workflow status was updated and populates it with a timestamp.

Similar to the alert assignment PR, workflow_user renders in the table with a user avatar instead of the raw profile_uid value stored in the alert. The filter in/out buttons on the row cell automatically add a filter that uses the raw value so that filtering works correctly.

Due to limitations of Kibana's user profile implementation, workflow_user is only populated if a user changes the alert status using the alert status route (POST /api/detection_engine/signals/status) within an interactive session, i.e. logs in rather than passes credentials with each API request (related issue).

Alerts table

Alert details

Checklist

Functional changes are hidden behind a feature flag. If not hidden, the PR explains why these changes are being implemented in a long-living feature branch.
Functional changes are covered with a test plan and automated tests.
Stability of new and changed tests is verified using the Flaky Test Runner.
- Flaky test run: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4130
Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless.
Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders.
- https://github.com/elastic/security-team/issues/4820
Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both).
- [Detection Engine] Alert status auditing (kibana.alert.workflow_user and kibana.alert.workflow_status_updated_at) security-docs#4325

…provider

elasticmachine · 2023-11-22T16:21:05Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2023-11-22T16:21:05Z

Pinging @elastic/security-solution (Team: SecuritySolution)

shahzad31

Obs UX Changes LGTM !!

pmuellr · 2023-11-22T17:11:46Z

The ResponseOps changes besides the alert table changes LGTM. I'd like to have someone from RAM review the alert table changes since I'm not sure what the ramifications of those changes are. @XavierM ?

e40pud

LGTM!

yctercero · 2023-12-01T05:01:11Z

...ins/security_solution/server/lib/detection_engine/routes/signals/open_close_signals_route.ts

-const getUpdateSignalStatusScript = (status: SetSignalsStatusSchemaDecoded['status']) => ({
-  source: `if (ctx._source['${ALERT_WORKFLOW_STATUS}'] != null) {
-      ctx._source['${ALERT_WORKFLOW_STATUS}'] = '${status}'
+const getUpdateSignalStatusScript = (


This only returns user profiles that are enabled right? Is a profile enabled automatically on login? Would there be a case of someone being logged onto the UI with a disabled profile and so this status is not logged?

yctercero · 2023-12-01T05:02:10Z

...security_solution_cypress/cypress/e2e/detection_response/detection_alerts/alert_status.cy.ts

@@ -39,6 +42,7 @@ import { ALERTS_URL } from '../../../urls/navigation';

 // FLAKY: https://github.com/elastic/kibana/issues/169091


I have a PR trying to fix the flake, just haven't gotten back to it yet! #171676

yctercero · 2023-12-01T05:04:17Z

Hey @marshallmain - sorry I just got to doing a very quick overview today. I plan to pull down and test tomorrow. Could we add the new PR checklist to the description? Thanks so much!

…pdated-at-3

PhilippeOberti

Thanks for adding this in the table and the alert details flyout @marshallmain!

I left a few small comments.

Also I'm thinking that this could be a good candidate for some Cypress tests? We already have everything setup for you, and even have one test for each action performed on the alert (this one when closing an alert and that one when acknowledging an alert). You probably would have to update the page filters to show the changed alert, reopen the flyout and test that the new section correctly shows up under the About section. Wdyt?
(I tried manually once a test was finished and it worked, so we should be able to do this programmatically without any issues). I'm happy to help out on this if you need!

I also have a question regarding the UI. Was adding this new section validated by UIUX? It seems a bit weird to me, and I would have thought instead to add something in the header next to the already existing status component (by probably updating this status.tsx file).

Finally, when the Mitre Attack component is rendered, the spacing is a bit small between mitre attack and this new status component. The challenge is both these are conditionally rendered. What do you think about removing this <EuiSpacer size="m" /> here and add one <EuiSpacer size="m" /> at the top of both the mitre_attack component and your new alert_status? That way the spacing will always look nice, when the components are visible or not.

PhilippeOberti · 2023-12-04T22:58:38Z

...k/plugins/security_solution/public/flyout/document_details/right/components/alert_status.tsx

+import { useBulkGetUserProfiles } from '../../../../common/components/user_profiles/use_bulk_get_user_profiles';
+import { PreferenceFormattedDate } from '../../../../common/components/formatted_date';
+
+export const AlertStatus: FC = () => {


can you please add a short documentation here, just to be consistent with all the other components in the document_details folder?

PhilippeOberti · 2023-12-04T23:07:06Z

...k/plugins/security_solution/public/flyout/document_details/right/components/alert_status.tsx

+  const statusUpdatedBy = searchHit.fields?.['kibana.alert.workflow_user'] as string;
+  const statusUpdatedAt = searchHit.fields?.['kibana.alert.workflow_status_updated_at'];


any reason for not using getFieldsData along side our x-pack/plugins/security_solution/public/flyout/document_details/shared/utils.tsx like we use in the rest of the document_details flyout code?

The getFieldsData is available from the useRightPanelContext you're already using.

It would be

const { getFieldsData } = useRightPanelContext(); const statusUpdatedBy = getField(getFieldsData('kibana.alert.workflow_user')); const statusUpdatedAt = getField(getFieldsData('kibana.alert.workflow_status_updated_at'));

I figured getting the raw data from the searchHit was simple enough, I can change it though

PhilippeOberti · 2023-12-04T23:09:38Z

x-pack/plugins/security_solution/public/flyout/document_details/right/components/test_ids.ts

@@ -43,6 +43,10 @@ const MITRE_ATTACK_TEST_ID = `${PREFIX}MitreAttack` as const;
 export const MITRE_ATTACK_TITLE_TEST_ID = `${MITRE_ATTACK_TEST_ID}Title` as const;
 export const MITRE_ATTACK_DETAILS_TEST_ID = `${MITRE_ATTACK_TEST_ID}Details` as const;

+export const WORKFLOW_STATUS_TEST_ID = `${PREFIX}WorkflowStatus` as const;


nit but this one doesn't need to be exported

marshallmain · 2023-12-05T01:20:22Z

I also have a question regarding the UI. Was adding this new section validated by UIUX? It seems a bit weird to me, and I would have thought instead to add something in the header next to the already existing status component (by probably updating this status.tsx file).

Is there a specific person that needs to approve changes to the flyout? I avoided adding it to the header because it looked cluttered to me already and adding status history information didn't seem to fit with the rest of the basic info that's displayed in the header.

I'm working on making the other suggested changes to the flyout, thanks for the review!

…ment

kibana-ci · 2023-12-05T02:58:32Z

💚 Build Succeeded

Buildkite Build
Commit: 1f3df6c
Cloud Deployment
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-171589-1f3df6c0a139
Security Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	4773	4774	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/rule-data-utils`	115	116	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	3.7MB	3.7MB	+42.0B
`infra`	1.9MB	1.9MB	+42.0B
`observability`	1.1MB	1.1MB	+42.0B
`securitySolution`	12.8MB	12.8MB	+3.9KB
`triggersActionsUi`	1.4MB	1.4MB	+42.0B
total			+4.0KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5661	+5661
`total size`	-	5.9MB	+5.9MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`apm`	36.7KB	36.8KB	+73.0B
`infra`	103.0KB	103.1KB	+73.0B
`observability`	101.8KB	101.9KB	+73.0B
`triggersActionsUi`	104.1KB	104.2KB	+73.0B
total			+292.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/rule-data-utils`	118	119	+1

References to deprecated APIs

id	before	after	diff
`securitySolution`	560	559	-1

History

💛 Build #181580 was flaky 0d8d808
💚 Build #181009 succeeded 1fe5fc3
💚 Build #180193 succeeded 54cb50f
💔 Build #180126 failed 95fc8e2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

PhilippeOberti · 2023-12-05T15:07:57Z

I also have a question regarding the UI. Was adding this new section validated by UIUX? It seems a bit weird to me, and I would have thought instead to add something in the header next to the already existing status component (by probably updating this status.tsx file).

Is there a specific person that needs to approve changes to the flyout? I avoided adding it to the header because it looked cluttered to me already and adding status history information didn't seem to fit with the rest of the basic info that's displayed in the header.

I'm working on making the other suggested changes to the flyout, thanks for the review!

@marshallmain: @ferenrigue owns the flyout design but she is on vacation right now. We could ask @codearos but if you want we can merge this as is and I'm happy to make small UI changes later once UIUX comes up with a solution they like?

PhilippeOberti

LGTM for the Threat Hunting Investigations team, thanks for making the changes @marshallmain

…nge to flyout (#205224) ## Summary Over a year ago, [this PR](#171589) added some information to the alert details flyout, to show when an alert's status (`closed`, `open` or `aknowledged`) had been modified last and by which user. Shortly after, [this follow up PR](#172888) removed the UI from the alert details flyout, as the information wasn't extremely important and was taking some valuable vertical space, pushing down below the `Highlighted fields` section, that users were finding very important. A few months later, we added the ability to persist which of the top sections (`About`, `Investigation`, `Visualizations`, `Insights` and `Response`) were collapsed or expanded. That way the user wouldn't have to always collapse or expand sections they would often don't need. This PR brings back the alert's last status changes to the `About` section, as the vertical space is no longer a big issues, because users can now collapse the entire `About` section. #### If data is not present, the last change UI is not shown ![Screenshot 2024-12-27 at 3 46 14 PM](https://github.com/user-attachments/assets/24e033d7-fb15-496a-97be-ecf78996d243) #### If the correct data is shown: ![Screenshot 2024-12-27 at 3 50 12 PM](https://github.com/user-attachments/assets/a13f54d8-1804-4baf-a12b-5203beb4f92d) ### How to test - have a few alerts in the alerts table - open the alert details flyout for one alert and change the status (button in the header) - verify that the last status change section is shown in the `About` section ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…nge to flyout (elastic#205224) ## Summary Over a year ago, [this PR](elastic#171589) added some information to the alert details flyout, to show when an alert's status (`closed`, `open` or `aknowledged`) had been modified last and by which user. Shortly after, [this follow up PR](elastic#172888) removed the UI from the alert details flyout, as the information wasn't extremely important and was taking some valuable vertical space, pushing down below the `Highlighted fields` section, that users were finding very important. A few months later, we added the ability to persist which of the top sections (`About`, `Investigation`, `Visualizations`, `Insights` and `Response`) were collapsed or expanded. That way the user wouldn't have to always collapse or expand sections they would often don't need. This PR brings back the alert's last status changes to the `About` section, as the vertical space is no longer a big issues, because users can now collapse the entire `About` section. #### If data is not present, the last change UI is not shown ![Screenshot 2024-12-27 at 3 46 14 PM](https://github.com/user-attachments/assets/24e033d7-fb15-496a-97be-ecf78996d243) #### If the correct data is shown: ![Screenshot 2024-12-27 at 3 50 12 PM](https://github.com/user-attachments/assets/a13f54d8-1804-4baf-a12b-5203beb4f92d) ### How to test - have a few alerts in the alerts table - open the alert details flyout for one alert and change the status (button in the header) - verify that the last status change section is shown in the `About` section ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit a4b1975)

…us change to flyout (#205224) (#205835) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Alert details] - bring back last alert status change to flyout (#205224)](#205224)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Philippe Oberti <[email protected]>

…nge to flyout (elastic#205224) ## Summary Over a year ago, [this PR](elastic#171589) added some information to the alert details flyout, to show when an alert's status (`closed`, `open` or `aknowledged`) had been modified last and by which user. Shortly after, [this follow up PR](elastic#172888) removed the UI from the alert details flyout, as the information wasn't extremely important and was taking some valuable vertical space, pushing down below the `Highlighted fields` section, that users were finding very important. A few months later, we added the ability to persist which of the top sections (`About`, `Investigation`, `Visualizations`, `Insights` and `Response`) were collapsed or expanded. That way the user wouldn't have to always collapse or expand sections they would often don't need. This PR brings back the alert's last status changes to the `About` section, as the vertical space is no longer a big issues, because users can now collapse the entire `About` section. #### If data is not present, the last change UI is not shown ![Screenshot 2024-12-27 at 3 46 14 PM](https://github.com/user-attachments/assets/24e033d7-fb15-496a-97be-ecf78996d243) #### If the correct data is shown: ![Screenshot 2024-12-27 at 3 50 12 PM](https://github.com/user-attachments/assets/a13f54d8-1804-4baf-a12b-5203beb4f92d) ### How to test - have a few alerts in the alerts table - open the alert details flyout for one alert and change the status (button in the header) - verify that the last status change section is shown in the `About` section ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

marshallmain and others added 9 commits November 20, 2023 14:03

Add user avatar rendering for alerts table

c856fb0

Populate workflow_user and workflow_updated_at on status change

b7c64e8

Add API tests, change field name, fix route to use existing services …

164eeda

…provider

Add alert status history to flyout

b962530

Add cypress test logic to verify workflow_user renders correctly

9bca9cc

Fix test mock

b5bd553

Fix tests

a11742a

Merge branch 'main' into workflow-user-updated-at-3

7aa9caa

Merge branch 'main' into workflow-user-updated-at-3

de56bab

marshallmain marked this pull request as ready for review November 22, 2023 16:20

marshallmain requested review from a team as code owners November 22, 2023 16:20

marshallmain requested a review from yctercero November 22, 2023 16:20

marshallmain added release_note:enhancement Team:Detection Engine Security Solution Detection Engine Area v8.12.0 Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Nov 22, 2023

marshallmain added the Feature:Detection Alerts Security Solution Detection Alerts Feature label Nov 22, 2023

shahzad31 approved these changes Nov 22, 2023

View reviewed changes

marshallmain mentioned this pull request Nov 28, 2023

[Detection Engine] Alert status auditing (kibana.alert.workflow_user and kibana.alert.workflow_status_updated_at) elastic/security-docs#4325

Closed

marshallmain requested a review from XavierM November 28, 2023 23:31

marshallmain added ci:cloud-deploy Create or update a Cloud deployment ci:project-deploy-security Create a Security Serverless Project labels Nov 28, 2023

Merge branch 'main' into workflow-user-updated-at-3

1819985

e40pud approved these changes Nov 29, 2023

View reviewed changes

marshallmain added 2 commits November 29, 2023 08:09

Remove generic typing on AlertsTableConfigurationRegistry

95fc8e2

Fix types

54cb50f

XavierM approved these changes Nov 29, 2023

View reviewed changes

yctercero reviewed Dec 1, 2023

View reviewed changes

marshallmain and others added 2 commits December 1, 2023 09:11

Merge branch 'main' into workflow-user-updated-at-3

1fe5fc3

Merge branch 'main' of github.com:elastic/kibana into workflow-user-u…

0d8d808

…pdated-at-3

PhilippeOberti reviewed Dec 4, 2023

View reviewed changes

Review: Right panel Cypress tests, EuiSpacer, capitalization, doc com…

1f3df6c

…ment

PhilippeOberti approved these changes Dec 5, 2023

View reviewed changes

marshallmain merged commit 4a89208 into elastic:main Dec 5, 2023

kibanamachine added the backport:skip This commit does not require backporting label Dec 5, 2023

marshallmain deleted the workflow-user-updated-at-3 branch December 5, 2023 19:13

PhilippeOberti mentioned this pull request Dec 7, 2023

[Security Solution] remove alert status UI from alerts details flyout #172888

Merged

marshallmain mentioned this pull request Jan 2, 2024

Security APP, see who change the signal.status #95621

Closed

PhilippeOberti mentioned this pull request Dec 27, 2024

[Security Solution][Alert details] - bring back last alert status change to flyout #205224

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Populate alert status auditing fields #171589

[Security Solution] Populate alert status auditing fields #171589

marshallmain commented Nov 20, 2023 •

edited

Loading

elasticmachine commented Nov 22, 2023

elasticmachine commented Nov 22, 2023

shahzad31 left a comment

pmuellr commented Nov 22, 2023

e40pud left a comment

yctercero Dec 1, 2023

yctercero Dec 1, 2023

yctercero commented Dec 1, 2023

PhilippeOberti left a comment

PhilippeOberti Dec 4, 2023

PhilippeOberti Dec 4, 2023

marshallmain Dec 5, 2023

PhilippeOberti Dec 4, 2023

marshallmain commented Dec 5, 2023

kibana-ci commented Dec 5, 2023 •

edited

Loading

API count

References to deprecated APIs

PhilippeOberti commented Dec 5, 2023

PhilippeOberti left a comment

		@@ -39,6 +42,7 @@ import { ALERTS_URL } from '../../../urls/navigation';

		// FLAKY: https://github.com/elastic/kibana/issues/169091

		const statusUpdatedBy = searchHit.fields?.['kibana.alert.workflow_user'] as string;
		const statusUpdatedAt = searchHit.fields?.['kibana.alert.workflow_status_updated_at'];

[Security Solution] Populate alert status auditing fields #171589

[Security Solution] Populate alert status auditing fields #171589

Conversation

marshallmain commented Nov 20, 2023 • edited Loading

Alerts table

Alert details

Checklist

elasticmachine commented Nov 22, 2023

elasticmachine commented Nov 22, 2023

shahzad31 left a comment

Choose a reason for hiding this comment

pmuellr commented Nov 22, 2023

e40pud left a comment

Choose a reason for hiding this comment

yctercero Dec 1, 2023

Choose a reason for hiding this comment

yctercero Dec 1, 2023

Choose a reason for hiding this comment

yctercero commented Dec 1, 2023

PhilippeOberti left a comment

Choose a reason for hiding this comment

PhilippeOberti Dec 4, 2023

Choose a reason for hiding this comment

PhilippeOberti Dec 4, 2023

Choose a reason for hiding this comment

marshallmain Dec 5, 2023

Choose a reason for hiding this comment

PhilippeOberti Dec 4, 2023

Choose a reason for hiding this comment

marshallmain commented Dec 5, 2023

kibana-ci commented Dec 5, 2023 • edited Loading

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Canvas Sharable Runtime

Page load bundle

API count

References to deprecated APIs

History

PhilippeOberti commented Dec 5, 2023

PhilippeOberti left a comment

Choose a reason for hiding this comment

marshallmain commented Nov 20, 2023 •

edited

Loading

kibana-ci commented Dec 5, 2023 •

edited

Loading