[Index Management] Add bulk edit data retention

[ElenaStoeva]

Closes #165142

Summary

This PR adds bulk edit data retention to Data streams table.

On all offerings apart from serverless security project:

Screen.Recording.2024-12-12.at.18.37.58.mov

On serverless security project:

Screen.Recording.2024-12-12.at.18.45.58.mov

How to test:

Testing in serverless security project:

1. Start ES with yarn es serverless --projectType security -E data_streams.lifecycle.retention.max=200d and Kibana with yarn serverless-security
2. Navigate to console and create a few data streams following these instructions
3. Navigate to Index Management -> Data streams
4. Select multiple data streams and click on bulk edit data retention button
5. Verify that:

• The help text displays the set max data retention with a link to cloud.
• The form doesn't allow setting a data retention higher than the max data retention.
• If any of the data streams has a reduced data retention period, an error banner is shown, displaying the affected data streams.

Testing in stateful and searc/oblt serverless project:

1. Start ES with yarn es serverless and Kibana with yarn serverless-security
2. Navigate to console and create a few data streams following this instructions
3. Navigate to Index Management -> Data streams
4. Select multiple data streams and click on bulk edit data retention button
5. Verify that:

• There is no help text displaying the max data retention and no validation for the data retention period.
• If any of the data streams has a reduced data retention period, an error banner is shown, displaying the affected data streams.

Configure cloud locally by adding the following to config/kibana.dev.yml and restarting Kibana:

xpack.cloud.id: 'test'
xpack.cloud.base_url: "https://cloud.elastic.co"
xpack.cloud.deployment_url: "/deployments/deploymentId"

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7619

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[SoniaSanzV]

Thank you for working on this @ElenaStoeva! I tested it locally and in general looks good, but I've seen some things that I think it can be improved:

General

The changes in the flyout have provoked that the modal width changes when modifying the retention for a single data stream:

When you edit the data retention for a few data streams and you first introduce a value but then select the maximum/infinite period, the data deletion warning is still there, which I believe may be confusing for users as it may seem that some data will be deleted

In the same line but not that serious, if I have a validation error and then I toggle the max retention button, the validation error is still there:

Super nit pick. In the mocks the Delete data streams icon and text is red but here is all black:

In the flyout the icon is red but the text is black, which I don't know if it's the expected look.

Security serverless

In serverless security I saw a few weird behaviors (it happens in both single edit and bulk):

• If a have some data stream set to max retention(toggle on) and then manually change the retention to the same value it says I will lose data, even thought the retention period is the same. It happens both in bulk and flyout.
• If I select a data stream and edit the retention to max (toggle on) and save, I have a toaster saying that retention is mandatory and it has been set to the maximun.

• When I click in the Manage project settings link, although it is an external link, it reload the page I'm working and I don't reach the Project settings page:
  

[ryankeairns]

@jovana-andjelkovic would you like to review this one?

[ElenaStoeva]

Thank you for addressing my feedback @ElenaStoeva ! Everything looks good to me but one thing: In serverless security, when I change the retention to max in bulk o in the flyout I'm still seeing the warning toaster saying I didn't set a data retention period

Hi @SoniaSanzV, thanks for re-reviewing and sorry for missing to address this point! This warning message actually comes from Elasticsearch and I believe that this behavior exists from before this PR. Our current logic is that if the response comes back with a warning from Es, we should display it in a toast message.

[SoniaSanzV]

Hi @SoniaSanzV, thanks for re-reviewing and sorry for missing to address this point! This warning message actually comes from Elasticsearch and I believe that this behavior exists from before this PR. Our current logic is that if the response comes back with a warning from Es, we should display it in a toast message.

Fair enough! It's not the best experience but since is not in our side, approving!

[jovana-andjelkovic]

@ElenaStoeva for some reason when testing the latest build, I get some of the same issues @SoniaSanzV noted above:

• [General] When editing individual data stream retention (not through bulk action) the modal size changes due to callout
• [Security] Entering value higher than project level retention maximum, and then toggling "maximum retention" retains the error and user cannot submit the form – same happens in bulk modal as well as individual edit on data stream
  
  

Other issue:

• [Security] When entering different values for same selection, the callout shows different content:
  • entering 20 days shows callout that doesn't list all affected data streams
  • entering 200 days on same selection shows all affected data streams list in callout

[ElenaStoeva]

Thanks for the review @jovana-andjelkovic!

• [General] When editing individual data stream retention (not through bulk action) the modal size changes due to callout

Hmm I thought this was fixed, not sure what happened with these changes - should be fixed now with 4ebb176.
On a side note, we only want to display this error callout for bulk edit, correct?

• [Security] Entering value higher than project level retention maximum, and then toggling "maximum retention" retains the error and user cannot submit the form – same happens in bulk modal as well as

I actually already addressed this in my previous comment: Ah this behavior actually exists from before this PR and it's tricky to fix. I think we've discussed that on Slack before and decided that it's okay to leave it as is.
I agree that this is not a great UX so I'm happy to open an issue so we can address it in a follow-up PR. Wdyt?

• [Security] When entering different values for same selection, the callout shows different content:

  • entering 20 days shows callout that doesn't list all affected data streams
  • entering 200 days on same selection shows all affected data streams list in callout

This is because for 20 days, there are 12 affected data streams and we decided that we only display the names for <= 10 data streams, right? For 200 days, there are 10 affected data streams, which is why they are displayed.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 429e891
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-203083-429e8912af2e
• Security Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
indexManagement	698	702	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
indexManagement	699.6KB	703.9KB	+4.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
indexManagement	47.6KB	47.6KB	+86.0B

History

• 💛 Build #262008 was flaky 06012ca
• 💚 Build #261856 succeeded 1c9ecd7
• 💔 Build #261750 failed dc21c91
• 💔 Build #260968 failed efd7ca1

cc @ElenaStoeva

[jovana-andjelkovic]

@ElenaStoeva thank you!

Hmm I thought this was fixed, not sure what happened with these changes - should be fixed now with 4ebb176.
On a side note, we only want to display this error callout for bulk edit, correct?

It doesn't display anymore on single edit ✔️ I do think we should have it there, as well. But as it wasn't originally specified and it's not planned part of this PR – maybe we can add an issue to backlog to add it in single edit as well?

This is because for 20 days, there are 12 affected data streams and we decided that we only display the names for <= 10 data streams, right? For 200 days, there are 10 affected data streams, which is why they are displayed.

Ah okay! In my head that limit was set on number of selected data streams rather than number of affected – but this works, maybe even better!

I think we can merge this PR! 🟢

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12413377278

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7619

[✅] x-pack/test/functional/apps/index_management/config.ts: 200/200 tests passed.

see run history