[Index Management] [Meta] Support for data stream lifecycle

[alisonelizabeth]

We need to support data stream lifecycle within Index Management.

Elasticsearch docs

Tasks

Preview Give feedback

1. [Index Management] Show data retention on data streams tab #165134
   Feature:Index Management Team:Kibana Management enhancement +3
   
2. [Index Management] Ability to edit data retention #165136
   Feature:Index Management Team:Kibana Management enhancement +3
   
3. [Index Management] View/edit data retention via index template #165139
   Feature:Index Management Team:Kibana Management enhancement +3
   
4. [Index Management] View/edit data retention via component template UI #165139 #165140
   Feature:Index Management Team:Kibana Management enhancement +3
   
5. [Index Management] Bulk edit data retention #165142
   Feature:Index Management Team:Kibana Management enhancement +3
   
6. [Index Management] Distinguish between data streams managed by ILM vs data stream lifecycle #165143
   Feature:Index Management Team:Kibana Management enhancement +3
7. [Index Management] Data stream details flyout should only show ILM link if ILM is enabled #152720
   Feature:ILM Feature:Index Management Team:Kibana Management +3
   
8. [Index Management] Represent "infinite" data retention #168494
   Feature:Index Management Team:Kibana Management bug +3
   
9. [Index Management] [Data Streams] Revisit logic for "Never delete data" toggle #167416
   Feature:Index Management Team:Kibana Management bug +3
   
10. [Index Management] [Data Streams] Be more explicit in the UI about data retention period #167475
    Feature:Index Management Team:Kibana Management enhancement +3
    
11. [Index Management] Update data retention time unit options #168495
    Feature:Index Management Team:Kibana Management enhancement +3
    
12. [Index Management] [Data Streams] Support managed_by field #167414
    Feature:Index Management Team:Kibana Management bug +3
    
13. [Index Management] UX support when data stream lifecycle is disabled #166385
    Feature:Index Management Team:Kibana Management +2
    

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[nicpenning]

A couple of thoughts here. Very good features here being worked on.

1. What time frame is days based on? Index time? @timestamp? Rollover?
2. Will it make sense to do retention based on data volume? For example, retain 100GB worth of data?

[tylerperk]

@nicpenning On 1, the time frame used for enacting retention (deletion) is 'after rollover', similar to ILM. Since we delete whole indices, this is to ensure we don't delete data earlier than expected just because it was indexed most recently.
For question 2, we don't support volume-based retention but we've had some requests for it (such as 47764) and I'm open to hearing more about the requirements for that. Perhaps you can comment on that issue and we continue the discussion there?

[nicpenning]

That makes sense. I didn't want to assume that logic, but it definitely checks out. Thanks!

[alisonelizabeth]

I'm going to close this meta issue. The initial set of work has been complete. We will open up a separate issue for the next phase.