Add ILM policy for new stack monitoring data streams

[jbaiera]

Adds a templated ILM policy with a 3 day or 50 gb rollover and a 3 day retention period by default. The retention period checks the value set in xpack.monitoring.history.duration and uses that for the retention period if it is present. Depending on which default it uses, the _meta field is updated with the reasoning for picking the default retention value:

The defaults

{
    ".monitoring-8-ilm-policy": {
        "version": 1,
        "modified_date": "2022-01-12T21:53:55.278Z",
        "policy": {
            "phases": {
                "warm": {
                    "min_age": "0ms",
                    "actions": {
                        "forcemerge": {
                            "max_num_segments": 1
                        }
                    }
                },
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_primary_shard_size": "50gb",
                            "max_age": "3d"
                        }
                    }
                },
                "delete": {
                    "min_age": "3d",
                    "actions": {
                        "delete": {
                            "delete_searchable_snapshot": true
                        }
                    }
                }
            },
            "_meta": {
                "defaults": {
                    "delete_min_age": "Using value of [3d] based on the monitoring plugin default"
                },
                "managed": true,
                "description": "Index lifecycle policy generated for [monitoring-*-8] data streams"
            }
        },
        "in_use_by": {
            "indices": [],
            "data_streams": [],
            "composable_templates": [
                ".monitoring-beats-mb",
                ".monitoring-kibana-mb",
                ".monitoring-es-mb",
                ".monitoring-logstash-mb"
            ]
        }
    }
}

Setting xpack.monitoring.history.duration to the value 4d at start up

{
    ".monitoring-8-ilm-policy": {
        "version": 1,
        "modified_date": "2022-01-12T20:42:01.835Z",
        "policy": {
            "phases": {
                "warm": {
                    "min_age": "0ms",
                    "actions": {
                        "forcemerge": {
                            "max_num_segments": 1
                        }
                    }
                },
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_primary_shard_size": "50gb",
                            "max_age": "3d"
                        }
                    }
                },
                "delete": {
                    "min_age": "4d",
                    "actions": {
                        "delete": {
                            "delete_searchable_snapshot": true
                        }
                    }
                }
            },
            "_meta": {
                "defaults": {
                    "delete_min_age": "Using value of [4d] based on the value of the [xpack.monitoring.history.duration] setting at node startup"
                },
                "managed": true,
                "description": "Index lifecycle policy generated for [monitoring-*-8] data streams"
            }
        },
        "in_use_by": {
            "indices": [],
            "data_streams": [],
            "composable_templates": [
                ".monitoring-beats-mb",
                ".monitoring-kibana-mb",
                ".monitoring-es-mb",
                ".monitoring-logstash-mb"
            ]
        }
    }
}

Also included is a small enhancement to allow for template variable replacement in ILM policy resource files.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jbaiera]

relates to #81839

[dakrone]

Generally looks good, I left a couple of comments.

Can you also add a test for this similar to what we do for StackTemplateRegistryTests? That way we can test the duration copying from the setting into the ILM policy and ensure it works correctly.

[jbaiera]

@elasticmachine update branch

[dakrone]

LGTM, I left two suuuper minor things

[matschaffer]

Reviewing this made me notice we're going to need a monitoring-entsearch as well. Looks like the mapping isn't there atm anyway so we'll just have to remember to include ILM when we add it (probably as part of elastic/kibana#121975)

[matschaffer]

I gave this a try locally and LGTM

[jbaiera]

@elasticmachine update branch