Elastic Agent to save selective monitor data [Support Selective Monitoring for Kubernetes]

[gizas]

Describe the enhancement:
As part of #614 , we would like to implement the selective autodiscovery (hints based) feature in the Managed Elastic Agent scenario. For more information about hints autodiscovery you can see here

!!!! The specific feature might not be required if the saving of data from agent is done directly to Elasticsearch and all the checks of changes are handled as part of #1347

We need to verify where the hints block configuration will be saved with additional metadata of agent that refers to and package policy that is bound. Then this data can be used from Fleet to make needed List of Actions as described #1347

Describe a specific use case for the enhancement or feature:
Example provided for use case that user defines a Redis hint to be discovered. The user has enabled auto-discover-hints (hints.enabled:true) on managed agent configuration.

Workflow will be the following:

1.Customer defines following hints block configuration, inside a specific kubernetes pod, under annotations block:

co.elastic.metrics/package: redis
co.elastic.metrics/host: http://localhost:6379
co.elastic.metrics/period: 10s 

2. This annotations config is discovered by kubernetes provider and will be emitted eventually to agent's data to be sent

This is where the feature starts:
Above data with specific annotations defined, will be saved in
a) a predefined place. (To be defined if we save things to Fleet Server or to Elasticsearch)
b) with a predefined format to be able to picked up from

Below step will be handled by new feature #1347
4. Fleet will monitor index updates and if an update-event includes hints block configuration, this will trigger Fleet to perform specific actions needed for the new package installation as described #1347

Note:
The choice of the place where to save the info can affect the way we make checks before we want to proceed with policy updates

Relevant Links:

• https://github.com/elastic/observability-dev/issues/2100 , Onweek Project that implemented relevant POC
• Selective monitoring of Kubernetes workloads using annotations for (managed) agent fleet #614 (comment)

cc @ChrsMark , @kpollich