Elastic-Agent support for selective monitor schema information

[gizas]

Describe the enhancement:
As part of #614 , we would like to implement the selective autodiscovery (hints based) feature in the Managed Elastic Agent scenario. For more information about hints autodiscovery you can see here

With specific feature we would like to be able on elastic-agent side to track events of a kubernetes cluster that include selective monitor data (hints) and upon a new emission of such data to be able to send a pre-defined schema of information towards Elastic Stack.

So main outcomes of this feature are:

• Definition of selective monitor schema that will include information for what kind of annotations will support and how
• Documentation of new schema updates
• Implementing changes on kubernetes provider (like eg. Add support for hints' based autodiscovery in kubernetes provider #662)

Describe a specific use case for the enhancement or feature:
Example provided for use case that user defines a Redis hint to be discovered. The user has enabled auto-discover-hints (hints.enabled:true) on managed agent configuration.

Workflow will be the following:

1.Customer defines following hints block configuration, inside a specific kubernetes pod, under annotations block:

co.elastic.metrics/package: redis
co.elastic.metrics/host: http://localhost:6379
co.elastic.metrics/period: 10s 

** This is where this feature starts:**
2. This annotations config is discovered by kubernetes provider and will be emitted eventually to agent's data to be sent

**Below step will be handled by new feature #1349 **
3. Above data with specific annotations defined, will be saved in a predefined place.

**Below step will be handled by new feature #1347 **
4. Fleet will monitor index updates and if an update-event includes hints block configuration, this will trigger Fleet to perform specific actions needed for the new package installation as described #1347

Note
Work is already done to support selective monitoring on standalone agent (#662 ). If any discrepancies found, both schemas need to be adjusted.

Relevant Links:

• https://github.com/elastic/observability-dev/issues/2100 , Onweek Project that implemented relevant POC
• Selective monitoring of Kubernetes workloads using annotations for (managed) agent fleet #614 (comment)

Logical Flow

cc @ChrsMark