Azure Monitor module to be able to support multiple subscriptions in the config

[gizas]

Describe the enhancement:
This request is coming from an existing customer that uses the Azure Monitor Integration with Elastic Agent.
The enhancement focuses on the configuration of Azure Monitor and that specifically users need to repeat the credentials for every query configuration they need to make. This makes the config block really difficult to housekeep and in some cases hit the issue 2460

The enhancement will try to:

• Investigate possible better ways to provide configuration with more compact ways
• Investigate if it is possible multiple elastic agent workers to be able to handle same subscription metrics without duplicates (vertical scaling of agents)
• See if multiple subscriptions can be configured in a specific struct and then multiple elastic agents to be able to load balance and handle each subscription (Horizontal scaling of agents)

Describe a specific use case for the enhancement or feature:
Customer shared below configuration.
base_cfg_storageaccount.yaml.txt

The example can be divided to two categories:

• Agents within the same deployment/statefulset would spread the load across themselves. That would ease the scaling of the agents significantly
• Alternatively, agents decide the set of subscriptions they can look (within a configured service principal) and then allow specific agent perform the metrics scraping against specific subscriptions. This would remove the duplication in the configs.

Additional Details

• Beats Code for Azure Monitor
• Azure SDK for go . Mind that this needs Subscription ID, Client ID, Client Secret and Tenant ID for authentication

[zmoog]

High-level Overview

This section outlines the core functionality of the Azure Monitor metricset in its present form.

1. Collect existing cloud resource definitions (e.g., VMs, DBs, etc.)
2. Collect metric definitions for each resource (e.g., CPU, memory, etc.)
3. Collect the metric values for each resource (sequentially)

The current metricset allows it to run as a single instance. Additional metricsets running with the same config (or agent policy) would collect the same data multiple times.

Areas of investigation

To simplify the configuration, we need to investigate how to:

• set up the Azure Monitor client to discover and collect resources, metric definitions, and metric values based on the service principal instead of from settings.

To scale the metricset vertically, we need to investigate how to:

• collect metric values in parallel instead of sequentially
• manage the number of concurrent requests to avoid throttling

To scale the metricset horizontally, we need to investigate how to:

• Share the list of existing resources/subscriptions/resource groups between agents

[zmoog]

To move forward in this information-gathering phase, I would investigate the Azure SDK to learn more about the options to discover resources based on service principal instead of the metricset settings.

[gizas]

@zmoog Adding this https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/monitor/azquery

This can be a possible replacement, as this can support azidentity ,which also can support multiple ways of authentication. Maybe we can think of DefaultAzureCredential and EnvironmentCredential which can be configured with environment variables.
So this will eliminate the configuration blocks from our files

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!