[Filebeat] Azure Module

[sorantis]

TLDR; Add a Filebeat module for Azure.

Summary
Microsoft Azure is the second largest provider of cloud services amounting to ~ 14% of the total cloud market share. One of the main factors for companies who're moving to Azure is the ability to have full observability over their virtual infrastructure in terms of allocated core Azure services.

We've already added an integration with Azure to Metricbeat, which covers the metrics side of the use case. In addition we have a Logstash module for Azure (currently experimental) that is able to integrate with Azure activity, SQL diagnostic logs and send this data to Elastic Stack. This module is able to cover the following use cases:

• Analyze infrastructure changes and authorization activity.
• Identify suspicious behaviors and potential malicious actors.
• Perform root-cause analysis by investigating user activity.
• Monitor and optimize your SQL DB deployments.

As mentioned before, this module is experimental, with no intention to move it to a more stable state. However, the logs use case for Azure cloud is still as relevant as before and we want to be able to cover the above capabilities in Filebeat.

This issue is intended to serve this purpose - create a Filebeat module that is able to retrieve relevant logs from the Azure Monitor and store the collected logs in Elastic Stack.

[andresrc]

The initial approach for this was adding a Kafka input to filebeat (see #7641 and #12850), that can be used to ingest data from Microsoft Azure Event Hubs that have Kafka compatibility enabled (see #13455), and then we could start adding filesets on top of that.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.