Azure Application Insights TSDB Enablement

[gpop63]

Test Environment Setup

• Created Test Environment

Data Stream: app_insights

• Add dimensions and metric_types [azure_application_insights] [app_insights] Add dimensions and metric_type for metrics #7610

Verification and validation

• Verification of data in visualisation after enabling TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface
• Verify if dimension field mapping is correct in the data stream template.

Data Stream: app_state

• Add dimensions and metric_types [azure_application_insights] [app_state] Add dimensions, metric_type to the app_state data stream #7550

Verification and validation

• Verification of data in visualisation after enabling TSDB flag in kibana
• Verification of the count of documents (before & after TSDB enablement) in Discover Interface
• Verify if dimension field mapping is correct in the data stream template.

TSDB Release PR

• PR to enable TSDB [azure_application_insights] Enable TSDB #8648

Screenshots

[gpop63]

Currently, both app_state and app_insights data streams use flattened type for azure.dimensions.

From my understanding there are 2 approaches since flattened type can't be a dimension at the moment:

• Keep it flattened, add an ingest pipeline to extract fields and mark them as dimensions
• Convert azure.dimensions to group which would provide more flexibility (breaking change?)

[gpop63]

Since app_state has predefined metrics/dimensions, it might make sense to convert azure.dimensions to a group.

app_insights data stream provides more flexibility to the user and they can define the metrics/dimensions they want to fetch. In this case, azure.dimensions could be converted to an object, to have some consistency across all cloud packages.

[gpop63]

Currently blocked because of the metrics grouping issue. #7879

This change aims to fix that: elastic/beats#36634