Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[azquery] add metrics batch feature #21996

Merged
merged 7 commits into from
Nov 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions sdk/monitor/azquery/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
# Release History

## 1.1.1 (Unreleased)
## 1.2.0-beta.1 (2023-11-16)

### Features Added

### Breaking Changes

### Bugs Fixed
* Added `MetricsBatchClient` to support batch querying metrics from Azure resources

### Other Changes
* Enabled spans for distributed tracing.
Expand Down
11 changes: 11 additions & 0 deletions sdk/monitor/azquery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ Example [logs client][example_logs_client]

Example [metrics client][example_metrics_client]

#### Create a metrics batch client

Example metrics batch client

## Key concepts

### Timespan
Expand All @@ -62,6 +66,12 @@ Each set of metric values is a time series with the following characteristics:
- The value itself
- Some metrics may have multiple dimensions as described in [multi-dimensional metrics][multi-metrics]. Custom metrics can have up to 10 dimensions.

### Metrics batch query

A user can also query metrics from multiple resources at once using the query_batch method of MetricsBatchClient. This uses a different API than the MetricsClient and requires that a user pass in a regional endpoint when instantiating the client (for example, "https://westus3.metrics.monitor.azure.com").

Note, each resource must be in the same region as the endpoint passed in when instantiating the client, and each resource must be in the same Azure subscription. Furthermore, the metric namespace that contains the metrics to be queried must also be passed. A list of metric namespaces can be found [here][metric_namespaces].

### Logs query rate limits and throttling

The Log Analytics service applies throttling when the request rate is too high. Limits, such as the maximum number of rows returned, are also applied on the Kusto queries. For more information, see [Query API][service_limits].
Expand Down Expand Up @@ -142,6 +152,7 @@ comments.
[log_analytics_workspace_create]: https://learn.microsoft.com/azure/azure-monitor/logs/quick-create-workspace
[logs_overview]: https://learn.microsoft.com/azure/azure-monitor/logs/data-platform-logs
[metrics_overview]: https://learn.microsoft.com/azure/azure-monitor/essentials/data-platform-metrics
[metric_namespaces]: https://learn.microsoft.com/azure/azure-monitor/reference/supported-metrics/metrics-index#metrics-by-resource-provider
[monitor_docs]: https://learn.microsoft.com/azure/azure-monitor/
[monitor_rest_docs]: https://learn.microsoft.com/rest/api/monitor/
[multi-metrics]: https://learn.microsoft.com/azure/azure-monitor/essentials/data-platform-metrics#multi-dimensional-metrics
Expand Down
2 changes: 1 addition & 1 deletion sdk/monitor/azquery/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "go",
"TagPrefix": "go/monitor/azquery",
"Tag": "go/monitor/azquery_4e1ec7fc44"
"Tag": "go/monitor/azquery_84f4b44326"
}
73 changes: 67 additions & 6 deletions sdk/monitor/azquery/autorest.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ title: Logs Query Client
clear-output-folder: false
go: true
input-file:
- https://github.com/Azure/azure-rest-api-specs/blob/72427ef3ff5875bd8409ef112ef5e6f3cf2b8795/specification/operationalinsights/data-plane/Microsoft.OperationalInsights/stable/2022-10-27/OperationalInsights.json
- https://github.com/Azure/azure-rest-api-specs/blob/21f5332f2dc7437d1446edf240e9a3d4c90c6431/specification/operationalinsights/data-plane/Microsoft.OperationalInsights/stable/2022-10-27/OperationalInsights.json
license-header: MICROSOFT_MIT_NO_VERSION
module: github.com/Azure/azure-sdk-for-go/sdk/monitor/azquery
openapi-type: "data-plane"
output-folder: ../azquery
security: "AADToken"
use: "@autorest/[email protected].60"
use: "@autorest/[email protected].61"
inject-spans: true
version: "^3.0.0"

Expand Down Expand Up @@ -160,9 +160,9 @@ directive:
``` yaml
title: Metrics Query Client
input-file:
- https://github.com/Azure/azure-rest-api-specs/blob/dba6ed1f03bda88ac6884c0a883246446cc72495/specification/monitor/resource-manager/Microsoft.Insights/stable/2018-01-01/metricDefinitions_API.json
- https://github.com/Azure/azure-rest-api-specs/blob/dba6ed1f03bda88ac6884c0a883246446cc72495/specification/monitor/resource-manager/Microsoft.Insights/stable/2018-01-01/metrics_API.json
- https://github.com/Azure/azure-rest-api-specs/blob/dba6ed1f03bda88ac6884c0a883246446cc72495/specification/monitor/resource-manager/Microsoft.Insights/preview/2017-12-01-preview/metricNamespaces_API.json
- https://github.com/Azure/azure-rest-api-specs/blob/21f5332f2dc7437d1446edf240e9a3d4c90c6431/specification/monitor/resource-manager/Microsoft.Insights/stable/2018-01-01/metricDefinitions_API.json
- https://github.com/Azure/azure-rest-api-specs/blob/21f5332f2dc7437d1446edf240e9a3d4c90c6431/specification/monitor/resource-manager/Microsoft.Insights/stable/2018-01-01/metrics_API.json
- https://github.com/Azure/azure-rest-api-specs/blob/21f5332f2dc7437d1446edf240e9a3d4c90c6431/specification/monitor/resource-manager/Microsoft.Insights/preview/2017-12-01-preview/metricNamespaces_API.json

directive:
# rename metric operations to generate as a separate metrics client
Expand Down Expand Up @@ -200,10 +200,71 @@ directive:
- from: options.go
where: $
transform: return $.replace(/Aggregation \*string/g, "Aggregation []*AggregationType");
- from: metrics_client.go
- from:
- metrics_client.go
- metricsbatch_client.go
where: $
transform: return $.replace(/\*options.Aggregation/g, "aggregationTypeToString(options.Aggregation)");
- from: swagger-document
where: $.parameters.AggregationsParameter
transform: $["description"] = "The list of aggregation types to retrieve"
```

``` yaml
title: Metrics Batch Query Client
input-file: https://github.com/Azure/azure-rest-api-specs/blob/21f5332f2dc7437d1446edf240e9a3d4c90c6431/specification/monitor/data-plane/Microsoft.Insights/preview/2023-05-01-preview/metricBatch.json

directive:
# rename Batch to QueryBatch
- rename-operation:
from: MetricsBatch_Batch
to: MetricsBatch_QueryBatch

# Rename MetricResultsResponse
- rename-model:
from: MetricResultsResponse
to: MetricResults
- from:
- models.go
- models_serde.go
where: $
transform: return $.replace(/MetricResultsValuesItem/g, "MetricValues");
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items
transform: $["description"] = "Metric data values."

# fix casing, rename batch metric fields
- from: swagger-document
where: $.parameters.StartTimeParameter
transform: $["x-ms-client-name"] = "StartTime"
- from: swagger-document
where: $.parameters.EndTimeParameter
transform: $["x-ms-client-name"] = "EndTime"
- from: swagger-document
where: $.definitions.ResourceIdList.properties.resourceids
transform: $["x-ms-client-name"] = "ResourceIDs"
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items.properties.starttime
transform: $["x-ms-client-name"] = "StartTime"
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items.properties.endtime
transform: $["x-ms-client-name"] = "EndTime"
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items.properties.resourceid
transform: $["x-ms-client-name"] = "ResourceID"
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items.properties.resourceregion
transform: $["x-ms-client-name"] = "ResourceRegion"
- from: swagger-document
where: $.definitions.MetricResults.properties.values.items.properties.value
transform: $["x-ms-client-name"] = "Values"

# delete unused error models
- from: models.go
where: $
transform: return $.replace(/((?:\/\/.*\s)+|)type AdditionalInfoErrorResponse.+\{(?:\s.+\s)+\}\s/g, "");
- from: models_serde.go
where: $
transform: return $.replace(/(?:\/\/.*\s)+func \(\w \*?AdditionalInfoErrorResponse.*\{\s(?:.+\s)+\}\s/g, "");

```
24 changes: 24 additions & 0 deletions sdk/monitor/azquery/custom_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ type LogsClientOptions struct {
azcore.ClientOptions
}

// MetricsBatchClientOptions contains optional settings for MetricsBatchClient.
type MetricsBatchClientOptions struct {
azcore.ClientOptions
}

// NewLogsClient creates a client that accesses Azure Monitor logs data.
func NewLogsClient(credential azcore.TokenCredential, options *LogsClientOptions) (*LogsClient, error) {
if options == nil {
Expand Down Expand Up @@ -74,6 +79,25 @@ func NewMetricsClient(credential azcore.TokenCredential, options *MetricsClientO
return &MetricsClient{host: c.Endpoint, internal: azcoreClient}, nil
}

// NewMetricsBatchClient creates a client that accesses Azure Monitor metrics data.
// MetricsBatchClient should be used for performing metrics queries on multiple monitored resources in the same region.
// A credential with authorization at the subscription level is required when using this client.
//
// endpoint - The regional endpoint to use, for example https://eastus.metrics.monitor.azure.com.
// The region should match the region of the requested resources. For global resources, the region should be 'global'.
func NewMetricsBatchClient(endpoint string, credential azcore.TokenCredential, options *MetricsBatchClientOptions) (*MetricsBatchClient, error) {
if options == nil {
options = &MetricsBatchClientOptions{}
}

authPolicy := runtime.NewBearerTokenPolicy(credential, []string{"https://metrics.monitor.azure.com" + "/.default"}, nil)
azcoreClient, err := azcore.NewClient(moduleName, version, runtime.PipelineOptions{PerRetry: []policy.Policy{authPolicy}}, &options.ClientOptions)
if err != nil {
return nil, err
}
return &MetricsBatchClient{endpoint: endpoint, internal: azcoreClient}, nil
}

// ErrorInfo - The code and message for an error.
type ErrorInfo struct {
// REQUIRED; A machine readable error code.
Expand Down
57 changes: 57 additions & 0 deletions sdk/monitor/azquery/example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (

var logsClient azquery.LogsClient
var metricsClient azquery.MetricsClient
var metricsBatchClient azquery.MetricsBatchClient
var kustoQuery1 string
var kustoQuery2 string
var kustoQuery3 string
Expand Down Expand Up @@ -55,6 +56,23 @@ func ExampleNewMetricsClient() {
_ = client
}

func ExampleNewMetricsBatchClient() {
// The regional endpoint to use. The region should match the region of the requested resources.
// For global resources, the region should be 'global'
endpoint := "https://eastus.metrics.monitor.azure.com"

cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
//TODO: handle error
}

client, err := azquery.NewMetricsBatchClient(endpoint, cred, nil)
if err != nil {
//TODO: handle error
}
_ = client
}

func ExampleLogsClient_QueryWorkspace() {
// QueryWorkspace allows users to query log data.

Expand Down Expand Up @@ -278,3 +296,42 @@ func ExampleMetricsClient_NewListNamespacesPager() {
}
}
}

func ExampleMetricsBatchClient_QueryBatch() {
// This sample uses the MetricsBatchClient to retrieve the "Ingress"
// metric along with the "Average" aggregation type for multiple resources.
// The query will execute over a timespan of 2 hours with a interval (granularity) of 5 minutes.

// In this example, storage account resource URIs are queried for metrics.
resourceURI1 := "/subscriptions/<id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account-1>"
resourceURI2 := "/subscriptions/<id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account-2>"

res, err := metricsBatchClient.QueryBatch(
context.Background(),
subscriptionID,
"Microsoft.Storage/storageAccounts",
[]string{"Ingress"},
azquery.ResourceIDList{ResourceIDs: to.SliceOfPtrs(resourceURI1, resourceURI2)},
&azquery.MetricsBatchClientQueryBatchOptions{
Aggregation: to.SliceOfPtrs(azquery.AggregationTypeAverage),
StartTime: to.Ptr("2023-11-15"),
EndTime: to.Ptr("2023-11-16"),
Interval: to.Ptr("PT5M"),
},
)
if err != nil {
//TODO: handle error
}

// Print out results
for _, result := range res.Values {
for _, metric := range result.Values {
fmt.Println(*metric.Name.Value + ": " + *metric.DisplayDescription)
for _, timeSeriesElement := range metric.TimeSeries {
for _, metricValue := range timeSeriesElement.Data {
fmt.Printf("The ingress at %v is %v.\n", metricValue.TimeStamp.String(), *metricValue.Average)
}
}
}
}
}
2 changes: 1 addition & 1 deletion sdk/monitor/azquery/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require (
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dnaeon/go-vcr v1.2.0 // indirect
github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
github.com/golang-jwt/jwt/v5 v5.1.0 // indirect
github.com/google/uuid v1.4.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
Expand Down
4 changes: 2 additions & 2 deletions sdk/monitor/azquery/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
github.com/golang-jwt/jwt/v5 v5.0.0 h1:1n1XNM9hk7O9mnQoNBGolZvzebBQ7p93ULHRc28XJUE=
github.com/golang-jwt/jwt/v5 v5.0.0/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang-jwt/jwt/v5 v5.1.0 h1:UGKbA/IPjtS6zLcdB7i5TyACMgSbOTiR8qzXgw8HWQU=
github.com/golang-jwt/jwt/v5 v5.1.0/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
Expand Down
Loading