Skip to content

Commit

Permalink
[Azure Monitor Exporter] Add network Statsbeat Metrics (Azure#23425)
Browse files Browse the repository at this point in the history
* Implement statsbeat state and begin setting up the statsbeatMetrics.

* Stub statsBeat metrics file.

* Add a number of custom features to the network statsbeat and begin defining them.

* Add _getAzureComputeMetadata logic.

* Get successful count using he observable gauge.

* Add the rest of the counts to the statsbeat metrics.

* Update the getAverageDuration method.

* Add statsbeatMetricHandler to manage the meter provider.

* Update the statsbeatMetricsHandler exporter config.

* Converge the statsbeat logic into one file, begin fixing the average request duration logic, and start outlining where statsbeat changes need to be made in the base exporter.

* Fix issues with statsbeat connection and setup.

* Add duration counting on success and failure requests.

* Make batched observable results in Statsbeat.

* Testing begin.

* Remove isStatsbeat being user configurable.

* Fix typo.

* Address code review comments.

* Remove intentionally broken test.

* Fix dependencies issues and format

* WIP

* WIP

* Manage the isStatsbeatExporter state in the baseExpoerter properly.

* WIP

* Update statsbeat names.

* Revert pnpm-lock.yaml file.

* Fix tests

* Fix how attributes are created and how EU endpoints are set.

* Add statsbeat tests.

* Fix package.json file.

* Continue work on tests.

* Write all non-AzureVM tests.

* Save progress on attempt to fix the Azure VM test.

* Fix Azure VM http call.

* Fix test for AzureVM.

* Remove comments.

* Clean up further comments.

* Update changelog.

* Remove unneded sinon import.

* Fix sinon types import.

* Fix import build issues.

* Update sdk/monitor/monitor-opentelemetry-exporter/CHANGELOG.md

Co-authored-by: Hector Hernandez <[email protected]>

* Fix lint issues.

* Address initial code review comments.

* Clean up and optimize statsbeat metrics collection logic.

* Update statsbeat logic to reduce checks and improve efficiency.

* Fix format.

* Remove last of unnecessary check.

* Create statsbeatExporter to replace use of metricExporter for statsbeat.

* Improve statsbeat metric exporting logic, handle legacy statusCodes, and document statsbeatExporter.

* Fix formatting.

Co-authored-by: Hector Hernandez <[email protected]>
  • Loading branch information
JacksonWeber and hectorhdzg authored Oct 28, 2022
1 parent 48deccf commit 0d72600
Show file tree
Hide file tree
Showing 13 changed files with 983 additions and 13 deletions.
1 change: 1 addition & 0 deletions sdk/monitor/monitor-opentelemetry-exporter/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- Added retriable behavior for 502, 503 and 504 status codes.
- Export Metric attributes and Histogram Min/Max values.
- Added new config options disableOfflineStorage, storageDirectory and exposed ApplicationInsightsClientOptionalParams for HTTP client extra configuration.
- Added Network Statsbeat Metrics.

### Breaking Changes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ export class ApplicationInsightsSampler implements Sampler {

// @public
export abstract class AzureMonitorBaseExporter {
constructor(options?: AzureMonitorExporterOptions);
constructor(options?: AzureMonitorExporterOptions, isStatsbeatExporter?: boolean);
protected _exportEnvelopes(envelopes: TelemetryItem[]): Promise<ExportResult>;
protected _instrumentationKey: string;
protected _shutdown(): Promise<void>;
Expand All @@ -59,6 +59,14 @@ export class AzureMonitorMetricExporter extends AzureMonitorBaseExporter impleme
shutdown(): Promise<void>;
}

// @internal
export class _AzureMonitorStatsbeatExporter extends AzureMonitorBaseExporter implements PushMetricExporter {
constructor(options: AzureMonitorExporterOptions);
export(metrics: ResourceMetrics, resultCallback: (result: ExportResult) => void): Promise<void>;
forceFlush(): Promise<void>;
shutdown(): Promise<void>;
}

// @public
export class AzureMonitorTraceExporter extends AzureMonitorBaseExporter implements SpanExporter {
constructor(options?: AzureMonitorExporterOptions);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

/**
* This example shows how to use
* [@opentelemetry/sdk-metrics-base](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-sdk-metrics-base)
* [@opentelemetry/sdk-metrics](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-sdk-metrics-base)
* to generate Metrics in a simple Node.js application and export them to Azure Monitor.
*
* @summary Basic use of Metrics in Node.js application.
Expand Down
62 changes: 55 additions & 7 deletions sdk/monitor/monitor-opentelemetry-exporter/src/export/base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import { PersistentStorage, Sender } from "../types";
import { isRetriable, BreezeResponse } from "../utils/breezeUtils";
import { DEFAULT_BREEZE_ENDPOINT, ENV_CONNECTION_STRING } from "../Declarations/Constants";
import { TelemetryItem as Envelope } from "../generated";
import { StatsbeatMetrics } from "./statsbeat/statsbeatMetrics";
import { MAX_STATSBEAT_FAILURES } from "./statsbeat/types";

const DEFAULT_BATCH_SEND_RETRY_INTERVAL_MS = 60_000;
/**
Expand All @@ -20,13 +22,15 @@ export abstract class AzureMonitorBaseExporter {
/**
* Instrumentation key to be used for exported envelopes
*/
protected _instrumentationKey: string;
protected _instrumentationKey: string = "";
private _endpointUrl: string = "";
private readonly _persister: PersistentStorage;
private readonly _sender: Sender;
private _numConsecutiveRedirects: number;
private _retryTimer: NodeJS.Timer | null;
private _endpointUrl: string;

private _statsbeatMetrics: StatsbeatMetrics | undefined;
private _isStatsbeatExporter: boolean;
private _statsbeatFailureCount: number = 0;
private _batchSendRetryIntervalMs: number = DEFAULT_BATCH_SEND_RETRY_INTERVAL_MS;
/**
* Exporter internal configuration
Expand All @@ -37,12 +41,14 @@ export abstract class AzureMonitorBaseExporter {
* Initializes a new instance of the AzureMonitorBaseExporter class.
* @param AzureMonitorExporterOptions - Exporter configuration.
*/
constructor(options: AzureMonitorExporterOptions = {}) {
constructor(options: AzureMonitorExporterOptions = {}, isStatsbeatExporter?: boolean) {
this._options = options;
this._numConsecutiveRedirects = 0;
this._instrumentationKey = "";
this._endpointUrl = DEFAULT_BREEZE_ENDPOINT;
const connectionString = this._options.connectionString || process.env[ENV_CONNECTION_STRING];
this._isStatsbeatExporter = isStatsbeatExporter ? isStatsbeatExporter : false;

if (connectionString) {
const parsedConnectionString = ConnectionStringParser.parse(connectionString);
this._instrumentationKey =
Expand All @@ -57,9 +63,13 @@ export abstract class AzureMonitorBaseExporter {
diag.error(message);
throw new Error(message);
}

this._sender = new HttpSender(this._endpointUrl, this._options);
this._persister = new FileSystemPersist(this._instrumentationKey, this._options);

if (!this._isStatsbeatExporter) {
// Initialize statsbeatMetrics
this._statsbeatMetrics = new StatsbeatMetrics(this._instrumentationKey, this._endpointUrl);
}
this._retryTimer = null;
diag.debug("AzureMonitorExporter was successfully setup");
}
Expand Down Expand Up @@ -93,12 +103,18 @@ export abstract class AzureMonitorBaseExporter {
*/
protected async _exportEnvelopes(envelopes: Envelope[]): Promise<ExportResult> {
diag.info(`Exporting ${envelopes.length} envelope(s)`);

if (envelopes.length < 1) {
return { code: ExportResultCode.SUCCESS };
}

try {
const startTime = new Date().getTime();
const { result, statusCode } = await this._sender.send(envelopes);
const endTime = new Date().getTime();
const duration = endTime - startTime;
this._numConsecutiveRedirects = 0;

if (statusCode === 200) {
// Success -- @todo: start retry timer
if (!this._retryTimer) {
Expand All @@ -108,9 +124,14 @@ export abstract class AzureMonitorBaseExporter {
}, this._batchSendRetryIntervalMs);
this._retryTimer.unref();
}
// If we are not exportings statsbeat and statsbeat is not disabled -- count success
this._statsbeatMetrics?.countSuccess(duration);
return { code: ExportResultCode.SUCCESS };
} else if (statusCode && isRetriable(statusCode)) {
// Failed -- persist failed data
if (statusCode === 429 || statusCode === 439) {
this._statsbeatMetrics?.countThrottle(statusCode);
}
if (result) {
diag.info(result);
const breezeResponse = JSON.parse(result) as BreezeResponse;
Expand All @@ -123,19 +144,29 @@ export abstract class AzureMonitorBaseExporter {
});
}
if (filteredEnvelopes.length > 0) {
this._statsbeatMetrics?.countRetry(statusCode);
// calls resultCallback(ExportResult) based on result of persister.push
return await this._persist(filteredEnvelopes);
}
// Failed -- not retriable
this._statsbeatMetrics?.countFailure(duration, statusCode);
return {
code: ExportResultCode.FAILED,
};
} else {
// calls resultCallback(ExportResult) based on result of persister.push
this._statsbeatMetrics?.countRetry(statusCode);
return await this._persist(envelopes);
}
} else {
// Failed -- not retriable
if (this._statsbeatMetrics) {
if (statusCode) {
this._statsbeatMetrics.countFailure(duration, statusCode);
}
} else {
this._incrementStatsbeatFailure();
}
return {
code: ExportResultCode.FAILED,
};
Expand All @@ -161,19 +192,25 @@ export abstract class AzureMonitorBaseExporter {
}
}
} else {
return { code: ExportResultCode.FAILED, error: new Error("Circular redirect") };
let redirectError = new Error("Circular redirect");
this._statsbeatMetrics?.countException(redirectError);
return { code: ExportResultCode.FAILED, error: redirectError };
}
} else if (restError.statusCode && isRetriable(restError.statusCode)) {
this._statsbeatMetrics?.countRetry(restError.statusCode);
return await this._persist(envelopes);
}
if (this._isNetworkError(restError)) {
if (restError.statusCode) {
this._statsbeatMetrics?.countRetry(restError.statusCode);
}
diag.error(
"Retrying due to transient client side error. Error message:",
restError.message
);
return await this._persist(envelopes);
}

this._statsbeatMetrics?.countException(restError);
diag.error(
"Envelopes could not be exported and are not retriable. Error message:",
restError.message
Expand All @@ -182,6 +219,17 @@ export abstract class AzureMonitorBaseExporter {
}
}

// Disable collection of statsbeat metrics after max failures
private _incrementStatsbeatFailure() {
this._statsbeatFailureCount++;
if (this._statsbeatFailureCount > MAX_STATSBEAT_FAILURES) {
this._isStatsbeatExporter = false;
this._statsbeatMetrics?.shutdown();
this._statsbeatMetrics = undefined;
this._statsbeatFailureCount = 0;
}
}

private async _sendFirstPersistedFile(): Promise<void> {
try {
const envelopes = (await this._persister.shift()) as Envelope[] | null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import {
} from "@opentelemetry/sdk-metrics";
import { ExportResult, ExportResultCode, suppressTracing } from "@opentelemetry/core";
import { AzureMonitorBaseExporter } from "./base";
import { AzureMonitorExporterOptions } from "../config";
import { TelemetryItem as Envelope } from "../generated";
import { resourceMetricsToEnvelope } from "../utils/metricUtils";
import { AzureMonitorExporterOptions } from "../config";

/**
* Azure Monitor OpenTelemetry Metric Exporter.
Expand All @@ -33,6 +33,7 @@ export class AzureMonitorMetricExporter
* Initializes a new instance of the AzureMonitorMetricExporter class.
* @param AzureExporterConfig - Exporter configuration.
*/

constructor(options: AzureMonitorExporterOptions = {}) {
super(options);
this._aggregationTemporality = AggregationTemporality.CUMULATIVE;
Expand Down
69 changes: 69 additions & 0 deletions sdk/monitor/monitor-opentelemetry-exporter/src/export/statsbeat.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
import { context } from "@opentelemetry/api";
import { PushMetricExporter, ResourceMetrics } from "@opentelemetry/sdk-metrics";
import { ExportResult, ExportResultCode, suppressTracing } from "@opentelemetry/core";
import { AzureMonitorExporterOptions } from "../config";
import { TelemetryItem as Envelope } from "../generated";
import { resourceMetricsToEnvelope } from "../utils/metricUtils";
import { AzureMonitorBaseExporter } from "./base";

/**
* @internal
* Azure Monitor Statsbeat Exporter
*/
export class _AzureMonitorStatsbeatExporter
extends AzureMonitorBaseExporter
implements PushMetricExporter
{
/**
* Flag to determine if the Exporter is shutdown.
*/
private _isShutdown = false;

/**
* Initializes a new instance of the AzureMonitorStatsbeatExporter class.
* @param options - Exporter configuration
*/
constructor(options: AzureMonitorExporterOptions) {
super(options, true);
}

/**
* Export Statsbeat metrics.
*/
async export(
metrics: ResourceMetrics,
resultCallback: (result: ExportResult) => void
): Promise<void> {
if (this._isShutdown) {
setTimeout(() => resultCallback({ code: ExportResultCode.FAILED }), 0);
return;
}

let envelopes: Envelope[] = resourceMetricsToEnvelope(
metrics,
this._instrumentationKey,
true // isStatsbeat flag passed to create a Statsbeat envelope.
);
// Supress tracing until OpenTelemetry Metrics SDK support it
context.with(suppressTracing(context.active()), async () => {
resultCallback(await this._exportEnvelopes(envelopes));
});
}

/**
* Shutdown AzureMonitorStatsbeatExporter.
*/
public async shutdown(): Promise<void> {
this._isShutdown = true;
return this._shutdown();
}

/**
* Force flush.
*/
public async forceFlush(): Promise<void> {
return Promise.resolve();
}
}
Loading

0 comments on commit 0d72600

Please sign in to comment.