Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend OTLP receiver to publish via EMF logs #1375

Merged
merged 4 commits into from
Oct 9, 2024
Merged

Conversation

sky333999
Copy link
Contributor

@sky333999 sky333999 commented Oct 7, 2024

Description of changes

#1361 introduced the support for an OTLP plugin for publishing metrics via PutMetricData to CloudWatch.
This PR extends the ability to use the OTLP plugin to publish metrics as EMF logs via PutLogEvents to CloudWatch.

  • Introduces otlp section under logs -> metrics_collected that can either be a json blob or an array of json blobs for multiple listeners.
  • OTLP Metrics are setup to flow through our a new hostDeltaMetrics/cloudwatchlogs pipeline using:
    • otlpreceiver - to receive the metrics on both HTTP and gRPC endpoints
    • cummulativetodelta processor - to convert cumulatives to deltas by default when publishing to CloudWatch
    • batch processor - to respect the force_flush_interval in the logs section of the agent config
    • awsemfexporter - to publish as EMF logs via PutLogEvents API calls to CloudWatch
  • Serves as a passthrough plugin where all OTLP metrics sent are passed along as CloudWatch metrics without the need for explicit EMF metric declarations. Similarly, all resource and datapoint attributes are converted to dimensions on the metrics without any rollups i.e. they are sent along at the same granularity at which they are received.
  • EMF logs are sent to aws/cwagent log group containing a log stream per unique run of the agent i.e. the log stream is based off a UUID generated on each run. Example log stream: otel-stream-52215dd6-abfa-44cb-a23d-25d79d1278fd.
  • Metrics from EMF logs are extracted into the CWAgent namespace.
  • Supports TLS configuration.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Updated unit tests
  • Manually tested using a OTLP sample app

Sample config:

{
  "logs": {
    "metrics_collected": {
      "otlp": {
        "grpc_endpoint": "0.0.0.0:4317",
        "http_endpoint": "0.0.0.0:4318"
      }
    }
  }
}

Sample EMF log event:

{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent",
            "Dimensions": [
                [
                    "service.name",
                    "roll.value",
                    "telemetry.sdk.language",
                    "telemetry.sdk.name",
                    "telemetry.sdk.version",
                    "telemetry.auto.version"
                ]
            ],
            "Metrics": [
                {
                    "Name": "dice.rolls"
                }
            ]
        }
    ],
    "Timestamp": "1728235699078",
    "Version": "0",
    "roll.value": "2",
    "service.name": "unknown_service",
    "telemetry.auto.version": "0.46b0",
    "telemetry.sdk.language": "python",
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.version": "1.25.0",
    "dice.rolls": 0
}
image

Requirements

Before commit the code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

@sky333999 sky333999 requested a review from a team as a code owner October 7, 2024 06:24
@jefchien jefchien force-pushed the sky333999/otlp-emf branch from b25bbdc to 3663e57 Compare October 9, 2024 04:22
musa-asad
musa-asad previously approved these changes Oct 9, 2024
@sky333999 sky333999 dismissed stale reviews from musa-asad and mitali-salvi via 4739875 October 9, 2024 19:40
@jefchien jefchien merged commit 5cdba47 into main Oct 9, 2024
6 checks passed
@jefchien jefchien deleted the sky333999/otlp-emf branch October 9, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants