Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add support to display Data Prepper metrics #1383

Open
kkondaka opened this issue Jan 11, 2023 · 1 comment
Open

[FEATURE] Add support to display Data Prepper metrics #1383

kkondaka opened this issue Jan 11, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@kkondaka
Copy link

kkondaka commented Jan 11, 2023

Is your feature request related to a problem?
Data Prepper (OSIS) added support for generating metrics from logs and traces. These metrics are of OTEL format (initially, SUMMARY and HISTOGRAM kinds are added). We need observability to interpret the OTEL format and display them accordingly.
.

What solution would you like?
The solution should be able to read the metrics from a specific index and interpret the fields as per OTEL format.
The index name to use for this would have a well known prefix (Data Prepper team is looking into this).

We need a way to visualize the histogram data as a histogram. Also when the time duration covers more than one histogram event, the data in different buckets need to be added to show the correct values in each buckets in the visualization.

What alternatives have you considered?
Alternative approach for having index with well known prefix is to take the index name as user input.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

@kkondaka kkondaka added enhancement New feature or request untriaged labels Jan 11, 2023
@kkondaka kkondaka changed the title [FEATURE] Add support to take custom index name and meta data [FEATURE] Add support to display Data Prepper metrics Jan 11, 2023
@kkondaka
Copy link
Author

Example OTEL metric data stored in opensearch

{
  "_index": "log-metrics",
  "_id": "HucJnoUB6OSw6TpavkQW",
  "_version": 1,
  "_score": 0,
  "_source": {
    "max": 74910000,
    "kind": "HISTOGRAM",
    "buckets": [
      {
        "min": -3.4028234663852886e+38,
        "max": 0.1,
        "count": 0
      },
      {
        "min": 0.1,
        "max": 25000000,
        "count": 1060
      },
      {
        "min": 25000000,
        "max": 75000000,
        "count": 2440
      },
      {
        "min": 75000000,
        "max": 100000000,
        "count": 0
      },
      {
        "min": 100000000,
        "max": 500000000,
        "count": 0
      },
      {
        "min": 500000000,
        "max": 3.4028234663852886e+38,
        "count": 0
      }
    ],
    "count": 3500,
    "bucketCountsList": [
      0,
      1060,
      2440,
      0,
      0,
      0
    ],
    "description": "Histogram of bytes in the events",
    "sum": 134772700000,
    "unit": "Bytes",
    "aggregationTemporality": "AGGREGATION_TEMPORALITY_DELTA",
    "min": 3000000,
    "bucketCounts": 6,
    "name": "histogram",
    "startTime": "2023-01-10T23:34:52.536379Z",
    "explicitBoundsCount": 5,
    "time": "2023-01-10T23:34:52.637805Z",
    "explicitBounds": [
      0.1,
      25000000,
      75000000,
      100000000,
      500000000
    ],
    "request": "/big.pdf",
    "clientip": "127.0.0.1",
    "aggr._duration": 101426000,
    "histogram_key": "bytes"
  },
  "fields": {
    "startTime": [
      "2023-01-10T23:34:52.536Z"
    ],
    "time": [
      "2023-01-10T23:34:52.637Z"
    ]
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants