Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(sqlqueryreceiver) add error count metrics for receiver #3

Closed
wants to merge 1 commit into from
Closed

feat(sqlqueryreceiver) add error count metrics for receiver #3

wants to merge 1 commit into from

Conversation

kasia-kujawa
Copy link
Owner

Format of metric:

# HELP otelcol_receiver_error_count Number of errors
# TYPE otelcol_receiver_error_count counter
otelcol_receiver_error_count{error_type="collect_error",query="query-0: select * from simple_logs",receiver="sqlquery",service_instance_id="9763b7d6-e907-438f-9b70-6e4882756785",service_name="otelcol-sumo",service_version="v0.70.0-sumo-0-198-g210a3dfb53"} 0
otelcol_receiver_error_count{error_type="collect_error",query="query-0: select * from test_logs",receiver="sqlquery/wrong_query",service_instance_id="9763b7d6-e907-438f-9b70-6e4882756785",service_name="otelcol-sumo",service_version="v0.70.0-sumo-0-198-g210a3dfb53"} 2
otelcol_receiver_error_count{error_type="start_error",query="query-0: select * from simple_logs",receiver="sqlquery",service_instance_id="9763b7d6-e907-438f-9b70-6e4882756785",service_name="otelcol-sumo",service_version="v0.70.0-sumo-0-198-g210a3dfb53"} 0
otelcol_receiver_error_count{error_type="start_error",query="query-0: select * from test_logs",receiver="sqlquery/wrong_query",service_instance_id="9763b7d6-e907-438f-9b70-6e4882756785",service_name="otelcol-sumo",service_version="v0.70.0-sumo-0-198-g210a3dfb53"} 0

for configuration:

receivers:
  sqlquery:
    collection_interval: 10s
    driver: postgres
    datasource: "host=127.0.0.1 port=5432 user=otel password=otel sslmode=disable"
    queries:
      - sql: "select * from simple_logs"
        logs:
        - body_column: BODY

  sqlquery/wrong_query:
    collection_interval: 10s
    driver: postgres
    datasource: "host=127.0.0.1 port=5432 user=otel password=otel sslmode=disable"
    queries:
      - sql: "select * from test_logs"
        logs:
        - body_column: BODY
exporters:
  logging:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

service:
  telemetry:
      logs:
        level: "debug"
  pipelines:
    logs:
      receivers:
        - sqlquery
      exporters:
        - logging
    logs/wrong_query:
      receivers:
        - sqlquery/wrong_query
      exporters:
        - logging

@kasia-kujawa kasia-kujawa force-pushed the kk-add-error-count-metric branch from 9df7227 to 0821dcc Compare May 12, 2023 11:14
kasia-kujawa pushed a commit that referenced this pull request May 12, 2023
@dmolenda-sumo
Merge pull request #3 from dmolenda-sumo/sqlqueryreceiver-fix-integra…
ef5782d 
…tion-tests

fix integration tests
kasia-kujawa pushed a commit that referenced this pull request Oct 13, 2024
@daidokoro @kentquirk @TylerHelmuth
[processor/transform] Add Conversion Function to OTTL for Exponential…
74b1048 
… Histo --> Histogram (open-telemetry#33824)

## Description

This PR adds a custom metric function to the transformprocessor to
convert exponential histograms to explicit histograms.

Link to tracking issue: Resolves open-telemetry#33827

**Function Name**
```
convert_exponential_histogram_to_explicit_histogram
```

**Arguments:**

- `distribution` (_upper, midpoint, uniform, random_)
- `ExplicitBoundaries: []float64`

**Usage example:**

```yaml
processors:
  transform:
    error_mode: propagate
    metric_statements:
    - context: metric
      statements:
        - convert_exponential_histogram_to_explicit_histogram("random", [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]) 
```

**Converts:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-31 09:35:25.212037 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
Bucket (32.000000, 64.000000], Count: 10
Bucket (64.000000, 128.000000], Count: 22
Bucket (128.000000, 256.000000], Count: 12
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

**To:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: Delta
HistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-30 21:37:07.830902 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
ExplicitBounds #0: 10.000000
ExplicitBounds #1: 20.000000
ExplicitBounds #2: 30.000000
ExplicitBounds #3: 40.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 60.000000
ExplicitBounds open-telemetry#6: 70.000000
ExplicitBounds open-telemetry#7: 80.000000
ExplicitBounds open-telemetry#8: 90.000000
ExplicitBounds open-telemetry#9: 100.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 2
Buckets #4, Count: 5
Buckets #5, Count: 0
Buckets open-telemetry#6, Count: 3
Buckets open-telemetry#7, Count: 7
Buckets open-telemetry#8, Count: 2
Buckets open-telemetry#9, Count: 4
Buckets open-telemetry#10, Count: 21
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

### Testing

- Several unit tests have been created. We have also tested by ingesting
and converting exponential histograms from the `statsdreceiver` as well
as directly via the `otlpreceiver` over grpc over several hours with a
large amount of data.

- We have clients that have been running this solution in production for
a number of weeks.

### Readme description:

### convert_exponential_hist_to_explicit_hist

`convert_exponential_hist_to_explicit_hist([ExplicitBounds])`

the `convert_exponential_hist_to_explicit_hist` function converts an
ExponentialHistogram to an Explicit (_normal_) Histogram.

`ExplicitBounds` is represents the list of bucket boundaries for the new
histogram. This argument is __required__ and __cannot be empty__.

__WARNING:__

The process of converting an ExponentialHistogram to an Explicit
Histogram is not perfect and may result in a loss of precision. It is
important to define an appropriate set of bucket boundaries to minimize
this loss. For example, selecting Boundaries that are too high or too
low may result histogram buckets that are too wide or too narrow,
respectively.

---------

Co-authored-by: Kent Quirk <[email protected]>
Co-authored-by: Tyler Helmuth <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kasia-kujawa