Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[exporter/splunkhec] Data race detected when gzip compression is enabled #17083

Closed
hvaghani221 opened this issue Dec 19, 2022 · 0 comments
Closed

[exporter/splunkhec] Data race detected when gzip compression is enabled #17083

hvaghani221 opened this issue Dec 19, 2022 · 0 comments
Labels
bug Something isn't working needs triage New item requiring triage

Comments

@hvaghani221
Copy link
Member

hvaghani221 commented Dec 19, 2022 •
edited
Loading

Component(s)

exporter/splunkhec

What happened?

Description

When executing the otel agent with a race flag, it detects some data race in gzip package resulting in invalid index access (i.e. index out of the bound access)

Steps to Reproduce

Run the agent that scrapes metrics from following deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mock-server
  template:
    metadata:
      labels:
        app: mock-server
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /stats/prometheus
        prometheus.io/port: "15020"

    spec:
      containers:
      - name: mock-server
        image: hvaghani/metricgen:1.0.0
        ports:
        - containerPort: 15020
      limit:

Expected Result

It should not throw any error

Actual Result

Agent panics because of invalid index access.

Collector version

0.66.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  prometheus_simple:
    collection_interval: 10s
    use_service_account: false
    endpoint: "localhost:15020"
    metrics_path: "/stats/prometheus"
  prometheus_simple/1:
    collection_interval: 11s
    use_service_account: false
    endpoint: "localhost:15020"
    metrics_path: "/stats/prometheus"
  prometheus_simple/2:
    collection_interval: 6s
    use_service_account: false
    endpoint: "localhost:15020"
    metrics_path: "/stats/prometheus"
  prometheus_simple/3:
    collection_interval: 9s
    use_service_account: false
    endpoint: "localhost:15020"
    metrics_path: "/stats/prometheus"
exporters:
  logging:
  splunk_hec:
    token: "00000000-0000-0000-0000-000000000000"
    endpoint: "https://splunk:8088/services/collector"
    source: "otel"
    sourcetype: "v1"
    index: "metric1"
    tls:
      insecure_skip_verify: true

service:
  pipelines:
    metrics:
      receivers: [prometheus_simple,prometheus_simple,prometheus_simple,prometheus_simple/1,prometheus_simple/2,prometheus_simple/3]
      exporters: [logging, splunk_hec]

Log output

WARNING: DATA RACE
Write at 0x00c0008a06c8 by goroutine 95:
  compress/gzip.(*Writer).Write()
      /usr/local/Cellar/go/1.19/libexec/src/compress/gzip/gzip.go:168 +0x52e
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*cancellableGzipWriter).Write()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:186 +0x55e
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*bufferState).accept()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:111 +0x530
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*client).pushMetricsRecords()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:466 +0x56b
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*client).pushMetricsDataInBatches()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:586 +0x779
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*client).pushMetricsData()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:228 +0x11a
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*client).pushMetricsData-fm()
      <autogenerated>:1 +0x64
  go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:65 +0x7e
  go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:208 +0x12b
  go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:387 +0x6b0
  go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:136 +0xe4
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:205 +0x64
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61 +0x151

Previous read at 0x00c0008a06c8 by goroutine 184:
  compress/gzip.(*Writer).Flush()
      /usr/local/Cellar/go/1.19/libexec/src/compress/gzip/gzip.go:209 +0x44
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*cancellableGzipWriter).close()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:190 +0x4c
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*bufferState).Close()
      /Users/hvaghani/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter/client.go:83 +0x7e
  net/http.(*readTrackingBody).Close()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:643 +0x57
  net/http.(*transferWriter).writeBody()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transfer.go:384 +0x8a1
  net/http.(*Request).write()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/request.go:701 +0x11e7
  net/http.(*persistConn).writeLoop()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:2395 +0x2e4
  net/http.(*Transport).dialConn.func6()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:1752 +0x39

Goroutine 95 (running) created at:
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:56 +0x67
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:204 +0x152
  go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func1()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:177 +0xe9
  go.opentelemetry.io/collector/component.StartFunc.Start()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/component.go:84 +0x97
  go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsExporter).Start()
      <autogenerated>:1 +0x29
  github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter.(*baseMetricsExporter).Start()
      <autogenerated>:1 +0x81
  go.opentelemetry.io/collector/service.(*builtPipelines).StartAll()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/service/pipelines.go:81 +0xb0a
  go.opentelemetry.io/collector/service.(*service).Start()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/service/service.go:97 +0x491
  go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/service/collector.go:163 +0x4f6
  go.opentelemetry.io/collector/service.(*Collector).Run()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/service/collector.go:188 +0x73
  go.opentelemetry.io/collector/service.NewCommand.func1()
      /Users/hvaghani/go/pkg/mod/go.opentelemetry.io/[email protected]/service/command.go:53 +0x7e7
  github.com/spf13/cobra.(*Command).execute()
      /Users/hvaghani/go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0xbf4
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/hvaghani/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x5fa
  github.com/spf13/cobra.(*Command).Execute()
      /Users/hvaghani/go/pkg/mod/github.com/spf13/[email protected]/command.go:968 +0x99
  main.runInteractive()
      /Users/hvaghani/go/src/github.com/harshit-splunk/agent/ocb/main.go:32 +0xa0
  main.run()
      /Users/hvaghani/go/src/github.com/harshit-splunk/agent/ocb/main_others.go:11 +0x217
  main.main()
      /Users/hvaghani/go/src/github.com/harshit-splunk/agent/ocb/main.go:25 +0x49

Goroutine 184 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:1752 +0x2619
  net/http.(*Transport).dialConnFor()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:1450 +0x13a
  net/http.(*Transport).queueForDial.func1()
      /usr/local/Cellar/go/1.19/libexec/src/net/http/transport.go:1419 +0x47

Additional context

Refer signalfx/splunk-otel-collector-chart#600 for more information
@hvaghani221 hvaghani221 added bug Something isn't working needs triage New item requiring triage labels Dec 19, 2022
@hvaghani221 hvaghani221 changed the title Data race detected when gzip compression is enabled [exporter/splunkhec] Data race detected when gzip compression is enabled Dec 19, 2022
@hvaghani221 hvaghani221 mentioned this issue Dec 19, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage New item requiring triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dmitryax @hvaghani221