[libbeat] TestKafkaPublish fails under Go 1.8 #30954

andrewkroh · 2022-03-22T14:33:19Z

When running TestKafkaPublish under Go 1.18 the integration test is timing-out.

=== RUN   TestKafkaPublish
=== RUN   TestKafkaPublish/run_test(0):_publish_single_event_to_test_topic
{"log.level":"debug","@timestamp":"2022-03-22T13:26:20.356Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:20.357Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:20.836Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.574Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.574Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.574Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(1):_publish_single_event_with_topic_from_type
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.574Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.575Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:22.855Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.206Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.206Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.207Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(2):_publish_single_event_with_formating_to_test_topic
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.207Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.208Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:24.255Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:25.899Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:25.900Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:25.904Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(3):_batch_publish_to_test_topic
{"log.level":"debug","@timestamp":"2022-03-22T13:26:25.914Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:25.914Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:26.258Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:26.258Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:26.258Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:26.259Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:26.259Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:27.917Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:27.918Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:27.919Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(4):_batch_publish_to_test_topic_from_type
{"log.level":"debug","@timestamp":"2022-03-22T13:26:27.920Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:27.938Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:28.146Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:28.147Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:28.147Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:28.147Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:28.147Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.656Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.661Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.663Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(5):_batch_publish_with_random_partitioner
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.663Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.665Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:29.949Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.530Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.530Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.531Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(6):_batch_publish_with_round_robin_partitioner
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.531Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.533Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:31.807Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:33.145Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:33.145Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:33.168Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(7):_batch_publish_with_hash_partitioner_without_key_(fallback_to_random)
{"log.level":"debug","@timestamp":"2022-03-22T13:26:33.211Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:33.212Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:35.141Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:36.659Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:36.659Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:36.686Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(8):_batch_publish_with_hash_partitioner_with_key
{"log.level":"debug","@timestamp":"2022-03-22T13:26:36.686Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:36.687Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:37.012Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:38.552Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:38.552Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:38.556Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(9):_batch_publish_with_fields_hash_partitioner
{"log.level":"debug","@timestamp":"2022-03-22T13:26:38.557Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:38.558Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:39.032Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.494Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.494Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.506Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(10):_publish_single_event_to_test_topic
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.507Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.508Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9092]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:40.757Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:42.522Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":141},"message":"closed kafka client","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:42.522Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":262},"message":"Stop kafka ack worker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:42.563Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":352},"message":"Stop kafka error handler","ecs.version":"1.6.0"}
=== RUN   TestKafkaPublish/run_test(11):_publish_single_event_to_test_topic_over_SASL/SCRAM
{"log.level":"debug","@timestamp":"2022-03-22T13:26:42.564Z","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-03-22T13:26:42.564Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:42.565Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":120},"message":"connect: [kafka:9093]","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:44.429Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":392},"message":"finished kafka batch","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-22T13:26:44.429Z","log.logger":"kafka","log.origin":{"file.name":"kafka/client.go","file.line":406},"message":"Kafka publish failed with: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","ecs.version":"1.6.0"}
coverage: 73.4% of statements
panic: test timed out after 10m0s

goroutine 756 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
	/usr/local/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc000581860, {0x1204456?, 0x8d5f85?}, 0x12547c8)
	/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc00064d7a0?)
	/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc000581860, 0xc000647ca0)
	/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0002e77c0?, {0x1968d20, 0x5, 0x5}, {0x7f41da72b108?, 0x40?, 0x19cdc60?})
	/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0002e77c0)
	/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
	_testmain.go:113 +0x255

goroutine 6 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0001064e0)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 7 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0001065b0)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 8 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000106680)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 9 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000106750)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 10 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000106820)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 11 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0001068f0)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 12 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0001069c0)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 13 [chan receive, 10 minutes]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000106a90)
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:212 +0x94
created by github.com/klauspost/compress/zstd.newBlockDec
	/go/pkg/mod/github.com/klauspost/[email protected]/zstd/blockdec.go:118 +0x165

goroutine 639 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*partitionConsumer).dispatcher(0xc0001fa280)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:343 +0x3d
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:161 +0x2ca

goroutine 72 [chan receive]:
github.com/rcrowley/go-metrics.(*meterArbiter).tick(0x19cd5a0)
	/go/pkg/mod/github.com/rcrowley/[email protected]/meter.go:239 +0x2a
created by github.com/rcrowley/go-metrics.NewMeter
	/go/pkg/mod/github.com/rcrowley/[email protected]/meter.go:46 +0xd3

goroutine 132 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0002e11e0, {0xc0002e2340?, 0xc0005a7170?}, 0xc0009f2550)
	/usr/local/go/src/testing/testing.go:1487 +0x37a
github.com/elastic/beats/v7/libbeat/outputs/kafka.TestKafkaPublish(0xc0002e11e0?)
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:258 +0x250f
testing.tRunner(0xc0002e11e0, 0x12547c8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1486 +0x35f

goroutine 720 [select, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic(0xc0009f5860, {0xc00058a760, 0x20}, 0x1, 0xc0009b37c0?)
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:509 +0x4d3
github.com/elastic/beats/v7/libbeat/outputs/kafka.TestKafkaPublish.func1(0xc0009f5860)
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:290 +0x4a5
testing.tRunner(0xc0009f5860, 0xc0009f2550)
	/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1486 +0x35f

goroutine 578 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x5857d0?, {0x13e55f0, 0xc0002e74a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 581 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x727fd0?, {0x13e55f0, 0xc00068a1e0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 634 [select, 9 minutes]:
github.com/Shopify/sarama.(*client).backgroundMetadataUpdater(0xc0000fc1b0)
	/go/pkg/mod/github.com/elastic/[email protected]/client.go:823 +0x128
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.NewClient
	/go/pkg/mod/github.com/elastic/[email protected]/client.go:184 +0x3d6

goroutine 263 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x504fd0?, {0x13e55f0, 0xc0000be3c0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 754 [select]:
github.com/Shopify/sarama.(*Broker).sendAndReceive(0xffffffffffffffff?, {0x13e6810, 0xc0004440c0}, {0x13e6858, 0xc000444100})
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:820 +0x12c
github.com/Shopify/sarama.(*Broker).Fetch(0xc000320e28?, 0xc000822f60?)
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:366 +0x4f
github.com/Shopify/sarama.(*brokerConsumer).fetchNewMessages(0xc000648640)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:948 +0x634
github.com/Shopify/sarama.(*brokerConsumer).subscriptionConsumer(0xc000648640)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:805 +0xe5
github.com/Shopify/sarama.withRecover(0x3836373334363832?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).newBrokerConsumer
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:746 +0x216

goroutine 755 [select, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x72bfd0?, {0x13e55f0, 0xc0001fa280})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 692 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*partitionConsumer).dispatcher(0xc0001fa320)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:343 +0x3d
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:161 +0x2ca

goroutine 145 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x505fd0?, {0x13e55f0, 0xc000140000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 28 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x507fd0?, {0x13e55f0, 0xc0008860a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 46 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0000be5a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 569 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x741fd0?, {0x13e55f0, 0xc0002f8140})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 65 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3297d0?, {0x13e55f0, 0xc0002e6000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 196 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0002e60a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 722 [select, 9 minutes]:
github.com/Shopify/sarama.(*client).backgroundMetadataUpdater(0xc0004b8750)
	/go/pkg/mod/github.com/elastic/[email protected]/client.go:823 +0x128
github.com/Shopify/sarama.withRecover(0xc00072a798?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.NewClient
	/go/pkg/mod/github.com/elastic/[email protected]/client.go:184 +0x3d6

goroutine 266 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0001fa820})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 31 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0002e6320})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 253 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x65a1f0?, {0x13e55f0, 0xc0000be0a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 508 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x502fd0?, {0x13e55f0, 0xc0002f8320})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 640 [chan receive]:
github.com/Shopify/sarama.(*partitionConsumer).responseFeeder(0xc0001fa280)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:463 +0x92
github.com/Shopify/sarama.withRecover(0x5a3734332e30323a?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:162 +0x345

goroutine 736 [chan receive]:
github.com/Shopify/sarama.(*partitionConsumer).responseFeeder(0xc0000be6e0)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:463 +0x92
github.com/Shopify/sarama.withRecover(0xc000325f98?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:162 +0x345

goroutine 323 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x5887d0?, {0x13e55f0, 0xc000886000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 199 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x327fd0?, {0x13e55f0, 0xc0001fa1e0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 310 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc00057e000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 691 [IO wait]:
internal/poll.runtime_pollWait(0x7f41b08ba7b8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0008ee080?, 0xc0002a1000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0008ee080, {0xc0002a1000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0008ee080, {0xc0002a1000?, 0x6fc23a8c2?, 0x72?})
	/usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000258098, {0xc0002a1000?, 0x92ac071646?, 0x19cdc60?})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000539b60, {0xc00003aeb8, 0x8, 0x19cdc60?})
	/usr/local/go/src/bufio/bufio.go:236 +0x1b4
github.com/Shopify/sarama.(*bufConn).Read(0x10000c000539e00?, {0xc00003aeb8?, 0xc00031ade0?, 0x7c3b96?})
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:107 +0x25
io.ReadAtLeast({0x7f419a7bfc00, 0xc0009dc300}, {0xc00003aeb8, 0x8, 0x8}, 0x8)
	/usr/local/go/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:350
github.com/Shopify/sarama.(*Broker).readFull(0xc0008d2000, {0xc00003aeb8, 0x8, 0x8})
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:748 +0xc5
github.com/Shopify/sarama.(*Broker).responseReceiver(0xc0008d2000)
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:904 +0x145
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*Broker).Open.func1
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:212 +0xb2b

goroutine 484 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x73ffd0?, {0x13e55f0, 0xc000140140})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 315 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3597d0?, {0x13e55f0, 0xc0008f0000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 297 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc00068a000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 381 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x5057d0?, {0x13e55f0, 0xc000886460})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 641 [select]:
github.com/Shopify/sarama.(*brokerConsumer).subscriptionManager(0xc000648640)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:772 +0x252
github.com/Shopify/sarama.withRecover(0xc000729f98?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).newBrokerConsumer
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:745 +0x19a

goroutine 420 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3557d0?, {0x13e55f0, 0xc0002f8000})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 423 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc00068a0a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 319 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc00068a140})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 689 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3577d0?, {0x13e55f0, 0xc0001410e0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 416 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x589fd0?, {0x13e55f0, 0xc000886780})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 206 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc000886820})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 538 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x352fd0?, {0x13e55f0, 0xc0002e7400})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 465 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3277d0?, {0x13e55f0, 0xc0002f83c0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 594 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc000886640})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 735 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*partitionConsumer).dispatcher(0xc0000be6e0)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:343 +0x3d
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:161 +0x2ca

goroutine 723 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*asyncProducer).dispatcher(0xc000169e30)
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:330 +0xd1
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.newAsyncProducer
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:166 +0x285

goroutine 727 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*topicProducer).dispatch(0xc0009b3800)
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:412 +0x54
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*asyncProducer).newTopicProducer
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:407 +0x1f6

goroutine 726 [chan receive, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.(*client).errorWorker(0xc000192800, 0x0?)
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/client.go:269 +0x11d
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.(*client).Connect
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/client.go:133 +0x39b

goroutine 657 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x3667d0?, {0x13e55f0, 0xc000140280})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 660 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0002f80a0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 724 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*asyncProducer).retryHandler(0xc000169e30)
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:1038 +0x196
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.newAsyncProducer
	/go/pkg/mod/github.com/elastic/[email protected]/async_producer.go:167 +0x305

goroutine 725 [chan receive, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.(*client).successWorker(0x0?, 0x0?)
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/client.go:258 +0xe7
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.(*client).Connect
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/client.go:132 +0x31b

goroutine 737 [select, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0000be6e0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 693 [chan receive]:
github.com/Shopify/sarama.(*partitionConsumer).responseFeeder(0xc0001fa320)
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:463 +0x92
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*consumer).ConsumePartition
	/go/pkg/mod/github.com/elastic/[email protected]/consumer.go:162 +0x345

goroutine 593 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x32a7d0?, {0x13e55f0, 0xc0000be460})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 663 [runnable]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x0?, {0x13e55f0, 0xc0008866e0})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

goroutine 633 [chan receive, 9 minutes]:
github.com/Shopify/sarama.(*Broker).responseReceiver(0xc000836380)
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:892 +0x86
github.com/Shopify/sarama.withRecover(0x0?)
	/go/pkg/mod/github.com/elastic/[email protected]/utils.go:43 +0x3e
created by github.com/Shopify/sarama.(*Broker).Open.func1
	/go/pkg/mod/github.com/elastic/[email protected]/broker.go:212 +0xb2b

goroutine 694 [select, 9 minutes]:
github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic.func3(0x7079742d?, {0x13e55f0, 0xc0001fa320})
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:491 +0xd0
created by github.com/elastic/beats/v7/libbeat/outputs/kafka.testReadFromKafkaTopic
	/go/src/github.com/elastic/beats/libbeat/outputs/kafka/kafka_integration_test.go:489 +0x1a7

The text was updated successfully, but these errors were encountered:

elasticmachine · 2022-03-22T14:33:21Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

faec · 2022-03-22T18:12:40Z

The failing test is the SASL/SCRAM support, and produces the infamous "client has run out of available brokers to talk to" message. I'm surprised to see that message, since most such errors after #30776 was merged (8 days ago) should give more detailed failure information -- I wonder if the source branch for the update was prior to that? If so, syncing again should give a more detailed message.

In any case, it looks like a SASL/SCRAM connection failure, which could suggest a change in TLS behavior / verification in 1.18.

belimawr · 2022-03-23T11:25:03Z

I didn't look the failing tests in details, but Go 1.18 release notes mention a small change in TLS:

TLS 1.0 and 1.1 disabled by default client-side
If Config.MinVersion is not set, it now defaults to TLS 1.2 for client connections. Any safely up-to-date server is expected to support TLS 1.2, and browsers have required it since 2020. TLS 1.0 and 1.1 are still supported by setting Config.MinVersion to VersionTLS10. The server-side default is unchanged at TLS 1.0.

The default can be temporarily reverted to TLS 1.0 by setting the GODEBUG=tls10default=1 environment variable. This option will be removed in Go 1.19.

I'm also already running Go 1.18 on my machine and had some python tests related to TLS failing, I didn't dig into the issue because CI was green. But I thing it might be related.

faec · 2022-03-23T15:38:38Z

I think the Kafka test container uses TLS 1.2, but there may be other issues e.g. it also uses the deprecated CN=hostname in its certs, if 1.18 started enforcing that deprecation then that would do it, and instances of that might turn up in the python tests too

andrewkroh added bug libbeat Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Mar 22, 2022

andrewkroh mentioned this issue Mar 22, 2022

[automation] Update go release version 1.18 #30922

Closed

andrewkroh mentioned this issue Apr 14, 2022

[automation] Update go release version 1.18.1 #31284

Closed

cmacknz mentioned this issue Jul 12, 2022

Support Go 1.18 #32328

Closed

5 tasks

This was referenced Jul 26, 2022

Upgrade golangci-lint and fix new lint warnings with Go 1.18 elastic/elastic-agent-libs#70

Merged

[automation] Update go release version 1.18.4 #32493

Closed

cmacknz mentioned this issue Aug 15, 2022

[automation] Update go release version 1.18.5 #32574

Merged

cmacknz closed this as completed in #32574 Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] TestKafkaPublish fails under Go 1.8 #30954

[libbeat] TestKafkaPublish fails under Go 1.8 #30954

andrewkroh commented Mar 22, 2022

elasticmachine commented Mar 22, 2022

faec commented Mar 22, 2022

belimawr commented Mar 23, 2022

faec commented Mar 23, 2022

[libbeat] TestKafkaPublish fails under Go 1.8 #30954

[libbeat] TestKafkaPublish fails under Go 1.8 #30954

Comments

andrewkroh commented Mar 22, 2022

elasticmachine commented Mar 22, 2022

faec commented Mar 22, 2022

belimawr commented Mar 23, 2022

faec commented Mar 23, 2022