Memory leak

[reallovelei]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: 1.14.0
Kafka Version: 2.11-0.11.0.1
Go Version: go version go1.9.2 linux/amd64

Configuration

What configuration values are you using for Sarama and Kafka?

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

Problem Description

heap profile: 0: 0 [38: 2051184] @ heap/1048576
0: 0 [0: 0] @ 0x850310 0x83e044 0x83cef0 0x84d2bf 0x84babd 0x871a3a 0x86d763 0x45e681

0x85030f hardess/vendor/github.com/Shopify/sarama.versionedDecode+0x4f /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/encoder_decoder.go:78

0x83e043 hardess/vendor/github.com/Shopify/sarama.(*Broker).sendAndReceive+0x243 /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/broker.go:428

0x83ceef hardess/vendor/github.com/Shopify/sarama.(*Broker).Fetch+0x6f /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/broker.go:259

0x84d2be hardess/vendor/github.com/Shopify/sarama.(*brokerConsumer).fetchNewMessages+0x4ae /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/consumer.go:811

0x84babc hardess/vendor/github.com/Shopify/sarama.(*brokerConsumer).subscriptionConsumer+0x13c /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/consumer.go:697

0x871a39 hardess/vendor/github.com/Shopify/sarama.(*brokerConsumer).(hardess/vendor/github.com/Shopify/sarama.subscriptionConsumer)-fm+0x29 /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/consumer.go:638

0x86d762 hardess/vendor/github.com/Shopify/sarama.withRecover+0x42 /usr/local/src/go/src/hardess/vendor/github.com/Shopify/sarama/utils.go:43

[eapache]

This seems a lot like #1046?

The "problem description" is just a stack trace; is that where you think the leak is occurring? do you have a more complete profile you can share showing it actually building over time? That location will definitely allocate a lot of memory so it may not be an actual leak.

As a general rule, Go has garbage collection so it's very difficult to leak memory.

[eapache]

I've answered similar tickets a lot, so I put a more detailed answer in the FAQ now: https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-is-the-consumer-leaking-memory-andor-goroutines

[chandradeepak]

@reallovelei
i have faced similar issue too and this is how we solved it

see if that makes any difference for you
By default sarama stores 256 messages for each partition in partition consumer . So in your case it if you have 4 partitions and each message is is close to 1mb them 1GB is allocated in memory .
And if you have multiple consumer working then it would be very high.
so reduce the Config.ChannelBufferSize( sarama config) to less like 10 . so that way each partition is consuming only 10mb and 4 partitions is 40mb .
below is the property i am talking about
https://github.com/Shopify/sarama/blob/master/config.go#L262