Sarama to Kafka communication failure if ssl enabled in kafka

[AnoopPutta]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Sarama Version: 9a9e66f
Kafka Version: 0.10.1.1
Go Version: 1.7

Configuration

What configuration values are you using for Sarama and Kafka?

t = &tls.Config{
Certificates: []tls.Certificate{cert},
InsecureSkipVerify: true,
}
cert will be populated with this
cert, err := tls.LoadX509KeyPair(*certFile, *keyFile)

Logs

[sarama]2017/03/31 08:48:18 Initializing new client
[sarama]2017/03/31 08:48:18 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/03/31 08:48:18 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/03/31 08:48:18 client/metadata fetching metadata for all topics from broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093
[sarama]2017/03/31 08:48:18 Failed to connect to broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:18 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:18 client/metadata no available broker to send metadata request to
[sarama]2017/03/31 08:48:18 client/brokers resurrecting 1 dead seed brokers
[sarama]2017/03/31 08:48:18 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama]2017/03/31 08:48:19 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/03/31 08:48:19 client/metadata fetching metadata for all topics from broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093
[sarama]2017/03/31 08:48:19 Failed to connect to broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata no available broker to send metadata request to
[sarama]2017/03/31 08:48:19 client/brokers resurrecting 1 dead seed brokers
[sarama]2017/03/31 08:48:19 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama]2017/03/31 08:48:19 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/03/31 08:48:19 client/metadata fetching metadata for all topics from broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093
[sarama]2017/03/31 08:48:19 Failed to connect to broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata no available broker to send metadata request to
[sarama]2017/03/31 08:48:19 client/brokers resurrecting 1 dead seed brokers
[sarama]2017/03/31 08:48:19 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama]2017/03/31 08:48:19 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama]2017/03/31 08:48:19 client/metadata fetching metadata for all topics from broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093
[sarama]2017/03/31 08:48:19 Failed to connect to broker datapipeline-perf-01-us-east-1-kafka-03.sse.com:9093: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata got error from broker while fetching metadata: tls: first record does not look like a TLS handshake
[sarama]2017/03/31 08:48:19 client/metadata no available broker to send metadata request to
[sarama]2017/03/31 08:48:19 client/brokers resurrecting 1 dead seed brokers
[sarama]2017/03/31 08:48:19 Closing Client
ERRO[0000] [ip-10-4-11-11][kafka.go:78 main:sendMessage] Failed to connect Sarama producer:%!(EXTRA *errors.errorString=kafka: client has run out of available brokers to talk to (Is your cluster reachable?))
kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Problem Description

When we enable ssl in kafka, communication is broken between sarama to kafka.
Logs are provided above

[eapache]

got error from broker while fetching metadata: tls: first record does not look like a TLS handshake

I don't really have anything I can add to this message. Are you sure you're connecting to the right port on the broker, and that the broker has TLS enabled? What do the logs from the broker say?

[wvanbergen]

Yup, this looks like a misconfigured broker, or you are connecting to the non-SSL port.

[tsouza]

I am also with this error and I am sure is connecting to the correct port. I can verify this with openssl s_client -connect <kafka_address> -tls1_2 the command returns successfully and <kafka_address> is the same used to configure the client.

[brikis98]

I'm seeing the same issue. I've enabled SSL in Kafka's server.properties:

listeners=SSL://0.0.0.0:9092
ssl.keystore.location=kafka.server.keystore.jks
ssl.keystore.password=XXX
ssl.truststore.location=kafka.server.truststore.jks
ssl.truststore.password=YYY
ssl.key.password=ZZZ
security.inter.broker.protocol=SSL

I can connect to the server just fine using the console consumer:

./bin/kafka-console-consumer.sh --bootstrap-server xx.xx.xx.xx:9092 --from-beginning --topic test --consumer.config client.properties 
foo
bar
baz

Where client.properties contains:

security.protocol=SSL
ssl.truststore.location=kafka.server.truststore.jks
ssl.truststore.password=XXX

And openssl works just fine too:

openssl s_client -connect xx.xx.xx.xx:9092 -tls1
CONNECTED(00000003)
(...)
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----

(...)

I tried to now connect to the exact same Kafka server with sarama:

caCert, err := ioutil.ReadFile("ca-cert.pem")
if err != nil {
  // ...
}

caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)

tlsConfig := &tls.Config{
  RootCAs: caCertPool,
  InsecureSkipVerify: true,
}

config := sarama.NewConfig()

config.Producer.Return.Successes = true
config.Producer.Return.Errors = true

config.Net.TLS.Config = tlsConfig
config.Net.TLS.Enable = true

producer, err := sarama.NewSyncProducer(serverIps, config)

This gives me the error:

client/metadata fetching metadata for all topics from broker xx.xx.xx.xx:9092
Failed to connect to broker xx.xx.xx.xx:9092: tls: first record does not look like a TLS handshake

[brikis98]

After a lot more digging, I found the solution: as explained in #643, you need to set -keyalg RSA when using the keytool -genkey to generate the Key Store.

[eapache]

Thanks for digging into this @brikis98. I've added this problem/solution to the FAQ so hopefully it's a little more visible the next time somebody runs into trouble.

https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl

[programmer4m]

Am facing similar kind off error :

WARN kafka message: client/metadata got error from broker while fetching metadata:%!(EXTRA sarama.PacketDecodingError=kafka: error decoding packet: message of length 1213486160 too large or too small)
any known solution or reason for this ?

[Aulilino]

After a lot more digging, I found the solution: as explained in #643, you need to set -keyalg RSA when using the keytool -genkey to generate the Key Store.

@brikis98 I am so curious about how u solve it. Could u provide more infomation about it. Tks a lot.