GSSAPI Kerberos Authentication: wrong Token ID

[MikhailMS]

Versions

Sarama	Kafka	Go
1.29.1	Cloudera Kafka	1.17.1

Configuration

Sarama setup

config := sarama.NewConfig()
config.Producer.RequiredAcks.    = sarama.WaitForAll
config.Producer.Retry.Max        = 10
config.Producer.Return.Successes = true

config.Net.SASL.Enable                    = true
config.Net.SASL.Mechanism                 = sarama.SASLTypeGSSAPI
config.Net.SASL.GSSAPI.ServiceName        = "kafka"
config.Net.SASL.GSSAPI.AuthType           = sarama.KRB5_KEYTAB_AUTH
config.Net.SASL.GSSAPI.KeyTabPath         = "./username.keytab"
config.Net.SASL.GSSAPI.KerberosConfigPath = "/etc/krb5.conf"
config.Net.SASL.GSSAPI.Username           = "<USER>"
config.Net.SASL.GSSAPI.Realm              = "<REALM>"
config.Net.SASL.GSSAPI.DisablePAFXFAST    = true

Kafka setup is unknown

Logs

logs: CLICK ME

[sarama] 2021/09/14 14:17:50 Initializing new client
[sarama] 2021/09/14 14:17:50 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2021/09/14 14:17:50 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2021/09/14 14:17:50 client/metadata fetching metadata for all topics from broker <BROKER>:9092
[sarama] 2021/09/14 14:17:51 Error while performing GSSAPI Kerberos Authentication: wrong Token ID. Expected 0504, was 6030
[sarama] 2021/09/14 14:17:51 Closed connection to broker <BROKER>:9092

Problem Description

For some reasons it looks like sarama has difficulties connecting to our team's Kafka using keytab

I've seen similar issue raised #1400 but it was closed with suggested solution that didn't work for me - unfortunately

I've used different options for krb5.conf and keytab file
1.

permitted_enctypes   = rc4-hmac
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
default_etypes       = arcfour-hmac-md5
default_etypes_des   = des-cbc-crc

with keytab encrypted with RC4-HMAC gives an error I outlined above
2.

# These options were suggested in #1366 (which was mentioned in #1400 )
default_tgs_enctypes = aes256-cts-hmac-sha1-96
default_tkt_enctypes = aes256-cts-hmac-sha1-96
permitted_enctypes   = aes256-cts-hmac-sha1-96

with keytab encrypted with RC4-HMAC gives an error

[sarama] Kerberos client error:
[Root cause: Encrypting_Error] KRBMessage_Handling_Error: AS Exchange Error: failed setting AS_REQ PAData for pre-authentication required < Encrypting_Error: error getting key from credentials: matching key not found in keytab. Looking for [<USER>] realm: <REALM> kvno: 0 etype: 18 

which I believe means that there is a mismatch in encryption, so I've tried keytab encrypted with AES256-CTS-HMAC-SHA1-96 and it gives an error

[sarama] client/metadata got error from broker -1 while fetching metadata:
[Root cause: KDC_Error] KDC_Error: TGS Exchange Error: kerberos error response from KDC when requesting for kafka/<BROKER>: KRB Error: (14) KDC_ERR_ETYPE_NOSUPP KDC has no support for encryption type

which I believe means that our Kerberos system is not set to accept AES256-CTS-HMAC-SHA1-96 at TGS step

So I tried to change default_tkt_enctypes = aes256-cts-hmac-sha1-96 to default_tkt_enctypes = rc4-hmac which returned an initial error (about wrong Token ID)

Any suggestions on how to get it working?

[ilyadiff]

I have the same problem in danielqsj/kafka_exporter#339
Has someone already found a solution or a reason?

[wormlocust]

I got it :
config := sarama.NewConfig()
config.Net.SASL.Mechanism = sarama.SASLTypeGSSAPI
config.Net.SASL.GSSAPI.KerberosConfigPath = "D:\code\golang\kerberosgo\main\krb5.conf"
config.Net.SASL.GSSAPI.KeyTabPath = "D:\code\golang\kerberosgo\main\kafka-client.keytab"
config.Net.SASL.GSSAPI.Realm = "TEST.COM"
config.Net.SASL.GSSAPI.ServiceName = "kafka"
config.Net.SASL.GSSAPI.AuthType = sarama.KRB5_KEYTAB_AUTH
config.Net.SASL.Enable = true
config.Net.SASL.GSSAPI.Username = "kafka-client"
config.Net.SASL.GSSAPI.DisablePAFXFAST = false
kerclient, err := sarama.NewKerberosClient(&config.Net.SASL.GSSAPI)
if err != nil {
fmt.Println("newkerberosclient error")
}
err = kerclient.Login()
if err != nil {
fmt.Println(err)
}
fmt.Println("========================")

consumer, err := sarama.NewConsumer([]string{"slave2:9092"}, config)
if err != nil {
	fmt.Println("consumer init error", err)
}

realm is "TEST.COM"
do not need IPv4 if kafka is set SASL_PLAINTEXT://slav2:9092
broker.id=0
listeners=SASL_PLAINTEXT://slave2:9092
advertised.listeners=SASL_PLAINTEXT://slave2:9092
num.network.threads=3
num.io.threads=8

[MikhailMS]

@ilyadiff
There is no solution to that one yet - Sarama uses gokrb5 for Kerberos, and that package has a bug of not handling rc4-hmac (though it states it can handle it), see jcmturner/gokrb5#460

I started to look into it, but feels like it is much work there (never touched Go or Kerberos protocol before) so unless someone else pick the issue, the fastest solution is probably to make your Kerberos uses modern enctypes

@wormlocust - unfortunately your solution didn't work for me, so probably you had some other issue :) but maybe it would help someone else

[MikhailMS]

Update to the previous message

I've started to look into the issue and while I was able to

• add support for RC4-HMAC to gokrb5 in my fork
• update Sarama code to choose correct wrapToken

when connecting to Kafka brokers I am getting connection reset by peer so I believe

• GSSAPI tokens are not properly handled in my fork OR
• Sarama is not handling SaslV0 correctly

So investigation continues

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.