transactional producer does not work with IAM auth

[ShashidharC]

• aws-msk-iam-sasl-signer-python version:1.0.1
• Python version:3.11
• Operating System: Linux docker
• Method of installation:
• Kafka library name: confluent-kafka
• Kafka library version: 2.4.0
• Provide us a sample code snippet of your producer/consumer

Description

We are using MSK cluster and doing IAM auth. However, when we implement the transactional producer, it fails to connect. It errors out saying.

When I remove the transactional behaviour it works well without any issue.

Producer conf:

AWS POST: https://repost.aws/questions/QUzHXpsvHDQOiLafIUL-VgFQ/kafka-transactions-not-working-while-using-iam-authentication-with-conlfuent-kafka-python-producer-client
Stack overflow: https://stackoverflow.com/questions/78021233/aws-msk-transactions-support

Could you please take a look?

[sankalpbhatia]

Since you mention this works well without transactions, I suspect it could be because of this reason mentioned in https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html

For Kafka versions 2.8.0 and above, the WriteDataIdempotently permission is deprecated (KIP-679). By default,enable.idempotence = true is set. Therefore, for Kafka versions 2.8.0 and above, IAM does not offer the same functionality as Kafka ACLs. It is not possible to WriteDataIdempotently to a topic by only providing WriteData access to that topic. This does not affect the case when WriteData is provided to ALL topics. In that case, WriteDataIdempotently is allowed. This is due to differences in implementation of IAM logic versus how the Kafka ACLs are implemented.

To work around this, we recommend using a policy similar to the sample below:


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:WriteDataIdempotently"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1/TestTopic"
            ]
        }
    ]
}
In this case, WriteData allows writes to TestTopic, while WriteDataIdempotently allows idempotent writes to the cluster. It is important to note that WriteDataIdempotently is a cluster level permission. It cannot be used at the topic level. If WriteDataIdempotently is restricted to the topic level, this policy will not work.

Can you check if modifying permissions this way works for you?

[ShashidharC]

Yes. @sankalpbhatia . We gave WriteDataIdempotently at cluster level. Not topic level. Also we have access to all the topics that we created.