KOP upgrade Problem

[hzluyang]

my current pulsar version is 2.9.2 kafka-protocal pulsar-protocol-handler-kafka-2.9.2.13.nar
So I plan to upgrade the pulsar cluster from bookeeper to broker from version 2.9.2 to 2.9.3

When i upgrade broker with kafka-protocal pulsar-protocol-handler-kafka-2.9.3.13.nar But It went wrong

Got error from broker:
2022-10-13T11:45:43,891+0800 [pulsar-io-4-93] ERROR io.streamnative.pulsar.handlers.kop.KafkaRequestHandler - Caught error in handler, closing channel
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 104857600: 4294246401 - discarded

Got error from kafkaclient 2.0.0:
F1013 11:04:08.548171 153179 kafka_exporter.go:865] Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

so i change nar from pulsar-protocol-handler-kafka-2.9.3.13.nar to pulsar-protocol-handler-kafka-2.9.3.1.nar

it works well
So what's happend between 2.9.3.13.nar to 2.9.3.1.nar

[Demogorgon314]

@hzluyang Do you have a full broker log here? Through the TooLongFrameException cannot see the root cause.

[hzluyang]

Some Error logbelow

seems like kafka_exporter cannot LIST_OFFSETS

2022-10-14T20:19:20,121+0800 [kop-event-thread] ERROR io.streamnative.pulsar.handlers.kop.KopEventManager - Error processing event null
2022-10-14T20:22:45,874+0800 [ForkJoinPool.commonPool-worker-36] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/__change_events] Failed to create subscription: reader-b5966d0bb4
2022-10-14T20:23:39,287+0800 [send-response-OrderedScheduler-1-0] ERROR io.streamnative.pulsar.handlers.kop.KafkaCommandDecoder - [[id: 0x59d22776, L:/10.198.41.211:9092 - R:/10.198.41.213:14336]] request RequestHeader(apiKey=LIST_OFFSETS, apiVersion=1, clientId=kafka_exporter, correlationId=23) is not completed for 30663819489 ns (> 30000 ms)

I find out if I upgrade protocals from 2.9.3.1.nar to 2.9.3.13.nar kafka_exporter cannot get metrics
https://github.com/danielqsj/kafka_exporter

[hzluyang]

@hzluyang Do you have a full broker log here? Through the TooLongFrameException cannot see the root cause.

However, some non-official Kafka clients might not work for kafka format. For example, old Golang Sarama client didn't assign relative offsets in compressed message sets before IBM/sarama#1002. In this case, the broker has to assign relative offsets and then do recompression. Since this behavior leads to some performance loss, KoP adds the mixed_kafka format to perform the conversion. The mixed_kafka format should be chosen when you have such an old Kafka client. Like kafka format, in this case, Pulsar consumers still cannot consume messages from Kafka producers.

maybe it can fix by set mixed_kafka I will try

[hzluyang]

@hzluyang Do you have a full broker log here? Through the TooLongFrameException cannot see the root cause.
@Demogorgon314
it turns out like 2.9.3.13 may cause kafka client Metaupdate failed
Timed out wait for a node assignment call: fetchMetadata

[Hongten]

@hzluyang Do you have a full broker log here? Through the TooLongFrameException cannot see the root cause.

However, some non-official Kafka clients might not work for kafka format. For example, old Golang Sarama client didn't assign relative offsets in compressed message sets before Shopify/sarama#1002. In this case, the broker has to assign relative offsets and then do recompression. Since this behavior leads to some performance loss, KoP adds the mixed_kafka format to perform the conversion. The mixed_kafka format should be chosen when you have such an old Kafka client. Like kafka format, in this case, Pulsar consumers still cannot consume messages from Kafka producers.

maybe it can fix by set mixed_kafka I will try

hi @hzluyang , what's the result if you set mixed_kafka? Could you share your test result?
thanks.