Cluster events

[davinliuda]

My kafka8.0 cluster has 2 brokers.
192.168.12.130:9092
192.168.12.130:9093

When I called rd_kafka_brokers_add("192.168.12.130:9092,192.168.12.130:9093"), the application run well, and all messages were sent to 9092. Then I killed 192.168.12.130:9092, the application doesn't send message to port 9093, and all message sending were failed, is this intended?

Btw, is that possible to get all the message from broker, including messages available and not available. If yes, can please advise how to config? Thanks!

[edenhill]

rdkafka will only send messages to the leader broker.
Please verify that 9093 becomes leader after you kill 9092 (also that it becomes leader before messages start timing out).

The brokers that you initially give to rdkafka (through rd_kafka_brokers_add() or the "metadata.broker.list" config property) are called the bootstrap brokers, rdkafka will connect to them only to acquire the full list of brokers in the cluster by querying the Metadata.
It will then connect to each broker in the Metadata response and use these new connections for producing messages.

Btw, is that possible to get all the message from broker, including messages available and not available. If yes, can please advise how to config? Thanks!

You mean get the message count per topic+partition?

[davinliuda]

Thanks for your quick response, really appreaciate your help.

My apply scenario is I want to know if the kafka culuster is available or not, only when it is yes, the application will send message, other wise, no messages will be sent.

My solution is adding a callback method, when RD_KAFKA_BROKER_STATE_UP event happens, using the method to inform the applicaiton that the cluster is available;
when RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN event happens. inform the application that the culuster is not available.

But unfortunately, I don't know the protocol and the detailed process logic of kafka, sending message will fail when the broker is not the leader.

Can you provide such callback method, notify the application when the culster is available or not available.

[edenhill]

I think what you will need is a state per topic + partition: does it have a leader broker ,and is that leader broker up.

I'll look in to it.

[edenhill]

See #137