Doctor Kafka generated invalid reassignment

[daywithoutrain]

We adapted to use DoctorKafka in our production system recently. However, we found that the first time after using doctorkafka, a broker host having some issues and resulted in many under replicated partitions. doctorkafka tried to resolve under replicated partitions, but it somehow generated an reassignment including the following entries: (the entries are copied from doctor kafka's log)

 {
            "topic": "mydomain.MyTopic1",
            "partition": 40,
            "replicas": [
                65633,
                65633,
                65633
            ]
        },
{
            "topic": "mydomain.MyTopic2",
            "partition": 2,
            "replicas": [
                65633,
                65633,
                65633
            ]
        },

...
This impacted our production system and delayed the downstream process. We have more than 50 broker hosts and 1 host was having trouble. I assume that the above assignment was generated due to a bug, is that correct? Have you seen this issue before? I'd appreciate if you could look into it ASAP. Thanks!

[yuyang08]

What is the replication factor for your topics? DoctorKafka has a known issue w.r.t. handling topics of replicationFactor = 1. If you set replication factor > 1, DoctorKafka should handle the topic assignment fine.

[daywithoutrain]

Thanks for your reply! We have replication factor of 3. It was never set to 1. Any ideas on what caused it? Thanks very much, - Jessica

…

On May 9, 2018, at 10:34 PM, Yu Yang ***@***.***> wrote: What is the replication factor for your topics? DoctorKafka has a known issue w.r.t. handling topics of replicationFactor = 1. If you set replication factor > 1, DoctorKafka should handle the topic assignment fine. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[yuyang08]

There might be a bug somewhere. Can you share the assignment plan by doctorkafka from $your_doctorkafka_site/servlet/actions, and the related doctorkafka logs?

[daywithoutrain]

I have the reassignment from the log, but it contains 307 assignments(and it's production data). Most of the them were fine and executed properly. But 9 of the assignment are invalid; they contain duplicated broker hosts in one assignment entry: { "topic": "domain1.dataset1", "partition": 0, "replicas": [65633, 65633, 65633] }, .... { "topic": "domain2.dataset2", "partition": 2, "replicas": [65633, 65633, 65633] }, ... { "topic": "domain3.dataset3", "partition": 40, "replicas": [65633, 65633, 65633] }, ... { "topic": "domain4.dataset4", "partition": 325, "replicas": [65633, 65556, 65565] }, ... { "topic": "domain6.datset6", "partition": 4, "replicas": [65552, 65552, 65552] } ... thanks,

…

On Thu, May 10, 2018 at 11:37 PM, Yu Yang ***@***.***> wrote: There might be a bug somewhere. Can you share the assignment plan by doctorkafka from $your_doctorkafka_site/servlet/actions, and the related doctorkafka logs? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGpvED51fBYY4dGT3FlOtIHG1VWDlM2ks5txTGfgaJpZM4T0eoW> .