Developing an MQTT to Apache Kafka bridge

[ppatierno]

A really common use case we have been seeing is about enabling an IoT scenario with MQTT based devices and using an Apache Kafka cluster as the events and storage platform running on Kubernetes via Strimzi.
In order to do that, there is the need to map the MQTT protocol to the custom Apache Kafka one and bridge from one to the other.
Quite often it's mostly about producing with MQTT devices but then consuming with applications on the other side. It is more about "telemetry" pattern than "command" to devices.
We should design a good mapping between MQTT and custom Apache Kafka protocols and then developing a pure Netty based MQTT server component (not a full MQTT broker) able to accept MQTT client connections and handling the corresponding communication based on the MQTT 3.1.1 specification.
Finally, developing the Kafka producer part to get messages from MQTT clients and sending them to an Apache Kafka cluster.

[9avneet]

@ppatierno ,I know Java and studied basics about MQTT (just after reading the
Issue,so can you please guide me how can I started with this project because I
Little new to open source.so it would be very helpful to me if ,you can guide please.
Thanks.

[ppatierno]

Hi, this is part of the GSoC 2023 which we submitted as part of the CNCF organization. If the organization is approved and the idea has a slot, the students can start to send proposal for this idea to the GSoC. It's not something we can start right now. And just one student will be chosen based on his proposal to the CNCF.

[9avneet]

Hi, this is part of the GSoC 2023 which we submitted as part of the CNCF organization. If the organization is approved and the idea has a slot, the students can start to send proposal for this idea to the GSoC. It's not something we can start right now. And just one student will be chosen based on his proposal to the CNCF.

Ok thanks .

[9avneet]

@ppatierno,Hi as I noticed that @Nate- double -u mentioned that,'all ideas are eligible to participate and you can start contribute ' so can you suggest some path by which I become able to proceed with this project as early as possible, because I am new to open source community,so your little guidence help me alot ,and I will able to contribute.
Thanks.

[AnshulAngaria]

Hi @ppatierno , I am a software developer at Avalara currenty working on 'Edge for Avatax' where we provide highly distributed, Iot based, kubernetes based, tax compliance solutions. We extensively use Kafka and MQTT for our Iot based solution.
I am looking to start contributing in open source and would love to contribute to this project.

[9avneet]

@AnshulAngaria ,can you tell me have you good understanding of Java or not?

[9avneet]

@ppatierno ,as I am new to open source,but I know Java,but I found difficult that how to start with this project.so ,can you pls help me in guiding that how can I start this project.

[Sajiyah-Salat]

I have done my dsa in java and I have good hold in java as well. no idea of mqtt but I am active learner so it would not be a issue. my two prs have recently merged in kubernetes. So I am looking forward to take this issue. but I think its reserved for gsoc so it should have gsoc label.

[ShivangShandilya]

@ppatierno I'm pretty intrigued with this project, can you suggest some resources to get started with??

[ppatierno]

Starting from today until April 4th, contributors will be able to send their proposals for working on the #GSoC #CNCF Strimzi project around MQTT and Apache Kafka. Looking forward to see your interest and read them! https://github.com/cncf/mentoring/blob/main/summerofcode/2023.md#proof-of-concept-of-an-mqtt-to-apache-kafka-bridge-for-producing-messages

[money8203]

Hey @ppatierno, I would love to work on this project as a part of LFX Mentorship

I've experience with Java and I believe this project will be a great opportunity for me to sharpen my skills further and gain some more hands-on experience. Looking forward to get more information and resources about the issue, to prepare for the mentorship.

[mugishasam123]

Hey @ppatierno, I am very interested in working on this project as a part of the LFX Mentorship program. I have prior experience working with Java and I believe this project would provide an excellent opportunity for me to enhance my skills and gain more practical experience in this field.

I am excited to learn more about the project and would appreciate any information or resources you can provide to help me prepare for the mentorship. Thank you for considering my application. (I already submitted my lfx mentorship proposal for this project)

[antonio-pedro99]

Hi @ppatierno, I am applying for LFX Mentorship program and I found this project very interesting as I have some experience with Java, Apache Kafka and MQTT. In addition, currently I am working with MQTT on my undergraduate project, which is an IoT system that allow users to monitor the air condition in different regions and places. Furthermore, I would like to get more insights of the problem and discuss more about possible solutions.

Looking forward to hearing from you!
cc: @kyguy

[ppatierno]

In order to engage with me and @kyguy I would go with filling and sharing a Google doc with us where you describe your proposal (what should be done and how). Something around the design and the steps. This is how it went for GSoC and we can provide feedback on the doc directly. You have our emails in the project proposal page on LFX mentoring on CNCF repo.

[ppatierno]

Developing this project was accepted as part of the LFX mentoring program.
More details here: https://mentorship.lfx.linuxfoundation.org/project/8d301adf-94d8-4e5d-821d-f904ed15c3f9
@antonio-pedro99 is going to work on it as a mentee with @kyguy and me as mentors.

[antonio-pedro99]

@ppatierno should this issue be closed?

[ppatierno]

We finally have an initial implementation for it in this repo https://github.com/strimzi/strimzi-mqtt-bridge
Going to close the issue.