Feature: Exactly-once semantics

[Psykepro]

Is the exactly-once semantics added to the road map of the library? Would be very nice to have it. As it is the most wanted semantics in the stream processing realm.

[frairon]

Hi Nikolay,
nope currently we don't have planned anything in this regard. If there's a bigger need for that however, we could at least check out the effort needed to add it to goka.

[Psykepro]

We definitely need it. It is a must for some applications where are some critical parts that need to ensure exactly-once delivery. And now it is easier than ever to implement:
https://www.confluent.io/blog/enabling-exactly-once-kafka-streams/

We are using a Python faust which is a Stream Processing library with Exactly-Once Semantics available. And we want to switch to GoLang and the only thing which is stopping us is that there is no Streams Processing library with exactly-once semantics in GoLang.

[frairon]

Absolutely, I understand the necessity of this. It's simply a matter of time and ressources, so we have to evaluate how easy/fast it is to add this to the framework and if it's worth the effort. Let me check in the next days (or feel free to give it a look too), but I can't promise anything.

[twmb]

As a little bit of self promotion, my franz-go client has EOS support, if the problem for supporting EOS in goka is due to sarama's lack of EOS support.

[frairon]

It looks like sarama simply does not support transactions, therefore no EOS, and apparently no one is working on it.

Replacing the underlying library with something else like franz-go sounds like quite a major change, since goka is quite tightly coupled to sarama. Also I'd really like to give it a try, so thanks for the hint @twmb :)
But for now, I'd say we simply cannot add support for EOS to goka in near/mid future with reasonable resources.

[tinkalgogoi]

Currently, Sarama added transactional API from version v1.37.2, Here is a good article showing the implementation of transactions in producer/consumer - https://www.andreamedda.com/posts/play-with-sarama-transactional-api