Synchronizing topic consumption of multiple business events originating from one event source

[thake]

First of all, thank you for your inspiring content about your take on event-driven architecture principles and the derived kafka guidelines. A good piece to read 👍

When I read

galapagos/kafka_guidelines.md

Lines 30 to 31 in c288278

	An Event Topic represents exactly **one** Business Event type. **Every** instance of this event type must be represented
	by exactly **one** message on this topic.

I wondered what the reasoning behind this decision was. If an event source produces several business events where the order is important, this decision may lead to ordering problems for consumers as each event needs to have its own topic (see https://www.confluent.io/blog/put-several-event-types-kafka-topic/).

Don't get me wrong, I do not want to criticize the decision. I'm just curious.

[albrechtflo-hg]

Hi @thake ,

Thank you for the positive feedback and the very good question!

We also had a lot of discussion about this limitation. But having only one Business Event Type on each topic had more benefits than drawbacks for us:

• It is easier to find desired Business Events, as the Topic name already fully specifies what to expect in it
• As we enforce publishing a JSON Schema for each event's payload, this schema is much simpler to create for one Business Event type than for several (and it would really complicate things too far to have multiple JSON schemas per topic, depending on the Event Type).
• As we synchronize the Topics with our Architecture Management System, we have a full transparency there what types of events flow between which applications, only by looking at the name of the interfaces (topics are mapped to interfaces there).

For your mentioned case, there are typically two patterns which can be used to resolve the issue:

• Generalize the Event Type, if appropriate (use this pattern carefully). If one producer generates "StockEmptied" and "StockRefilled" events, they could be combined to "StockChanged" events. But caution: The principles clearly state that a topic must be "complete", so there may be no other "stock changing" events somewhere else. So it is often hard to find the "correct" generalization for a given producer scenario.
• Specify the order by a "business specific" order criterion. For example, include an "occurred" timestamp in the message (the CloudEvents format mentioned in the Kafka Guidelines already defines such a timestamp), which refers to the original "when" did the "real" event happen. If a timestamp is not enough or not available, include some other kind of ordering attribute, e.g. a sequence number, which can be used by consumers to "sort" received events.
  Of course, this introduces another issue: You lose real "streaming" ability, as the consumer then has to "detect" that a previous event from another topic is potentially missing and has to "wait" for it. For many business applications, this may be okay (e.g. new Kafka messages only lead to database updates, and data is retrieved later for further processing or user display), but if not, the first pattern may fit better.

If, for a given company, the ordering of different (but related) Business Event Types is that critical for the majority of use cases (and the limitation of the second pattern is not acceptable), I think that would justify to remove this restriction in their adaption of EDA. Technically, Galapagos does not impose this restriction (other than only one JSON schema can be active for a topic at a time), so it would not reduce its usability in that case.

[thake]

@albrechtflo-hg thank you very much for taking your time to answer! The answer really helped me understand the reasoning. The resources you put online really helped define a sharper view on some aspects of EDA.