Domain Event Pattern - can and should it be done?

[rsoesemann]

In Domain Driven Design a single pattern is drawing a lot of attention to itself - Domain Events and Event Queues.

(slide via http://de.slideshare.net/andysal/never-mind-the-bollocks-heres-the-domain-driven-design)

Domain Events models non-technical events as their own entity and allow code that needs to react to those event to subscribe to such events. A kind of Pub/Sub, Observer pattern to decouple code.

I'm a strong believer in applying Enterprise Pattern to the Apex space especially if they are adapted to the special characteristics of the Force.com platform. I see a great use case in using libraries like the awesome FinancialForce Apex Common.

So my questions to the expert out there are:

1. Does it make sense to port something like this to the Salesforce.com world?
2. Have you done it and with which experience?
3. How would one implement the Event queue? Using Async Queuable jobs? Streaming API?

[afawcett]

• Are Apex Triggers kind of events? One could argue Apex Triggers are a kind of native event, by writing one your subscribing to CRUD events (before insert, after insert etc) of your domain object. These events are of course more database / persistence layer focused and the event parameters are restricted to the record data and not for example any kind of state / context from the controlling service (if there is one). Scope wise such subscriptions exist org wide.
• Roll your own? Apex interfaces can be used to implement an eventing system and model an event either as loosely bound generic interface or a domain object specific one. Subscription could occur at the Apex request level (parameters on the service layer?) or via config through Custom Settings / Custom Metadata at the org level. Providing this sort of facility as part of a package level thing might provide more service orientated hooks for integrations that are more reliable in terms of when they occur. Than leaving developers to go via Apex Triggers that are subject to firing multiple times (due to chunking and/or process builder customisations) within a single service operation. Implementors of the event interface could be sync (in process) or async (using an adapter pattern to proxy the handler into async) without the code firing the event (calling implementors of the interface) necessarily knowing as such. In order to make async event firing work in a service caller agnostic way, one would have to consider that enqueuing async work from an async context has platform restrictions so this would need to be considered if you wanted maintain the truly caller context agnostic design of the service layer, perhaps simply by lettering the subscription control this.

@up2go-rsoesemann Do you have any use cases / scenarios to share that prompted this? As always good to throw some of this at some scenarios and see what sticks... 👍

[rsoesemann]

Great to see you step in here and stimulate the discussion, @afawcett

Well, let me share a bit of context here. In some of our products I have seen us go from synchronous code to react to events (a.k.a. triggers) to asynchronous just because we hit limits.

The complexity and problems (how to keep it ACID, how to handle errors, provide UI,..), this introduced was way to high to cope for us as a smaller ISV who wants to stay focused on domain problems. We "begged" for infrastructure components coming from Salesforce to cope with this...they didn't come!

In the most cases we changed the datamodel and reduced complexity so we could go back to just in time synchronous code. How long will that work I ask myself??

Then I saw the "Enterprise Service Bus" package from BigAssForce and learned from Matt Bingham that this is built solely on native parts of the platform and it looks awesome.

I also once tried to built something similiar on my own to circumvent Batch restrictions. The SObjectWorkQueue is still available on GitHub, but the native FlexQueue and Queuable basically made it obsolet.

So to answer your use case / objective question: I actually do less care about Eventing but more about decoupling triggering vs. reacting code and having a solid and reliable infrastructure to wire them without glue code. I want to leave my events the same and just say I want to react in an async fashion without having to care about all the drawbacks of async myself.

[mattandneil]

We've seen many implementation approaches with potential. And some kooky ones too including:

• an Outbound Messaging workflow that loops back into the platform,
• using an Inbound Email Service which sends outbound emails to itself,
• chaining Schedulable jobs to achieve an ever-widening cascade of processing,

But one of the keys to a reliable framework, is that it the services handling the domain events must not leak into the framework. For example if the event originates from an API, the handler cannot make HttpRequests back into salesforce, "Callout Loop Not Allowed". Or if the service container updates any SObject, your event handler cannot touch setup objects or sharing rules, "mixed DML".

You really want an intermediary actor that instantiates a new execution context for each logical service or event handler:

	➡️

So how does one isolate the eventing framework from unexpected traumas that befall the services?

What do we risk by verging on long running processes which are discouraged by salesforce?

(etc)

Ultimately it boils down to a backbone that separates the responsibilities of emitting the domain event from the asynchronous handler, while simultaneously preserving the maximum possible spectrum of available Apex features.

[lifewithryan]

Two things before I chime in:
#1 - I'm so not in your league here but I'm going to mention some things anyway
#2 - typing from my phone while sipping coffee, will do my best.

I come from the Django world and one thing they built in is a signals framework along with the ability for devs to create their own signals. One would listen for these signals and upon catching one, could kick of any number of things. Mostly these were very basic. However where it got powerful was in the apps that other people created. Many people created apps that would fire their own signals so that other people could write apps that intercepted these signals as well, or maybe just their code. One such example was an ecommerce module someone wrote called Satchmo. That thing fired numerous events, item added to cart, etc.
I see the power in this in that ISVs could be firing events for anything their app does and devs who implement it can key custom functionality off those events. Triggers only listen to DB related events but other things could be going on brides DB events...but I digress.

I look at this like lightning events in a way. If everyone created components and custom events the possibilities become endless when getting components to work together. I think we are kind of heading there already with components and lightning events.

Now I may be way off my interpretation of what you folks are discussing and if so, I apologize, but if I'm even halfway close -- and it was kept simple (don't Java it to death) -- I'd like to see something like this as part of the core platform :)

[rsoesemann]

@lifewithryan thanks for "chiming" in. The Signals framework you are mentioning comes to close to what I would like to have a decoupling infrastructure component in Salesforce.com.

Lightning events allow the same decoupling of modules for UI components. But as I understand Lightning events are only on the client side and can not be used for Backend (Apex) signaling.

But I like your comparison. Although UI "components" existed before in Visualforce, without having events one could emit without caring who is going to consume them, they did not help built independent and complex UI.

The same is true for backend, Apex code which will stay tightly coupled without Async. Resorting to Async alone might help decoupling. But without an infrastructure layer it then becomes unreliable.

IMHO Salesforce is not going to solve this soon. So we have to built it on our own. Why not do this as part of a widely accepted Open Source library.

[mattandneil]

P.S. have you considered Durable Generic Streaming for your cross-component communications layer?

[rsoesemann]

@bigassforce Neil, I didn't mean to make the point "we should built an open source alternative of your product. That would be extra-cynic, as I am currently evaluating your product for my employer.

This discussion here is to about whether one should add some Enterprise Patterns around Events to this library. I mentioned you and your products here to invite you to share some general thoughts. Not to ask you to give away your IP.

I have seen demos of your package and there is a ton more things you provide. A great UI, a configurable Service Bus, a Webhooks implementation and a ton more. Nothing anyone would love to build and donate as Open Source.

I really could imagine that your OASIS spec getting widespread attention and quasi standard character by being part of the fflib library.... Don't you?!

[lifewithryan]

@Up2Go (for short): re-lightning events

You are correct in that lightning events are only handled on the front-end, but their handlers can always kick off backend work. That said, I was using lightning outlet as an example. I love the idea of event driven functionality particularly when integrating with other third party apps, etc.

[ImJohnMDaniel]

@up2go-rsoesemann @bigassforce @lifewithryan @afawcett

I am just curious. Has anything resulted from this discussion? Is anyone moving forward with anything mentioned here? I am very interested in where this conversation might go. Cheers!

[mattandneil]

this is a common requirement for developers out there.

[rsoesemann]

@ImJohnMDaniel @bigassforce Great to see some life in here. I agree that a open message spec would be a logical first step for an ecosystem or vendor solutions (like your great Services package) and potential Open Source frameworks for Domain Events.

For all that joined the discussion we should discuss

Common use cases

1. Consistently decouple (complexity, limits) triggering code from the follow-up code
2. Have consist and reliable error handling in cases of async failures
3. Separate Domain code from infrastructure
4. Allow communication between base and extension packages

Implementation concepts

1. Native Apex constructs like Batch and Schedulable
2. Durable Generic Streaming as @bigassforce recommended

General Goalsetting

1. Is it desirable to have such infrastructure code inside Apex Commons or a pattern library like this?
2. Is it doable? Are there enough commiters out there that would contribute?

[MayTheSForceBeWithYou]

Yes please! Would love to test this out.

On Mar 21, 2016, at 4:24 AM, Matt and Neil [email protected] wrote:

@ImJohnMDaniel

Remember our sit-down beer and chips last Dreamforce with Nathan? That conversation was really about the same thing: binding disparate event handlers without a package dependency. Rob also nudged us (thanks again @up2go-rsoesemann) confirming this is a common requirement for developers out there.

Thankfully this problem is solved. We are imminently pushing the missing piece into the OASIS open specification for comment and consumption. Our hope is that any new requirements are contributed back in the hope of inter-operability across ISVs and not just within a single vendor.

Would you guys (incl @MayTheSForceBeWithYou, @lifewithryan, @afawcett) like a personal heads-up before RFC?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[afawcett]

So to answer your use case / objective question: I actually do less care about Eventing but more about decoupling triggering vs. reacting code and having a solid and reliable infrastructure to wire them without glue code. I want to leave my events the same and just say I want to react in an async fashion without having to care about all the drawbacks of async myself.

Thanks for the confirmation @up2go-rsoesemann, in that case, while i do think this is a great discussion, i don't see the scope of the library in this repo going beyond a set of base classes that help developers drive out separation of concerns at a code level. How that code is invoked at a system level i would say is the concern of the existing platform features and as needed additional infrastructures such as the one you mention from @bigassforce or if there is support for it the creation of something in the community.

Happy for sure to keep this discussion going, and love the content and ideas thus far. I would however like to recommend to avoid confusion over expectations on this library, that we continue the conversation in another collaborative place and link back to this perhaps.

Whats everyones thoughts on where to take this conversation to next? 👍

[afawcett]

Great discussion everyone, time to close this one i think though. Cheers! 👍