Usage Scenarios

[clemensv]

Collapsed the 12 scenarios from the email to 4 and added clarifying discussion

[ultrasaurus]

Suggested changes from today's discussion in Serverless WG break out session

[ultrasaurus]

Suggested wording to clarify:

Events are typically produced related to a context or a producer-chosen classification. For example: a temperature sensor in a room might be context-qualified by room, floor, and building. A sports result might be classified by league and team.

• context-qualified by mount position, room, floor, and building.

-- removing "base" might help

[ultrasaurus]

The producer application could run anywhere, such as on a server or a device.

[ultrasaurus]

Note: the rendering of the event by an intermediary does not change the semantic meaning of the event, specifically that the producer remains the same, even if the event were not originally produced in the CloudEvent format.

[cathyhongzhang]

Sara: If we decide that the producer will still be the the original entity that emits the event, then we need to spell this out clearly. Let's take the following two examples to make sure we are on the same page.

1. In an IoT example, a motion detection event is originally emitted by a motion detector and then sent to the cloud platform via a HTTP API GW, the producer will be the motion detector, not the API GW. Could you confirm this is what you propose?
2. In the same IoT example, if the motion detection event triggers a video/image to be saved into a storage which in turn triggers a motion video/image storage event to the cloud platform, does this mean the semantic meaning of the event has been changed? Is the producer the storage entity or is it still the motion detector? I think we need to clearly define what constitutes "change of the semantic meaning of the event"

[ultrasaurus]

@cathyhongzhang agree with your interpretation. If we added the following text, do you think that would be clear?

For example, an IoT devices capture an image and provides that as the payload of its motion detection event:

1. The devices transmits a stream of events using a proprietary protocol, there exists an API Gateway that transforms these into CloudEvents, the IoT device is the producer, even though the API Gateway created the data in the CloudEvent format because the semantic meaning of the event remains unchanged
2. The devices transmits a stream of events using a proprietary protocol and an intermediary saves the image in an ftp server, which then emits a object.created event. The ftp server is the producer of the object.created event (not the motion detection event).

[inlined]

Agree strongly with the spirit of this thread. I think I agree vehemently with everything said in the PR so far, though the vocab is dense (e.g. reader may not understand what "rendering" means by the time it they reach it in this use case) and it would be great to have some grounding examples (e.g. the IoT stories above).

[deissnerk]

@inlined I like the idea of more examples, too. There is still #53 open. A starter doc could contain exactly the desired examples. I wrote a comment there a few days ago regarding patterns. This gateway that renders events as CloudEvents could be one of them. Mapping the context attributes to a topic hierarchy to propagate events using a message broker could be another one.

[duglin]

I agree that the text in this PR is very helpful, and on one of our calls this week I asked whether this should live in the spec or in a companion doc because its non-normative text. For now, and to keep us moving forward, I think putting this into the spec is fine and we can do follow-on PRs to extract it if we think it would be better to live in another doc (one in which we could really elaborate on things with more examples, and not bloat the spec).

To me the biggest goal of this PR is to get alignment on our thinking first , which I think we're really close on - we can leave word-for-word wordsmith-ing and document(s) organization for later.

[ultrasaurus]

graft onto above paragraphs...

display, archival, analytics, workflow processing, monitoring the condition and/or providing transparency into the operation of a business solution and its foundational building blocks.

Proposed merging of two paragraphs:

Applications consume events for the purposes such as display, archival, analytics, workflow processing, monitoring the condition and/or providing transparency into the operation of a business solution and its foundational building blocks.

[cathyhongzhang]

Propose to replace "applications" with "applications/platforms"

[ultrasaurus]

see above

[ultrasaurus]

numbered list to help discussion

[clemensv]

the second block would have to be 5. and 6. and that seems to throw some markdown tooling off.

[ultrasaurus]

relative to the originating context (which could be clock-time or other sequence number)

[cathyhongzhang]

The consuming application is also interested in

• correlating event instances from multiple event producers and send them to the same workflow instance.

[ultrasaurus]

we could change the point two bullets above ("identifying and selecting the origin context or the producer-assigned classification") to say:

"identifying and selecting the origin context or the producer-assigned classification, potentially correlating multiple events"

[duglin]

Agreed to add this. Cathy is not asking for a new attribute at this time, she just wants to make sure the spec doesn't preclude the usecase

[cathyhongzhang]

Yes, I would like to makes sure "correlating event instances from multiple event producers" is a key interest point of the consuming application and the event data (metadata+payload) should carry "correlation-token" information. As to how to identify this correlation-token in the event data, there could be multiple ways of doing it depending on where/how the event producer put this info. For example, this correlation-token could be part of the origin context or could be a unique string in the event payload or could be an explicitly defined metadata field of the event.

[ultrasaurus]

For example: an HR producer application may choose to use a customer ID in the event (to limit PII exposure), because different consumers may have different access permissions, and may need additional authorization, obtain additional detail about the customer that may not be present in the event or act on the customer record.

[ultrasaurus]

@cathyhongzhang agree with your interpretation. If we added the following text, do you think that would be clear?

For example, an IoT devices capture an image and provides that as the payload of its motion detection event:

1. The devices transmits a stream of events using a proprietary protocol, there exists an API Gateway that transforms these into CloudEvents, the IoT device is the producer, even though the API Gateway created the data in the CloudEvent format because the semantic meaning of the event remains unchanged
2. The devices transmits a stream of events using a proprietary protocol and an intermediary saves the image in an ftp server, which then emits a object.created event. The ftp server is the producer of the object.created event (not the motion detection event).

[cathyhongzhang]

Propose to replace "applications" with "applications/platforms"

[cathyhongzhang]

The consuming application is also interested in

• correlating event instances from multiple event producers and send them to the same workflow instance.

[rachelmyers]

Suggested change: Middleware routes events from producers to consumers, or on to other middleware, without changing the event.

[deissnerk]

without changing the event.

Does that include the context attributes or just the payload?

[cathyhongzhang]

I think when middleware routes the event, it should include both the original context attributes and payload since the consumer has interest in the context attributes too. For example, some context attribute is needed to identify/extract a field in the payload.

[deissnerk]

I was referring to without changing. Do we exclude that there might be context attributes that can be changed, removed or added by the middleware?

[clemensv]

"Without changing" would be contradicted by the details of the section, because transcoding and transforming is doubtlessly a change. The transform description has a clause on it that semantic changes are not allowed, and that includes omissions.

On @deissnerk's point: I'd like to rename the "extensions" section we currently have in the spec to "annotations" and clarify that this section MAY be edited by intermediaries but MUST be forwarded. That would align with the notion that the AMQP TC arrived at, where the "bare message" which holds the content and core properties is immutable, but can be annotated in transit.

[austencollins]

"Without changing" would be contradicted by the details of the section, because transcoding and transforming is doubtlessly a change. The transform description has a clause on it that semantic changes are not allowed, and that includes omissions.

+1

On @deissnerk's point: I'd like to rename the "extensions" section we currently have in the spec to "annotations" and clarify that this section MAY be edited by intermediaries but MUST be forwarded. That would align with the notion that the AMQP TC arrived at, where the "bare message" which holds the content and core properties is immutable, but can be annotated in transit.

@clemensv Some brief context: We chose extensions because it is term OpenAPI uses to describe custom attributes to its specification and we felt having some similarities between CloudEvents and OpenAPI would be helpful. That said, I like your suggestion of having that section be editable. It means that section will take on a new purpose, in addition to merely being a place for custom attributes. I believe having a section containing editable attributes may solve a handful of problems.

[alexdebrie]

Is it true that middleware should never change the content of an event? A few examples I could think of:

• Scrubbing PII from an event before forwarding to downstream consumers;
• Enriching an event, such as hydrating an object on the event. Imagine a producer includes a productId in the event payload, then the middleware hydrates the product object based on that id to include product name, price, inventory, etc.

In both of these situations, I would still consider the original producer the source of the event rather than the middleware, as suggested by this thread.

[lfourie]

@alexdebrie middleware may modify (transcode/transform) the content/payload of the event as mentioned in the paragraphs below. The question is whether middleware can modify the common event metadata attributes?

[alexdebrie]

@lfourie Yea, I'm wondering if it's ever appropriate for middleware to modify the event payload before passing along. Example:

Removing personally identifiable information (PII) from payload

Perhaps my Users service emits a user.created event which includes the social security number of the newly created user. I want to configure my middleware to scrub that attribute before sending to downstream consumers.

From:

{
  "firstName": "Alex",
  "lastName": "DeBrie",
  "dateOfBirth": "01-01-1980",
  "ssn": "555-55-5555",
}

to:

{
  "firstName": "Alex",
  "lastName": "DeBrie",
  "dateOfBirth": "01-01-1980",
}

In my mind, this is still a user.created event that's been emitted from my Users service rather than an entirely new event emitted by the middleware as suggested by @ultrasaurus's object.created example.

[clemensv]

@alexdebrie IMO, if you're removing (or adding) anything, including PII, you're making a semantic change. A scrubbed event is no longer the original but rather originates from whoever does the scrubbing (or enriching); it is a derivative. A very basic test might be whether you can compute the same integrity hash over the payload values (not metadata or bits on wire) before and after transcoding or transform.

ESPECIALLY if you have PII involved, you'll want to have a clear break between the event data before and after the scrub, because you will want to have absolutely no doubt in any audit trails about what kind of data you're dealing with when the GDPR police shows up.

All that does not at all mean that such an intermediary weren't allowed to publish the derived event as to relate to the same context. Your event doesn't originate from the "users" service, but it still relates to exact same user context.

The event will have a different id (to delineate it from the unscrubbed original) and if you want to include provenance details in the event detail data, you might have a trail there that gives the id of the original event and a hash of that original event.

This is actually brilliant example for why "source" isn't the right concept in the current spec, because your scenario calls for an original and a derivative event to be posted related to the exact same context by different actors for different segments of a system.

[ultrasaurus]

additional questions/feedback on #3 and #4

[ultrasaurus]

This seems to overlap with rendering in #1 above

[clemensv]

I don't see the overlap/conflict

[ultrasaurus]

no conflict, just would be more readable if it was only in one place. I would move the bit about "rendering" from producer section to here (since transcoding seems like what you were describing as "rendering" to me, yet the language above speaks to a somewhat different use case)

[ultrasaurus]

what does "subject" mean in this context?

[ultrasaurus]

in today's calls @clemensv clarified that this might be a specific file that is being acted on, which led to the event being generated. I think this needs just a little more explanation.

[cathyhongzhang]

Agree with Sara. I think we need to define/explain what is meant by the subject. We may also need to define/explain classification of events. Classification can be based on different event source type (eg. storage, message notification etc.) or based on different cloud event vendor. What is this classification?

[ultrasaurus]

Is a concern of the event or of the transport from producer to middleware and from middleware to consumer?

[clemensv]

No. Even if we were to decide that Cloud Event events were always self-contained JSON, the event data itself might quite well still be a base64 encoded binary holding Protobuf encoded data and the middleware may want to transcode that into Avro. That's not a transport concern.

[ultrasaurus]

If middleware consumes the event, isn't it a consumer then?

[deissnerk]

@ultrasaurus Good point. To me consuming an event means that some action is triggered by it and that it is not forwarded. Of course some new event can be emitted as part of the action. The text above mentions consumption via a middleware. In that case the middleware provides the means to consume the event but is not the consumer.

[cathyhongzhang]

Agree with both comments here. For example, a serverless platform will consume the event and trigger a function execution and pass filtered event data to the function. Is the serverless platform the middleware or the consumer or both?

[clemensv]

@deissnerk @ultrasaurus via is indeed the keyword here, as @deissnerk explains

[ultrasaurus]

My point is that the producer determines how it will communicate about its events. There seemed to be agreement when we discussed this. I would like the text here to make clear that an operator can connect producer and middleware, not that a producer somehow specifies it's middleware -- I wouldn't prevent that in the specification, but this language seems to imply that we always encode middleware configuration in the producer, which I don't think is your intent.

[clemensv]

The "producer" is a piece of software and as such it is written, configured, and deployed. The operator may be changing configuration. I think those are IT ground rules that need no elaboration.

[ultrasaurus]

nit: user => application developer

[clemensv]

I'm telling an app to specifically show the MSFT, GOOG, and IBM stock ticks for me, it's arguable that the subscription rule has been user supplied. This section is about frameworks that deal with dispatching events and therefore I chose this to be a bit broader than just application developers.

[ultrasaurus]

Would be good to include what it is about the metadata commonality that is interesting to a framework

[clemensv]

"so that similar activities can be handled uniformly."

[ultrasaurus]

In this context, we use the term "application" to be inclusive of custom applications or services or platforms that host applications or services.

[ultrasaurus]

Propose additional text to clarify (based on today's discussion):
In the list below, a specific platform or application could take on multiple roles.

[clemensv]

We chose to eliminate "platform" related to events in this discussion. The only place where I keep "platform" is in the Frameworks section where it refers to "event platform infrastructures" that frameworks aim to provide commonality over. That should be sufficient for framework folks to know what this means.

[ultrasaurus]

From today's conversation, I think adding this clarification to the above point could address question raised by @cathyhongzhang : Filtering does not change the content of the event, rather selects particular events to forward to a consumer.

[clemensv]

None of the activities change the event semantically, and the one activity that touches the event structure is specifically stating that semantics may not be changed. A clarification for filtering specifically seems superfluous to me.

[ultrasaurus]

in today's calls @clemensv clarified that this might be a specific file that is being acted on, which led to the event being generated. I think this needs just a little more explanation.

[ultrasaurus]

Propose additional text to clarify (based on today's discussion):
In the list below, a specific platform or application could take on multiple roles.

[inlined]

Really liking this so far.

[inlined]

Thanks for this preamble; I really like the statement that we're worrying about the complex case, though an in-house implementation is totally in scope as well.

Will continue reviewing; I also wonder about the case where the event source developer and the deployer are different (e.g. OSS or on-prem deployments). This doesn't impact security as much as it impacts how each party influence the event shape.

[inlined]

Agree strongly with the spirit of this thread. I think I agree vehemently with everything said in the PR so far, though the vocab is dense (e.g. reader may not understand what "rendering" means by the time it they reach it in this use case) and it would be great to have some grounding examples (e.g. the IoT stories above).

[inlined]

At risk of being nitty, I didn't understand what the dash in "-modules" and "-instances" modified until I tried to suggest a rewording. This intro joins complex lists; it might be better to break the sentence up so it is more easily parsed. WDYT about:

Applications consume events originating from devices, people, applications (e.g. OSS deployments), application plugins, or instances (e.g. SaaS). These events are consumed for display, archival, analytics, workflow processing, or other handling.

[clemensv]

I'm just using "applications, application-modules, and application-instances" (what the above expands to in common usage) to catch most concepts people may have in mind for where the event may come from. To some "application" is a deployed instance on a device, to some it's any deployment anywhere, some cares about the specific piece of an app that emits and event and I'm just trying make that all the same here.

Being nitty in return, "OSS" refers to a collaboration model when building software ("OSS" doesn't define a qualitative difference between Apache Kafka and Azure Event Hubs once you have deployable bits) and "SaaS" refers to a delivery model for a product category, so neither of them fit as examples in the respective places, IMO.

[inlined]

From the phrasing, it seems that the middleware can only filter for the correct event data layout, not request the correct event data layout. Is this the intended statement? I can stand by either way but don't want to do so on accident.

[clemensv]

We're effectively defining a one-way flow here, so the middleware only gets to operate on what it gets.

[ultrasaurus]

Is the intent here to suggest that there be a single discriminator for this purpose? I think it might move us forward more quickly to not be overly specific at this point. Maybe just change line 133 to say:

To satisfy these needs, middleware will be interested in common metadata that allows:

• classification of events so that consumers can express interest in one or multiple such classes.
• distinguishing the subject of a particular event of that class.

etc.

[clemensv]

this isn't meant to jump to conclusions but to call out what the middleware needs.

[lfourie]

Looks like you have several levels of discriminator for classification and filtering. This first one seems to map to the source-type, eg, s3 storage, second one maps to source-id, eg bucket within s3 storage.

[clemensv]

the first maps to "topic" and the second to "subject" in #95 as further elaborated on in #112

[lfourie]

As discussed in the meeting this appears to map to namespace, eg, the event encoding and data is to be interpreted within the context of a specific namespace value, eg, azure, aws, oracle

[clemensv]

No. The encoding maps to the specification's "content-type" field.

[lfourie]

Is the event emitted by the middleware identical to the received event apart from possible event payload transformation? If not, there is a need to be able to correlate events received by middleware with the events emitted by middleware.

[clemensv]

It's assumed that the middleware doesn't make semantic changes and as I explain here I would be in favor declaring the event metadata immutable except for an annotation section.

[duglin]

I believe that @clemensv has addressed all of the comments on this PR.

I'd like to ask that people review this PR for conceptual correctness in prep for tomorrow's call. Is this more right than wrong? Does it move us forward in the right direction? If so, then please LGTM it. That does not mean we can not make editorial tweaks and additions in follow-on PRs. The goal here is to make incremental progress, not achieve perfection on the first try.

LGTM

[markpeek]

Agreed, the content look really good. LGTM to commit and we could tweak with subsequent PR's.

[austencollins]

LGTM. Nice work @clemensv, @ultrasaurus, @duglin, @cathyhongzhang, @rachelmyers and others! There are some helpful suggestions in the comments. However, the PR in its current state offers enough clarity that I'm in favor of merging it now to help improve our focus, and optimizing it later.

[ultrasaurus]

The usage scenarios described here seem to conflate transport / middleware with event generation in places. I don't believe that is the intent of @clemensv, yet I know there are many members of the WG who are very concerned about transport. I'm not convinced that we are not aligned in the separation of concerns, so committing this PR seems premature.

One idea would be to separate into commit producer/consumer and then do another pass at middleware/frameworks. I think the description of producer moved us a huge step forward on the "what is source?" question.

Thanks @clemensv

[clemensv]

@ultrasaurus

Regarding this comment

Please point to each sentence where you see the text "conflate transport / middleware with event generation", and suggest alternate wording for each.

"Transport" doesn't occur even once in this PR; neither does "protocol" or any synonyms of either.

Middleware appears as a description of an usage scenario family, representing middleware builders. I represent one of those with a fleet of products and engagements in several OSS efforts, and the section reflects our perspective and our requirements on event producers. If you believe there is an omission, please provide an amendment.

There is no mention in the middleware section of it acting as an event producer; the section explicitly scopes out that middleware can change the event semantically such that it would take on into that role inadvertently. The text therefore draws a very explicit line between the producer and middleware roles.

If a software that plays a middleware role were to produce an event, the respective function would fall under the producer role. The section preamble states explicitly "A single application context can always take on multiple roles concurrently"

If you believe there are further roles than builders of Producer, Consumer, Middleware, or supporting Framework software that must be enumerated to further consensus on the core artifact, please contribute a section for review that does not overlap with the present sections.

[shalka]

@duglin my proposed change:

"For example, a weather station transmits a 12 byte event payload indicating weather conditions once every 5 minutes over LoRaWAN. A LoRaWAN gateway might then be used to publish the event to an Internet destination and in the Cloud Events format. The weather station is the producer and the intermediary gateway plays a middleware role (see 3)."

should be

"For example, a weather station transmits a 12-byte, proprietary event payload indicating weather conditions once every 5 minutes over LoRaWAN. A LoRaWAN gateway is then used to publish the event to an Internet destination in the Cloud Events format. The LoRaWAN gateway is the event producer, publishing on behalf of the weather station, and will set event metadata appropriately to reflect the source of the event."

[rachelmyers]

Proposed addition based on conversation during March 23 call: In practice, Middleware can take on role of a Producer when it changes the semantic meaning of an event, a Consumer when it takes action based on an event, or Middleware when it routes events without making semantic changes. cc @duglin

[cathyhongzhang]

Like these additions!

[markfisher]

based on March 23 WG call, I propose adding something like:

"These usage scenarios are by no means exhaustive, and the specification does not aim to be prescriptive about usage."

[rachelmyers]

Addition based on March 23 call: These scenarios are not normative; anyone is free to create a system that mixes these scenarios. These cases establish a common vocabulary of event Producer, Consumer, Middleware, and Framework.

[duglin]

I added some changes based on today's call. I believe this URL will show the diff:
d07a1b9...clemensv:user-stories

@clemensv looked these over before he vanished on his vacation and approved them.

[duglin]

With all of the good work people put into this over the last week (with near daily meetings!), I think it covers the main points people wanted to see. Any additional tweaks can be done via follow-on PRs.

LGTM

[markpeek]

@duglin your edits LGTM to consolidate the input from the 2018-02-23 meeting and the above input from @rachelmyers and @markfisher.

My only suggestion (which I could put into a follow-on PR) would be to move the middleware addition to the top. This would be combine:
In these scenarios, we keep the roles of event producer and event consumer distinct. A single application context can always take on multiple roles concurrently, including being both a producer and a consumer of events.
and
In practice, middleware can take on role of a producer when it changes the semantic meaning of an event, a consumer when it takes action based on an event, or middleware when it routes events without making semantic changes.
and replace at the top with:
In these scenarios, we keep the roles of event producer, event consumer, and middleware distinct. A single application context can always take on multiple roles concurrently, such as being both a producer and a consumer of events. In the case of middleware, it could take on role of a producer when it changes the semantic meaning of an event, a consumer when it takes action based on an event, or as a more typical middleware when it routes events without making semantic changes.

[duglin]

@markpeek seems like a reasonable follow-on PR to me.

[duglin]

Approved on 3/29 call.