Event bridges cannot tell if structured JSON data is base64 encoded.

[alanconway]

I am implementing a bridge between different event transport bindings and formats:
https://godoc.org/github.com/alanconway/scratch/lightning
I don't know how to correctly determine if a JSON structured data attribute is base64 encoded or not.

Spec says: "If the contenttype media type is known to contain text, the data attribute value is not further interpreted and treated as a text string. Otherwise, it is decoded and treated as a binary value."

I'm not aware of a reliable way to determine that a MIME type is "known to contain text". RFC1341 provides an explicit Content-Transfer-Encoding when data is encoded, it does not expect mail clients to "just know". My nasty hack is to use this regexp to detect "non-text" media-types, clearly not reliable or future-proof:
/(^$)|(^text$)|(^text/)|(^application.*/+-./+-.)/

I think the JSON format should have a "contenttransferencoding" attribute (rfc1341 compliant) - as should future formats that can't handle raw binary data. It doesn't have to be part of the abstract event, but it would reduce re-encoding if it was: base64 encoded data could flow unmodified through different binary and structured event bindings and only be decoded when the body is actually required.

This is related to #261 but I think deserves separate treatment since it affects the core use case of events - accurately forwarding event data - even if you don't care about corner cases like binary values in Maps.

[duglin]

I'm kind of surprised that there's no way to encode this information into the Content-Type header.
@clemensv any thoughts on this?

[alanconway]

On Thu, Jan 24, 2019 at 8:24 AM Doug Davis ***@***.***> wrote: I'm kind of surprised that there's no way to encode this information into the Content-Type header. @clemensv <https://github.com/clemensv> any thoughts on this?

The MIME RFC is pretty clear on that. Base64 is a Content-Transfer-Encoding which is orthogonal to a MIME type. Content-Transfer-Encoding is an extra layer of encoding for transports with limitations on binary data (e.g. JSON or email) - it is a feature of how you're sending the data, not the type that the data represents. Content-Transfer-Encoding is also used for things like adding compression, which again is not a feature of the type, but of how you've chosen to transfer it. —

…

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#371 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XmM65JFOPDMrrZwYVJihOZ1Lc_oNks5vGbQkgaJpZM4aIPS1> .

[clemensv]

As much as it pains me - @alanconway is right.

Since this is a specific JSON structured format problem, I wonder whether we can add an attribute that only applies to the structured JSON encoding, preferably inside the event format spec, and that attribute MUST be set when the transfer encoding is anything but plain text or an embedded JSON graph.

[alanconway]

On Tue, Jan 29, 2019 at 7:49 AM Clemens Vasters ***@***.***> wrote: As much as it pains me - @alanconway <https://github.com/alanconway> is right. Since this is a specific JSON structured format problem, I wonder whether we can add an attribute that only applies to the structured JSON encoding, preferably inside the event format spec, and that attribute MUST be set when the transfer encoding is anything but plain text or an embedded JSON graph.

I think that works. I also thought about a general cloud-event encoding attribute. The benefit is that bridges transforming JSON-structured events to/from binary form could blindly copy the payload and propagate the encoding attribute. The drawback is that producers and consumers of *all* bindings and formats would then need to understand base64. On balance it feels like sacrificing performance with JSON events is better than imposing base64 on everybody. On a related note: HTTP transfer-encoding/content-encoding allows for compression. There's no problem for an event bridge since its clearly marked, and any HTTP client knows how to deal with it. However I wonder if we need to allow for payload compression of cloud events? —

…

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#371 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XrSAnaop-9HTaS-zW2jBDnI2T9nwks5vIENugaJpZM4aIPS1> .

[duglin]

@clemensv were you going to PR this one?

[duglin]

@clemensv any chance of getting a PR for this one in time for this week's call?

[n3wscott]

I just bumped into this one too. I want to base64 encode my payload but I can not.

[n3wscott]

This is fixed now!

[duglin]

I believe we can close this now that #387 is merged. @alanconway please speak up if I'm wrong and we need to reopen it.