[Schema Registry Avro] Providing a message encoder #18365

deyaaeldeen · 2021-10-26T17:20:03Z

Revamps the schema registry encoder to work on messages instead of buffers based on the recommendation of the Azure messaging architect.

This changes the APIs as follows:

  const buffer: NodeJS.Buffer = await serializer.serialize(value, schema);

becomes

  const message: MessageWithMetadata = await encoder.encodeMessageData(value, schema);

where MessageWithMetadata has a data field as well as a contentType field. The latter's format is avro/binary+<Schema ID>

For derserializing, the change is as follows:

  const deserializedValue = await serializer.deserialize(buffer);

becomes:

  const decodedObject = await encoder.decodeMessageData(message);

See the newly added samples for how to send such messages with Event Hubs. See #19458 for updating Event Hubs to support such messages.

Note that #19458 needs to be merged first to enable the samples in this PR to typecheck.

azure-sdk · 2021-10-26T17:30:14Z

API changes have been detected in @azure/schema-registry-avro. You can review API changes here

sdk/schemaregistry/schema-registry-avro/src/models.ts

…avro-serializer-design

azure-sdk · 2021-12-16T05:29:29Z

API changes have been detected in @azure/schema-registry-avro. You can review API changes here

deyaaeldeen · 2021-12-20T19:27:46Z

sdk/schemaregistry/schema-registry-avro/samples-dev/withEventHubsConsumerClient.ts

+              const decodedEvent = await encoder.decodeMessageData({
+                contentType: event.contentType,
+                data: Uint8Array.from(Object.values(event.body))
+              });


To streamline the experience of decoding received messages, I am thinking of adding native Event Hubs received message type as a supported input type in decodeMessageData. This could also apply to received message types in other messaging clients.

in JS, we can do that without importing by just duplicating the relevant bits of the native type. For instance, we can define the following input type for Event Hubs messages:

interface ReceviedEventHubsMessage { body: any, contentType?: string }

/cc @JoshLove-msft

What about adding some kind of utility method to convert from the different native types to MessageWithMetadata?

But actually I like your idea better since it would be less work for the user.

sdk/schemaregistry/schema-registry-avro/README.md

JoshLove-msft · 2022-01-03T19:41:13Z

sdk/schemaregistry/schema-registry-avro/review/schema-registry-avro.api.md

+// @public
+export class SchemaRegistryAvroEncoder {
+    constructor(client: SchemaRegistry, options?: SchemaRegistryAvroEncoderOptions);
+    decodeMessageData(message: MessageWithMetadata, readerSchema?: string): Promise<unknown>;


probably can just use "schema" as the param name.

Co-authored-by: JoshLove-msft <[email protected]>

xirzec

Objects feels so much better in JS than opaque streams

xirzec · 2022-01-05T22:48:01Z

sdk/schemaregistry/schema-registry-avro/CHANGELOG.md

 ### Breaking Changes
+- The `SchemaRegistryAvroSerializer` class has been renamed to `SchemaRegistryAvroEncoder`
+- The `serialize` method has been renamed to `encodeMessageData` and it now returns a `MessageWithMetadata`
+- The `deserialize` method has been renamed to `encodeMessageData` and it now takes a `MessageWithMetadata` as input


typo

Suggested change

- The `deserialize` method has been renamed to `encodeMessageData` and it now takes a `MessageWithMetadata` as input

- The `deserialize` method has been renamed to `decodeMessageData` and it now takes a `MessageWithMetadata` as input

xirzec · 2022-01-05T22:48:57Z

sdk/schemaregistry/schema-registry-avro/review/schema-registry-avro.api.md

@@ -4,19 +4,23 @@

 ```ts

-/// <reference types="node" />


Fixes #20061 ## Overview Revamps the schema registry encoder to work on messages instead of buffers based on the recommendation of the Azure messaging architect. This changes the APIs as follows: ```ts const buffer: NodeJS.Buffer = await serializer.serialize(value, schema); ``` becomes ```ts const message: MessageWithMetadata = await encoder.encodeMessageData(value, schema); ``` where `MessageWithMetadata` has a `body` field as well as a `contentType` field. The latter's format is `avro/binary+<Schema ID>` For derserializing, the change is as follows: ```ts const deserializedValue = await serializer.deserialize(buffer); ``` becomes: ```ts const decodedObject = await encoder.decodeMessageData(message); ``` ## Improvement upon #15959 This design introduces a new `messageAdapter` option in the encoder constructor to support processing of any message type (e.g. [cloud event](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md)): ```ts const encoder = new SchemaRegistryAvroEncoder(schemaRegistryClient, { groupName, messageAdapter: adapter }); ``` where `adapter` is a message adapter that follows the following contract: ```ts interface MessageAdapter<MessageT> { produceMessage: (messageWithMetadata: MessageWithMetadata) => MessageT; consumeMessage: (message: MessageT) => MessageWithMetadata; } interface MessageWithMetadata { body: Uint8Array; contentType: string; } ``` For convenience, the PR adds a couple of convenience adapter factories for Event Hubs's `EventData` and Event Grid's `SendCloudEventInput<Uint8Array>`. For example, the `createCloudEventAdapter` factory can be called to construct an adapter for the latter as follows: ```ts const adapter = createCloudEventAdapter({ type: "azure.sdk.eventgrid.samples.cloudevent", source: "/azure/sdk/schemaregistry/samples/withEventGrid", }), ``` Note that these adapter factories are exported by their respective messaging package without explicitly implementing the contract and the PR adds new encoder tests that check whether the produced adapters follow the contract. This organization could change in the future if we create a new core place for the contract to be imported from. See the newly added samples for how to send such messages with Event Hubs and Event Grid. Schema Registry commitment tracking: #15959 Tracking issue: #18608 First iteration design: #18365

deyaaeldeen · 2022-01-26T17:56:56Z

Closing in favor of #19842

deyaaeldeen requested review from conniey, ramya-rao-a, yunhaoling, JoshLove-msft and swathipil October 26, 2021 17:20

ghost added the Schema Registry label Oct 26, 2021

[Schema Registry Avro] Providing message encoder

1b10967

deyaaeldeen force-pushed the schemaregistry/new-avro-serializer-design branch from e8fec65 to 1b10967 Compare October 26, 2021 17:22

deyaaeldeen mentioned this pull request Nov 9, 2021

[Schema Registry Avro] GA package TO-DOs #18608

Closed

2 tasks

JoshLove-msft reviewed Nov 9, 2021

View reviewed changes

sdk/schemaregistry/schema-registry-avro/src/models.ts Outdated Show resolved Hide resolved

deyaaeldeen added 4 commits December 15, 2021 19:29

Merge remote-tracking branch 'upstream/main' into schemaregistry/new-…

01188f1

…avro-serializer-design

finish implementation

dc29a9f

edit

020d822

edit

3892375

deyaaeldeen added 2 commits December 17, 2021 17:56

add samples for event hubs interaction

df67b66

edit

8ccf8b1

deyaaeldeen marked this pull request as ready for review December 18, 2021 02:01

deyaaeldeen requested review from bterlson, jeremymeng, nguerrera and xirzec as code owners December 18, 2021 02:01

edit

0f60dc0

deyaaeldeen mentioned this pull request Dec 18, 2021

[Event Hubs] Add support for MessageWithMetadata #19458

Closed

deyaaeldeen commented Dec 20, 2021

View reviewed changes

JoshLove-msft reviewed Jan 3, 2022

View reviewed changes

sdk/schemaregistry/schema-registry-avro/README.md Outdated Show resolved Hide resolved

JoshLove-msft reviewed Jan 3, 2022

View reviewed changes

sdk/schemaregistry/schema-registry-avro/README.md Outdated Show resolved Hide resolved

JoshLove-msft reviewed Jan 3, 2022

View reviewed changes

JoshLove-msft approved these changes Jan 3, 2022

View reviewed changes

Apply suggestions from code review

3bed639

Co-authored-by: JoshLove-msft <[email protected]>

xirzec approved these changes Jan 5, 2022

View reviewed changes

merge upstream/main

8a69f09

deyaaeldeen mentioned this pull request Jan 14, 2022

[Schema Registry Avro] New Encoder Design #19842

Merged

deyaaeldeen closed this Jan 26, 2022

deyaaeldeen deleted the schemaregistry/new-avro-serializer-design branch January 26, 2022 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Schema Registry Avro] Providing a message encoder #18365

[Schema Registry Avro] Providing a message encoder #18365

deyaaeldeen commented Oct 26, 2021 •

edited

Loading

azure-sdk commented Oct 26, 2021

azure-sdk commented Dec 16, 2021

deyaaeldeen Dec 20, 2021

JoshLove-msft Jan 3, 2022

JoshLove-msft Jan 3, 2022

JoshLove-msft Jan 3, 2022

xirzec left a comment

xirzec Jan 5, 2022

xirzec Jan 5, 2022

deyaaeldeen commented Jan 26, 2022

	- The `deserialize` method has been renamed to `encodeMessageData` and it now takes a `MessageWithMetadata` as input
	- The `deserialize` method has been renamed to `decodeMessageData` and it now takes a `MessageWithMetadata` as input

[Schema Registry Avro] Providing a message encoder #18365

[Schema Registry Avro] Providing a message encoder #18365

Conversation

deyaaeldeen commented Oct 26, 2021 • edited Loading

azure-sdk commented Oct 26, 2021

azure-sdk commented Dec 16, 2021

deyaaeldeen Dec 20, 2021

Choose a reason for hiding this comment

JoshLove-msft Jan 3, 2022

Choose a reason for hiding this comment

JoshLove-msft Jan 3, 2022

Choose a reason for hiding this comment

JoshLove-msft Jan 3, 2022

Choose a reason for hiding this comment

xirzec left a comment

Choose a reason for hiding this comment

xirzec Jan 5, 2022

Choose a reason for hiding this comment

xirzec Jan 5, 2022

Choose a reason for hiding this comment

deyaaeldeen commented Jan 26, 2022

deyaaeldeen commented Oct 26, 2021 •

edited

Loading