Missing systemProperties when consuming messages from IoTHub

[seank-com]

• Package Name:
  @azure/event-hubs
• Package Version:
  5.0.0
• Operating system:
  Linux in a node:alpine Docker container
• nodejs
  • version: 13.8.0
• browser
  • name/version: N/A
• typescript
  • version: N/A
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug

When I configure routes on my IoTHub to put messages storage, they land looking like this:

{
  "EnqueuedTimeUtc":"2020-03-12T17:18:18.5420000Z",
  "Properties":{
    "$.cdid":"EVT6-2-5"
  },
  "SystemProperties":{
    "messageId":"dcb706cc-5170-4de3-b2b1-0fcbed75a14c",
    "correlationId":"b4d58f12-58d4-4454-b8a2-bc7f40db14c8",
    "connectionDeviceId":"EVT6-2-5",
    "connectionAuthMethod":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "connectionDeviceGenerationId":"637159231596457880",
    "contentType":"UTF-8",
    "enqueuedTime":"2020-03-12T17:18:18.5420000Z"
  },
  "Body":"eyJjb...TRjIn0="
}

When I instead consume them using this library, they look like this

{
  "body":{
    ...
  },
  "properties":{
    "$.cdid":"EVT6-2-5"
  },
  "offset":"2308144",
  "sequenceNumber":4717,
  "enqueuedTimeUtc":"2020-03-12T17:57:46.104Z",
  "systemProperties":{
    "iothub-connection-device-id":"EVT6-2-5",
    "iothub-connection-auth-method":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id":"637159231596457880",
    "iothub-enqueuedtime":1584035866072,
    "iothub-message-source":"Telemetry"
  }
}

As you can see, messageId, correlationId & contentType are all missing.

To Reproduce
Steps to reproduce the behavior:

1. goto portal and create an IoTHub and a storage account

2. goto Message routing blade and select Custom endpoints tab and add a Storage route
   a. Name: storage
   b. link it to a container named iothub in the storage account you created in step 1
   c. select JSON for Encoding
   d. File name format: {iothub}/{partition}/{YYYY}/{MM}/{DD}/{HH}/{mm}

3. goto the Routes tab and create two routes
   a. Name: storage Endpoint: storage Data source: Device Telemetry Messages Enable route: Enable Route query: true
   b. Name: default Endpoint: events Data source: Device Telemetry Messages Enable route: Enable Route query: true

4. goto Built-in endpoints blade and create a new consumer group

5. copy the Event Hub-compatible endpoint and consumer group name

6. goto the IoT devices blade and create a device

7. goto to blade for the device and copy the Primary Connection String

8. Add copied string to environment

Service Side

1. npm install --save @azure/event-hubs
2. Implement code like the following on the service side:

const { EventHubConsumerClient } = require('@azure/event-hubs');

var iotClient = new EventHubConsumerClient(
        process.env.IOTHUB_CONSUMER_GROUP,
         process.env.IOTHUB_EVENTHUB_COMPATIBLE_ENDPOINT);

var subscription = iotClient.subscribe(
        {
            processEvents: (events, context) => {
                if (events.length > 0) {
                    events.forEach(element => {
                        // INSPECT ELEMENT HERE
                    });
                    }
            },
            processError: (err, context) => {
            }
        },
        {
            maxBatchSize: 1
        }
    );

Client Side

1. Modify the Android sample with Primary Connection String from above.
2. Deploy and run.
3. Click "Send Message" button

Expected behavior
Expected element in service side code above to contain message-id, correlation-id and content-type

Additional context

I've debugged this a little and it appears the information is lost when the message is converted to an EventDataInternal object. If the following code was added around line 153 of eventData.ts I believe everything would be ok.

if (msg.message_id) {
  if (!data.systemProperties) {
    data.systemProperties = {};
  }
  data.systemProperties["message-id"] = msg.message_id;
}
if (msg.correlation_id) {
  if (!data.systemProperties) {
    data.systemProperties = {};
  }
  data.systemProperties["correlation-id"] = msg.correlation_id;
}
if (msg.content_type) {
  if (!data.systemProperties) {
    data.systemProperties = {};
  }
  data.systemProperties["content-type"] = msg.content_type;
}

I've never worked with Typescript before and there was no information in the README.md or CONTRIBUTING.md on how to build and run unittests, so I don't feel comfortable putting together a PR

[xirzec]

@chradek can you please look into this? Seems like we have an opportunity to better document for contributions too.

[ramya-rao-a]

@seank-com, Apologies for the delay in responding

When I configure routes on my IoTHub to put messages storage,

Can you elaborate on how you do this so that we can try and repro this?

[seank-com]

I've updated the repo steps with more information about creating routes.

[ramya-rao-a]

Thanks @seank-com

When I configure routes on my IoTHub to put messages storage, they land looking like this:

Do you mean the final contents in the storage blob?

@serkantkaraca,
Is the message id, content type and correlation id discussed above expected to be part of the event coming from Event Hubs?

[serkantkaraca]

@ramya-rao-a yes.

See the code here for the complete set of properties - https://github.com/Azure/azure-sdk-for-net/blob/1264615ef858e1c368e1ba2b89771fdb1be69428/sdk/eventhub/Microsoft.Azure.EventHubs/src/Amqp/AmqpMessageConverter.cs#L219

[ramya-rao-a]

Thanks @serkantkaraca

@chradek Let's have the changes @serkantkaraca has pointed above in the upcoming April update

[ramya-rao-a]

Another reference to the missing properties based on the spec from AMQP can be found in section 3.2.4 in http://www.amqp.org/sites/amqp.org/files/amqp.pdf

[seank-com]

@ramya-rao-a , yes after the messages land dump the contents of the json blob from the storage container. You can download it from the containers blade of the storage account in the portal.

[chradek]

@azure/event-hubs version 5.1.0 has been released to address this issue:
https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/event-hubs/CHANGELOG.md#510-2020-04-07

Please let us know if you still encounter any issues after upgrading the Event Hubs SDK.

[seank-com]

Verified! Thanks for the quick turn-around.