[Core] Introduce JsonData type in Azure.Core

[annelo-msft]

API Change Needed

Yes

Background and motivation

The JsonData type provides a dynamic layer for accessing JSON and is intended to improve the developer experience, over JsonDocument which we recommend to users today, when working with protocol methods. This type will improve usability for raw JSON that comes back from protocol methods as BinaryData.

Goals

• Make samples showing clients with protocol methods look like .NET codecode calling protocol methods easier to read.
• Make code reading semi-structured data, e.g. code reading storage table data, easier to read.
• Make it easy easier for customers to update code when Azure SDK-supported models are added to library in a later version

Scenarios

• Use content from a protocol method’s response through a dynamic layer, as though it were a model type
• Use content from a pageable service method, treating each element in a returned collection as an individual dynamic, e.g. Pageable<dynamic>
• Use content from an LRO, treating the final returned value as the dynamic content, e.g. Operation
• Round-tripping is possible, i.e. get payload from response, update a value in payload, send in request.
• Read storage table data using readable syntax. See CloudMachine/DatabaseTests.cs at main · KrzysztofCwalina/CloudMachine (github.com)

Samples

• Example motivating scenario per @tg-msft
• Same scenario using JsonNode per @JoshLove-msft
• [For Discussion] Move JsonData to Core #31769
  • Description includes initial requirements
  • Provides illustration of use in CLU and Confidential Ledger samples
• [For Discussion] dynamic JSON input type #32501
  • Explores making properties settable for round-tripping

Notes

In the past, we could not finalize the prototype of JsonData in Azure.Core.Experimental because we tried to make it read-write. If we scope the problem to just parsing responses, we think we can provide it as a useful type for customers of DPG clients.

API / Feature Proposal

Creation

• Can get dynamic from Response, e.g. response.Content.ToDynamic();
• Internal API allows JsonData to be constructed from BinaryData
• Internal API allows JsonData to be constructed from object (e.g. from anonymous type)

Serialization/Deserialization

• Serialize JsonData instance to JSON UTF8/bytes
• Deserialize to any type supported by JsonSerializer using explicit cast

Usage as output model

• If current node is a leaf node (not an array or an object)

  • Can use a cast to type where either cast or conversion is supported by language, e.g. (DateTime)json
  • Throws if you try to access a member, e.g. json.Property
  • Throws if you try to use an indexer, e.g. json[0] or json[“$foo”]

• If current node is an array node

  • Can get length via Length property, e.g. json.Length
  • Can foreach over, e.g. foreach (int i in json) { }
  • Can access values using indexer that takes int, e.g. json[0]

• If current node is an object

  • Can get and set properties as dynamic members, e.g. json.Property
  • Can cast to model to any type that JsonSerializer can deserialize to
  • Getting a property that isn’t in the JSON returns null and doesn’t throw (this mirrors behavior in Azure SDK model “optional properties”)
  • Can access properties whose names have legal JSON, but illegal C#, characters in them

Other

• Can configure type so properties can be read as PascalCase
• Good debugging experience
• Reasonable performance, i.e. dymanic has perf overhead, and it’s OK because it’s “transparent”. What we don’t want is “hidden” costs, e.g. setting a property results in disproportionally large allocation.
• Extensibility for arbitrary data formats, e.g. XML.

Prior Description

• Current prototype (read/write) per @ellismg
• API View of an immutable surface + constructors
• Proposed updates (read only) per @jsquire

Time Constraints and Dependencies

If possible, we would like to have this type by late September (@KrzysztofCwalina to comment on timeline needs)

Stakeholders

@KrzysztofCwalina, DPG customers

[JoshLove-msft]

I guess the obvious question is why this belongs in Azure.Core. There doesn't seem to be anything specific to Azure here, and seems like it could go in the BCL, similar to BinaryData.

Also, how does this fit in with the JsonNode type introduced in System.Text.Json?

[ellismg]

Some historical context - when we originally designed JsonData, the JsonNode type did not exist. We actually spent one or two meetings talking with the BCL team as they were working on early JsonNode designs but I am not sure we ever went back to look at the final design and see if we could use it over our handcrafted JsonData (I'm certain that I did not do this exercise).

Another thing pushing the custom time at the time were schedule concerns from our end (ameliorated now as they've shipped) and there was also at the time some concern about having a type that was Net Standard 2.0 compatible (or whatever the minimum version of NS that the Azure SDKs are targeting these days).

[JoshLove-msft]

Ah that makes sense given the timelines. JsonNode was added pretty recently. Regarding the minimum version considerations, since STJ is shipped as an OOB package, I wonder if we could consider upgrading the central repo dependency considering that this is a BCL library that has very strong guarantees when it comes to compatibility. That is, if this type actually addresses our needs from the SDK side.

[annelo-msft]

@KrzysztofCwalina, do you have a clear set of requirements for your current need for JsonData? Would JsonNode suffice for those purposes?

[KrzysztofCwalina]

Does JsonNode support dynamic? I don't think so, but if it does, we could use it.

The requirement is to be able to do:

Response response = client.Get();
dynamic json = JsonData.Parse(response.Content);
Console.WriteLine(json.Foo);
Console.WriteLine(json.Foo.Bar);
Console.WritelIne(json);
Console.WriteLine(((int)json) + 5);
Console.WriteLine(json[5]); // not sure if it's possible, if not we need to invent a way to deal with arrays.

The main work needed, given that we have the prototype, is to:

1. Remove mutating APIs.
2. Decide how casts work, e.g. var date = (DateTime)json; or cast to custom model: var model = (FooModel)json.Foo
3. Decide how arrays work (see above)
4. Decide how roundtripping would work. It will not be very easy/usable, but it should be possible.
5. Implement a good debugger visualizer
6. Update our protocol method docs

[JoshLove-msft]

Does JsonNode support dynamic? I don't think so, but if it does, we could use it.

It doesn't look like it does, but it has indexers so you could do things like:

JsonNode json = JsonNode.Parse(response.ContentStream);
Console.WriteLine(json["foo"]);
Console.WriteLine(json["foo"]["bar"]);

It also has a bunch of implicit operators so I think you can do:

Console.WriteLine(((int)json) + 5);

And indexing into an array is supported:

Console.WriteLine(json[5]);

[KrzysztofCwalina]

The casts and array accessors look good, but I think the property access syntax is so much messier. We might want to do a user study to compare these two.

[JoshLove-msft]

JsonNode usage example with TextAnalytics - https://gist.github.com/JoshLove-msft/34897c9f5d3250e8b23167ce5c01c2b6

[annelo-msft]

Per conversation with @KrzysztofCwalina, the DPG user experience will come down to:

• JsonData + anonymous types, with RequestContent.Merge method for round-tripping or
• JsonNode for input/output

His observation is that anonymous types are the clear winner over JsonNode on the input side. There were several cons here:

• New users coming to DPG will need to learn how to use JsonData, anonymous types, and the Merge API vs. learning one JsonNode API, supported by BCL docs
• Anonymous types work well when you can use static initializers - if inputs need to be populated dynamically from objects that could vary in their composition (e.g. the output of a DB query), this may not work well.

Most problems that JsonData has around lack of Intellisense and compiler safety are shared by JsonNode, although you do get compile-time checks on JsonNode usage that are deferred to runtime with JsonData.

Overall, the thinking behind JsonData is that with this approach the code, once written, best expresses the user's intent. There remains to be an in-depth analysis of complex types and advanced scenarios.

[JoshLove-msft]

Agreed that anonymous types are easier for input, especially when not roundtripping. I don't see how using JsonNode for return types would preclude using anonymous types for input.

[annelo-msft]

@JoshLove-msft, I made the same point. @KrzysztofCwalina's perspective was that we either embrace JsonNode for the full round-trip, or we go with the JsonData/Merge/anonymous type story. It's true that anonymous types could still be used for input with JsonNode, but if round-tripping requires using JsonNode for input, it's not clear why we would write DPG docs and samples using JsonNode for output and not input. So the argument was largely driven around the usability of anonymous types over JsonNode for input, not the comparison of JsonNode to JsonData.

As we mentioned earlier, @KrzysztofCwalina did also feel that we need more analysis of JsonData/anonymous types vs. JsonNode for complex types and advanced scenarios - that's work I haven't done yet that remains to be filled in before either approach would be greenlighted.

[JoshLove-msft]

but if round-tripping requires using JsonNode for input, it's not clear why we would write DPG docs and samples using JsonNode for output and not input

I think it depends on the scenario. For the initial request you can use anonymous types or JsonNode. When you get back a JsonNode, and you need to send it back to the service, you would use JsonNode.

I think the usability is better with JsonNode when you are working with a response from the service that you either want to send as-is or mutate. For the initial request, anonymous types are easier.

[annelo-msft]

Sure. I think the overarching concern there is that we'd like consistency in the documentation around DPG, and not to have to special case things too much, since DPG is going to be advanced enough as it is.

[JoshLove-msft]

I wonder if making JsonNode more anonymous type friendly would be an option?
If there was a Parse overload or even a constructor (similar to how BinaryData works) that took an object, that seems like it would get us almost the same experience as JsonData.

[JoshLove-msft]

But thinking about this more, I'm not sure I understand the argument around anonymous types for input. For JsonData, I assume we would use anonymous types for input only + JsonData for round-tripping, right? Or am I missing the end-to-end flow here? Do you have an example of RequestContent.Merge works?

Nevermind, found this from an email thread:

// ***
// Round-trip: JsonData and Anonymous types
// ***
Response current = await client.GetKeyValueAsync("FontColor");
RequestContent updated = RequestContent.Merge(current, patch:
    new
    {
        value = "updated value";
    }
);

dynamic keyValue = new JsonData(current.Content);
Response putResponse = await client.PutKeyValueAsync((string)keyValue.key, updated, ContentType.ApplicationJson);

I think I understand the argument now. When using JsonData, we always use anonymous types for input, even for round-tripping. I think this will be sort of complicated to use the Merge API for nested models though. For instance, let's say you wanted to change one property of a sub model, do users need to specify all other properties, or would the Merge just ignore any missing properties? i.e. how would I delete a property vs how would I set a value to null? Or do we not need that type of semantics (we don't need to let users remove properties)?

[jsquire]

would the Merge just ignore any missing properties

I believe it should follow the HTTP PATCH semantics - keep the existing data and overwrite anything that is provided when calling Merge.

Therre's precedent for this kind of pattern if you consider C# and F# records and the with syntax - though those create copies rather than in-place mutation. (example)

[annelo-msft]

If there was a Parse overload or even a constructor (similar to how BinaryData works) that took an object, that seems like it would get us almost the same experience as JsonData.

@JoshLove-msft, this makes me curious about the BinaryData API. Do you know why some conversions to BinaryData are handled as constructors and some as FromXx static methods?

[JoshLove-msft]

@JoshLove-msft, this makes me curious about the BinaryData API. Do you know why some conversions to BinaryData are handled as constructors and some as FromXx static methods?

I believe there are static methods and constructors for each conversion with the exception of streams. The static methods allow you to use async. IIRC we didn't add a stream constructor because it would seem to imply that we are wrapping the stream, when in reality, we would be consuming it and storing the bytes in memory.

[JoshLove-msft]

I believe it should follow the HTTP PATCH semantics - keep the existing data and overwrite anything that is provided when calling Merge.

This makes sense - I suppose we don't really need the ability to remove properties for DPG as you would generally be getting and then updating data with the same JSON schema.

[annelo-msft]

I believe there are static methods and constructors for each conversion with the exception of streams. The static methods allow you to use async.

Can you say more about why we had the constructors? Neither JsonNode nor JsonDocument have constructors, so I'm wondering what the semantics of the constructor imply that is different from the Parse method, to understand whether we would want a constructor taking object on JsonData.

@ellismg, do you know the .NET thinking around this?

[JoshLove-msft]

IIRC the thinking was that constructors are more discoverable, but we also needed at least some static methods to support async operations, i.e. FromStreamAsync.

[ellismg]

What @JoshLove-msft said tracks with my understanding.

[annelo-msft]

Released in https://www.nuget.org/packages/Azure.Core/1.33.0