Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Developers can customize the JSON serialization contracts of their types #63686

Closed
13 tasks done
Tracked by #63762
eiriktsarpalis opened this issue Jan 12, 2022 · 41 comments
Closed
13 tasks done
Tracked by #63762
Assignees
Labels
api-approved API was approved in API review, it can be implemented area-System.Text.Json Cost:L Work that requires one engineer up to 4 weeks Priority:0 Work that we can't release without Team:Libraries User Story A single user-facing feature. Can be grouped under an epic.
Milestone

Comments

@eiriktsarpalis
Copy link
Member

eiriktsarpalis commented Jan 12, 2022

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

  1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

    1. It requires that the user owns the type declarations they are looking to customize.
    2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
    3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
    4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.
  2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

    1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
    2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects (we're planning on addressing this independently via Developers should be able to pass state to custom converters. #63795).

API Proposal

namespace System.Text.Json 
{
    public sealed partial class JsonSerializerOptions
    {
        public IJsonTypeInfoResolver TypeInfoResolver { [RequiresUnreferencedCode] get; set; }
    }
}

namespace System.Text.Json.Serialization 
{
    public abstract partial class JsonSerializerContext : IJsonTypeInfoResolver
    {
        // Explicit interface implementation calling into the equivalent JsonSerializerContext abstract method
        JsonTypeInfo System.Text.Json.Serialization.Metadata.IJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options);
    }
}

namespace System.Text.Json.Serialization.Metadata 
{
    public interface IJsonTypeInfoResolver
    {
        JsonTypeInfo? GetTypeInfo(Type type, JsonSerializerOptions options);
    }

    // Provides the default reflection-based contract metadata resolution
    public class DefaultJsonTypeInfoResolver : IJsonTypeInfoResolver
    {
        [RequiresUnreferencedCode]
        public DefaultJsonTypeInfoResolver() { }

        public virtual JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options);
        
        public IList<Action<JsonTypeInfo>> Modifiers { get; }
    }

    public static class JsonTypeInfoResolver
    {
        public static IJsonTypeInfoResolver Combine(params IJsonTypeInfoResolver[] resolvers);
    }

    // Determines the kind of contract metadata a given JsonTypeInfo instance is customizing
    public enum JsonTypeInfoKind
    {
        None = 0, // Type is either a primitive value or uses a custom converter -- contract metadata does not apply here.
        Object = 1, // Type is serialized as a POCO with properties
        Enumerable = 2, // Type is serialized as a collection with elements
        Dictionary = 3 // Type is serialized as a dictionary with key/value pair entries
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    // note: added abstract
    public abstract partial class JsonTypeInfo
    {
        public Type Type { get; }
        public JsonSerializerOptions Options { get; }

        // The converter instance associated with the type for the given options instance -- this cannot be changed.
        public JsonConverter Converter { get; }

        // The kind of contract metadata we're customizing
        public JsonTypeInfoKind Kind { get; }

        // Untyped default constructor delegate -- deserialization not supported if set to null.
        public Func<object>? CreateObject { get; set; }

        // List of property metadata for types serialized as POCOs.
        public IList<JsonPropertyInfo> Properties { get; }

        // Equivalent to JsonNumberHandlingAttribute annotations.
        public JsonNumberHandling? NumberHandling { get; set; }

        // Factory methods for JsonTypeInfo
        public static JsonTypeInfo<T> CreateJsonTypeInfo<Τ>(JsonSerializerOptions options) { }
        public static JsonTypeInfo CreateJsonTypeInfo(Type type, JsonSerializerOptions options) { }

        // Factory methods for JsonPropertyInfo
        public JsonPropertyInfo CreateJsonPropertyInfo(Type propertyType, string name) { }
    }

    // remove: [EditorBrowsable(EditorBrowsableState.Never)]
    public abstract partial class JsonTypeInfo<T> : JsonTypeInfo
    {
        // Default typed constructor delegate
        public new Func<T>? CreateObject { get; set; }
    }

    // remove: [System.ComponentModel.EditorBrowsableAttribute(System.ComponentModel.EditorBrowsableState.Never)]
    public abstract partial class JsonPropertyInfo
    {
        public JsonSerializerOptions Options { get; }
        public Type PropertyType { get; }
        public string Name { get; set; }

        // Custom converter override at the property level, equivalent to `JsonConverterAttribute` annotation.
        public JsonConverter? CustomConverter { get; set; }

        // Untyped getter delegate
        public Func<object, object?>? Get { get; set; }

        // Untyped setter delegate
        public Action<object, object?>? Set { get; set; }
    
        // Predicate determining whether a property value should be serialized
        public Func<object, object?, bool>? ShouldSerialize { get; set; }

        // Equivalent to JsonNumberHandlingAttribute overrides.
        public JsonNumberHandling? NumberHandling { get; set; }
    }
}

Usage examples

Custom resolver with constructed JsonTypeInfo

static void Main()
{
    MyType[] values = new MyType[] {
        new() { MyStringId = "123", HideName = true, Name = "John" },
        new() { MyStringId = "124", Name = "James" },
        new() { MyStringId = "-1" },
    };
    JsonSerializerOptions options = new();
    options.TypeInfoResolver = new MyCustomResolver();
    string output = JsonSerializer.Serialize(values, options);
    // [{"ID":123},{"ID":124,"Name":"James"},{}]
}

class MyType
{
    public bool HideName { get; set; }
    public string Name { get; set; }
    public string? MyStringId { get; set; } = "-1";
}

class MyCustomResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        if (type != typeof(MyType))
            return base.GetTypeInfo(type, options);

        JsonTypeInfo<MyType> jti = JsonTypeInfo.CreateJsonTypeInfo<MyType>(options);
        jti.CreateObject = () => new MyType();

        JsonPropertyInfo<int> propId = jti.CreateJsonPropertyInfo<int>("ID");

        propId.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.MyStringId == null ? -1 : ConvertStringIdToInt(m.MyStringId);
        };

        propId.Set = (object o, int val) =>
        {
            MyType m = (MyType)o;
            m.MyStringId = ConvertIntIdToString(val);
        };

        propId.CanSerialize = (object o, int val) =>
        {
            MyType m = (MyType)o;
            return val != -1;
        };

        jti.Properties.Add(propId);

        JsonPropertyInfo<string> propName = jti.CreateJsonPropertyInfo<string>("Name");
        propName.Get = (object o) =>
        {
            MyType m = (MyType)o;
            return m.Name;
        };

        propName.Set = (object o, string val) =>
        {
            MyType m = (MyType)o;
            m.Name = val;
        };

        propName.CanSerialize = (object o, string val) =>
        {
            MyType m = (MyType)o;
            return !m.HideName && m.Name != null;
        };

        jti.Properties.Add(propName);

        return jti;
    }

    private static int ConvertStringIdToInt(string id) => int.Parse(id);
    private static string ConvertIntIdToString(int id) => id.ToString();
}

Adding support for DataMemberAttribute annotations

public class SystemRuntimeSerializationAttributeResolver : DefaultJsonTypeInfoResolver
{
    public override JsonTypeInfo GetTypeInfo(Type type, JsonSerializerOptions options)
    {
        JsonTypeInfo jsonTypeInfo = base.GetJsonTypeInfo(type, options);

        if (jsonTypeInfo.Kind == JsonTypeInfoKind.Object &&
            type.GetCustomAttribute<DataContractAttribute>() is not null)
        {
            jsonTypeInfo.Properties.Clear();

            foreach ((PropertyInfo propertyInfo, DataMemberAttribute attr) in type.GetProperties(BindingFlags.Instance | BindingFlags.Public)
                .Select((prop) => (prop, prop.GetCustomAttribute<DataMemberAttribute>() as DataMemberAttribute))
                .Where((x) => x.Item2 != null)
                .OrderBy((x) => x.Item2!.Order))
            {
                JsonPropertyInfo jsonPropertyInfo = jsonTypeInfo.CreateJsonPropertyInfo(propertyInfo.PropertyType, attr.Name ?? propertyInfo.Name);
                jsonPropertyInfo.Get =
                    propertyInfo.CanRead
                    ? propertyInfo.GetValue
                    : null;

                jsonPropertyInfo.Set = propertyInfo.CanWrite
                    ? (obj, value) => propertyInfo.SetValue(obj, value)
                    : null;

                jsonTypeInfo.Properties.Add(jsonPropertyInfo);
            }
        }

        return jsonTypeInfo;
    }
}

Combining resolver

Doc comparing 3 design variants with recommendation can be found here: https://gist.github.com/krwq/c61f33faccc708bfa569b3c8aebb45d6

JsonPropertyInfo vs JsonPropertyInfo<T> vs JsonPropertyInfo<TDeclaringType, TPropertyType>

We have considered different approaches here and it all boils down to perf of the property setter.
According to simple perf tests run on different combinations of declaring types and property types as well 4 different approaches of setters using setter in form of:
delegate void PropertySetter<DeclaringType, PropertyType>(ref DeclaringType obj, PropertyType val);

proves to be overall fastest. Current implementation would require a bit of work for this to be changed and such support can be added later. Given above we've decided to for a time being support only non-generic PropertyInfo with the slowest setter since such type already exists and corresponding setter would have to be added regardless of choice. In the future PropertyInfo<TDeclaringType, TPropertyType> should be added to support for the fastest possible case.

Here are benchmark results: https://gist.github.com/krwq/d9d1bad3d59ff30f8db2a53a27adc755
Here is the benchmark code: https://gist.github.com/krwq/eb06529f0c99614579f84b69720ab46e

Acceptance Criteria

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that should be user customizable in the contract model:

Use cases

Open Questions

Progress

cc @steveharter @JamesNK

@eiriktsarpalis eiriktsarpalis added area-System.Text.Json User Story A single user-facing feature. Can be grouped under an epic. Priority:0 Work that we can't release without Cost:L Work that requires one engineer up to 4 weeks Team:Libraries labels Jan 12, 2022
@eiriktsarpalis eiriktsarpalis added this to the 7.0.0 milestone Jan 12, 2022
@eiriktsarpalis eiriktsarpalis self-assigned this Jan 12, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jan 12, 2022
@ghost
Copy link

ghost commented Jan 12, 2022

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

Background and Motivation

Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level:

  1. Customizing the contract via attribute annotations. This is generally our recommended approach when making straightforward tweaks to the contract, however it comes with a few drawbacks:

    1. It requires that the user owns the type declarations they are looking to customize.
    2. It forces repetition when applying the same rule across multiple members. This can be problematic for users defining large sets of DTOs or library authors looking to extend a rule to arbitrary sets of types.
    3. There are inherent limits to the degree of customization achievable. For example, while it is possible to skip a property via JsonIgnoreAttribute, it is impossible to add a JSON property to the contract that doesn't correspond to a .NET property.
    4. For certain users that prefer serializing their domain model directly, introducing System.Text.Json dependencies to the domain layer is considered poor practice.
  2. Authoring custom converters. While this mechanism is general-purpose enough to satisfy most customization requirements, it suffers from a couple of problems:

    1. Making straightforward amendments like modifying serializable properties, specifying constructors or injecting serialization callbacks can be cumbersome, since it effectively requires replicating the entire object/collection serialization logic.
    2. Currently, custom converters do not support async/resumable serialization which can result in performance bottlenecks when serializing large objects.

Proposal

System.Text.Json already defines a JSON contract model internally, which is surfaced in public APIs via the JsonTypeInfo/JsonPropertyInfo types as opaque tokens for consumption by the source generator APIs. This is a proposal to define an IContractResolver-like construct that builds on top of the existing contract model and lets users generate custom contracts for System.Text.Json using runtime reflection.

Here is a provisional list of settings that could be user customizable in the contract model:

Progress

  • Author a working prototype.
  • API proposal & review.
  • Implementation & tests.
  • Conceptual documentation & blog posts.

cc @steveharter @JamesNK

Author: eiriktsarpalis
Assignees: eiriktsarpalis
Labels:

area-System.Text.Json, User Story, Priority:0, Cost:L, Team:Libraries

Milestone: 7.0.0

@DRKV333
Copy link

DRKV333 commented Jan 20, 2022

I would like to add one more use case for a contract system like this, that would be nice to have support for in System.Text.Json.
Sometimes, especially in dynamically typed languages, people think it's ok to just throw different properties of an object, usually with different types, into a single array.

So, for instance, if you're serializing a person, instead of:

{
    "name": "John",
    "age": 21
}

you'd end up with:
["John", 21]
where the meaning of each item is denoted by their position in the array.

I would like to be able to serialize/deserialize this into a .NET class. Dealing with this currently in System.Text.Json, would involve making a separate converter for each object that is serialized this way, using slow runtime reflection, or coming up with my own fast reflection mechanism (which is a bit redundant, since the serializer already has this internally). In Newtonsoft.Json the situation can be handled by a single generic converter, without any direct use of reflection.

https://github.com/DRKV333/Janki/blob/d3ffc4743b861cb318debbb53ad482ee2a5d9eb6/JankiClientCards/Converters/ObjectToArrayJsonConverter.cs

This setup essentially requires:

  • Resolving an object's serialization contract in a converter. JsonSerializer.ContractResolver.ResolveContract
  • Enumerating all serializable properties of the object in question, in the right order. JsonObjectContract.Properties
  • Creating a new instance of the object. JsonObjectContract.DefaultCreator
  • Reading and writing the value of a specific property on an object instance. JsonProperty.ValueProvider.GetValue/SetValue
  • Accessing a property's converter to serialize and deserialize the property's value. JsonProperty.Converter

Though if I understand it correctly, most of these should be covered by the current list of planned features.

@Danielku15
Copy link

@bartonjs Did you consider the cost of boxing/unboxing in this API design? My concerns mainly are regarding the JsonPropertyInfo members which rely on object at various places. For value types this would mean the serializer has to box/unbox them during serialization.

@eiriktsarpalis
Copy link
Member Author

Did you consider the cost of boxing/unboxing in this API design? My concerns mainly are regarding the JsonPropertyInfo members which rely on object at various places. For value types this would mean the serializer has to box/unbox them during serialization.

Yes. The short answer is boxing structs is necessary in order for setter delegates to work as expected. See #63686 (comment) for more details.

@krwq
Copy link
Member

krwq commented Jun 13, 2022

@Danielku15 see the JsonPropertyInfo vs JsonPropertyInfo<T> vs JsonPropertyInfo<TDeclaringType, TPropertyType> section in the issue description. We've found the fastest way is to use ref everywhere but we've also found the current implementation is a bit convoluted to make that work in this release. We've left that option for future since non-generic JsonPropertyInfo will need to exist anyway even if we went with more complex design now. Also we need to make sure that this design will work correctly with polymorphism which is being worked on now as well.

Also note that internally we will use some optimizations to avoid boxing for DefaultJsonTypeInfoResolver so it should have minimal impact but we hope to improve that further and make it public in the future

@krwq
Copy link
Member

krwq commented Jun 24, 2022

FYI: main feature branch is already merged and this will be available in preview 6 (or earlier on nightly builds). There are still some improvements planned (see #71123). Please let us know if you have any feedback

@DRKV333
Copy link

DRKV333 commented Jun 25, 2022

I'm a little bit confused about how the use-case I mentioned here: #63686 (comment) could be handled with with this new system. I feel like it might be possible, but I couldn't figure it out.
I tried just making a new JsonConverter<>, and using options.TypeInfoResolver.GetTypeInfo(typeToConvert, options) to get the contract for the object I'm currently reading. But this JsonTypeInfo does not have any properties... I'm guessing this is because the type has a custom converter, and in this case the resolver doesn't bother with finding them.
I then tried making a custom IJsonTypeInfoResolver to maybe get access to the original JsonTypeInfo, and then substitute a different one with my custom converter. But it looks like there's no way to supply a different converter from a custom resolver, cause JsonTypeInfo.Converter is read-only...

@krwq
Copy link
Member

krwq commented Jun 27, 2022

@DRKV333 unfortunately this scenario is out of scope in this release although we've had long discussion on supporting this - we've decided it's better to support less in this release than supporting everything at once in 2 releases (having said that I'd really wish we could add support for that but at the same time this feature is already large as is and our team has limited resources).

Currently converter determines the JsonTypeInfoKind and also currently there is no way to write custom converter which sets it to value you want, it will always have JsonTypeInfoKind.None which means it won't use properties.
Presumably in the future possible way to support this is by providing abstract converter which can be overriden and has Read/Write which take JsonTypeInfo or just properties and will mean converter knows what to do with properties therefore its kind is Object. There are couple of drawbacks with supporting that in V1: we need to provide way to call serializer recursively. It needs separate API proposal and a bit of thinking how will that work with all existing internal knobs.

FWIW you could possibly write JsonConverterFactory which uses DefaultJsonTypeInfoResolver to get properties for all types you want to serialize as PLIST - if you get into reasonable solution perhaps you will have some ideas how this could be exposed better and create API proposal which we could take into consideration for vNext.

@eiriktsarpalis
Copy link
Member Author

eiriktsarpalis commented Jun 28, 2022

With the bulk of the implementation already merged in Preview 6, we should consider adding the following APIs so that all STJ attribute annotations/interfaces are mapped to the metadata model:

public class JsonTypeInfo
{
    // Maps to IJsonOnSerializing
    public Action<object>? OnSerializing { get; set; }
    // Maps to IJsonOnSerialized
    public Action<object>? OnSerialized { get; set; }
    // Maps to IJsonOnDeserializing
    public Action<object>? OnDeserializing { get; set; }
    // Maps to IJsonOnDeserialized
    public Action<object>? OnDeserialized { get; set; }
}

public class JsonPropertyInfo
{
    // Maps to JsonPropertyOrderAttribute
    public int Order { get; set; } = 0;
    // Maps to JsonExtensionDataAttribute
    public bool IsExtensionData { get; set; }
}

A few remarks on the above:

  • We don't expose typed callbacks on JsonTypeInfo<T> since it would have the same issues as typed setters acting on struct types.
  • While property order can still be specified implicitly by rearranging elements in the Properties list, this can be cumbersome since there's no way to do in-place sorting in an IList<JsonPropertyInfo> that cannot be set. Users are forced to clear the list and reinsert properties in the desired order. Order would act as a secondary key when sorting properties when the metadata gets locked.
  • This only leaves out JsonConstructorAttribute from the model, which is partially addressed by JsonTypeInfo.CreateObject but currently does not provide a way to specify parameterized constructors. Supporting parameterized constructors is nontrivial and should best be addressed in a future release. See Developers can customize the JSON serialization contracts of their types #63686 (comment) for a brief discussion on design considerations.

@eiriktsarpalis
Copy link
Member Author

With .NET 7 work for the feature being mostly done, I have created a user story tracking contract customization work planned for .NET 8: #71967

@krwq
Copy link
Member

krwq commented Jul 20, 2022

Since all of the bullet points are now closed and we've shipped what we've planned I think this can be closed now. Any remaining issues should be tracked separately

@krwq krwq closed this as completed Jul 20, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Aug 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api-approved API was approved in API review, it can be implemented area-System.Text.Json Cost:L Work that requires one engineer up to 4 weeks Priority:0 Work that we can't release without Team:Libraries User Story A single user-facing feature. Can be grouped under an epic.
Projects
None yet
Development

No branches or pull requests

8 participants