custom serialization, deserialization for existing types

[ariebovenberg]

With Scalars, it is possible to create new classes which have custom GraphQL load/dump logic.

However, it can be useful to define load/dump logic for existing types. Examples:

• datetime/date/time objects
• More precise float/int handling, depending on server implementation.

My first thoughts are something like this:

class DateTime(quiz.Serializer):
     """A proxy for python's datetime time, 
     serializing to an integer timestamp"""
     annotation = datetime  # for documenting the accepted types

     @staticmethod
     def load(value: int) -> datetime:
         return datetime.fromtimestamp(value)
         
     @staticmethod
     def dump(obj: datetime) -> int:
         return obj.timestamp()


class Float(quiz.Serializer):
     """A float with support for loading integers from GrqphQL"""
     annotation = float
     
     @staticmethod
     def load(value: Union[int, float]) -> float:
         return float(value)
         
     @staticmethod
     def dump(obj: Union[int, float]) -> float:
         return float(obj)

class Int32(quiz.Serializer):
     """a 32-bit integer (the default for GraphQL)"""
     annotation = int
     
     @staticmethod
     def load(value: int) -> int:
         return int(value)
     
     @staticmethod
     def dump(obj) -> int:
         if not MIN_INT > obj > MAX_INT:
             raise ValueError('number not representable by 32-bit int')
         return int(obj)
         

What I'm happy about:

• This works well with the idea of GenericScalar
• This works well with the idea of passing type overrides to Schema constructors (as is now done with scalars).

Not sure about:

• The name, maybe something else like TypeProxy?
• a class with only static methods is a bit anti-pattern.

[rmarren1]

This could solve the Int / Float problem, but ideally the solution would allow the user to change the code used to validate that a value is a float (e.g. isinstance(value, float) to something like isinstance(value, (int, float))). With this you would need to de-serialize integers as floats across the board. Edit: I see now that your example would work for this.

I also wanted to have enum types return the string value rather than enum object, and this solution would work well for that case.

This pattern looks a bit confusing, maybe I am not quite understanding it. How do these quiz.Serializer objects get related to the classes they are defined for? It could be a mapping in the Schema constructor perhaps (e.g. Float -> FloatSerializar)

My original idea was to organize code for serialization, deserialization, and validation into the type definitions themselves. It looks like the de-serialization and validation logic is here, the serialization logic (I think) is here. This could be instead organized by having a __gql_load__, __gql_dump__ and __validate__ method in each type definition.

This would result in a very similar pattern to the existing custom Scalars treatment, with the only difference being the existence of defaults and the __validate__ method (but Scalars could get this too).

Also, perhaps the code used to validate values given its type can also be customized by the user in this manner?

Here's what I was thinking:

In quiz/types.py

class Float(quiz.Scalar):
    @staticmethod
    def __validate__(value):
        assert isinstance(value, float)

    @staticmethod
    def __gql_dump__(value):
        return str(value)

    @classmethod
    def __gql_load__(cls, value):
        return float(value)
    
class Enum(enum.Enum):
    @classmethod
    def __validate__(cls, value):
        assert value, cls._members_names_
        assert value in cls._member_names_

    @staticmethod
    def __gql_dump__(value):
        return value.value

    @classmethod
    def __gql_load__(cls, value):
        return cls(value)
...

def load_field(type_, field, value):
    type_.__validate__(value)
    return type_.__gql_load__(value)

quiz/build.py

Instead of argument_as_gql(v), could use type(v).__gql_load__(v)

In users custom code

class Float(quiz.types.Float):
    @staticmethod
    def __validate__(value):
        assert isinstance(value, float)
    
class Enum(quiz.types.Enum):
    @static
    def __gql_load__(value):
        return value

...

schema = quiz.Schema.from_path(..., scalars=[URI, MyOtherScalar, ...], override=[Float, Enum])

[ariebovenberg]

This pattern looks a bit confusing, maybe I am not quite understanding it. How do these quiz.Serializer objects get related to the classes they are defined for?

The Serializer objects would be passed to the Schema in the same way as scalars.

schema = quiz.Schema.from_path(..., override=[URI, MyOtherScalar, Float, MyEnum])

My original idea was to organize code for serialization, deserialization, and validation into the type definitions themselves.

I agree that this is the best way to go about it.
What I did not communicate well is that the Serializer is meant as a type definition. The difference with Scalar being that Serializer is not meant to have instances.

This would result in a very similar pattern to the existing custom Scalars treatment, with the only difference being the existence of defaults and the validate method (but Scalars could get this too).

Yes, this.

It looks like the de-serialization and validation logic is here, the serialization logic (I think) is here.

The serialization logic is a bit fragmented accross the codebase. Definitely one of the uglier bits. Here is a rough overview of what happens when creating and executing a query, regarding (de)serialization and validation:

Step 1: The query is validated with quiz.types.validate(). Validity of provided values is checked with isinstance() in _validate_args().

Example of such a validation error, with GitHub API:

schema.query[
    _
    # the `owner` field has the wrong type here.
    # isinstance(4, str) is false
    .repository(owner=4, name='Hello World')[
        _.createdAt
    ]
]

Step 2: The (validated) query is serialized by calling __gql__ on it. The Query delegates this to the SelectionSet, which delegates to Field, which uses argument_as_gql().

Step 3: The response from the server (JSON) is loaded through quiz.types.load, which delegates to load_field.

The code snipped you posted look good. I have some small remarks and will post later today.

[rmarren1]

Thanks, this gives me a better picture. One last question -- how would serializer definitions actually get applied? E.g., I define MyEnum to override Enum, how does serialization know to use MyEnum?

I thought there would need to either be a mapping between over-rides and internally defined types (e.g. override={quiz.types.Enum: myEnum, ...}), or the over-rides would need to exactly match the internal class names (override=[Enum, ...]), but perhaps you had something else in mind.

[ariebovenberg]

Hmmm, what I think you're aiming for is to override the base of all Enum classes. This will have to be a different mechanism from overriding specific classes.

Enum base class

To specify a base class for all enums, I would prefer an explicit solution. Something like this:

schema.from_path(..., enum_base=MyEnumBase)

• the default enum_base will be quiz.types.Enum (current behavior), but any enum.Enum subclass will do.
• all schema-created Enum classes will inherit from the enum_case class.

class MyEnumBase(quiz.Enum):
    """A custom Enum base class. 
    It accepts and returns (valid) strings when interacting with GraphQL"""
    
    @classmethod
    def __gql_dump__(cls, value: str) -> str:
        if value in cls._member_names_:
            return value
        else:
            raise ValidationError(
                '{!r} is not a valid enum member'.format(value))
        
     @classmethod
     def __gql_load__(cls, data: str) -> str:
         # I'm using an assert here, because this is just a sanity check.
         assert data in cls._member_names_, 'unexpected enum value from server'
         return data

Class overrides

schema.from_path(..., overrides=[MySpecificEnum, MyScalar, ...])

Default scalars in quiz.types:

class Float(quiz.ScalarProxy):
    """A GraphQL type definition for `float`.
    It is not meant to be instantiated."""
    
    @staticmethod
    def __gql_dump__(value: Union[float, int]) -> str:
        if math.isnan(value) or math.isinf(value):
            raise quiz.ValidationError('Float value cannot be NaN or Infinity')
        return str(value)
        
    @staticmethod
    def __gql_load__(value: Union[float, int]) -> float:
        return float(value)
        
        
class ID(quiz.ScalarProxy):
    """The GraphQL ID type. Accepts and returns `str`"""
    
    @staticmethod
    def __gql_dump__(value: str) -> str:
        return value
        
    @staticmethod
    def __gql_load__(value: str) -> str:
        return value
        
        
class Integer(quiz.ScalarProxy):
    """The GraphQL integer type. Accepts and returns 32-bit integers"""
    
     @staticmethod
     def __gql_dump__(obj) -> str:
         if not MIN_INT > obj > MAX_INT:
             raise quiz.ValidationError(
                 'number not representable as 32-bit int')
         return str(obj)
         
     @staticmethod
     def __gql_dump__(value: int) -> int:
         return value
         
         
class Boolean(quiz.ScalarProxy):
    ...
    
    
class String(quiz.ScalarProxy):
    ...
         
         
class GenericScalar(quiz.ScalarProxy):
    """Base class for generic GraphQL scalars.
    Accepts any of int, float, bool, and str"""
    
    @staticmethod
    def __gql_dump__(obj) -> str:
        ...
        
    @staticmethod
    def __gql_load__(obj: T) -> T:
        ...
        
    

User-defined override types could look like this:

class DateTime(quiz.ScalarProxy):
    """An example of a custom scalar proxy. 
    Not meant to be instantiated, simply accepts and loads `datatime`"""
    
    @staticmethod
    def __gql_dump__(value: datetime) -> str:
        return str(value.timestamp())
        
    @staticmethod
    def __gql_load__(value: int) -> datetime:
        return datetime.fromtimestamp(value)
        
        
class URI(quiz.Scalar):
    """An example of a custom scalar"""
    
    def __init__(self, url: str):
        self.components = urllib.parse.urlparse(url)

    def __gql_dump__(self) -> str:
        return self.components.geturl()

    @classmethod
    def __gql_load__(cls, data: str) -> URI:
        return cls(data)

• __validate__ becomes part of the __gql_dump__ interface. If validation fails, a ValidationError or similar can be raised.

[rmarren1]

Yeah, the use case I am going for would need to override the base Enum. This solution looks good, I'll see if I can get it started this week

[ariebovenberg]

@rmarren1 great! I'd like to separate this from the serialization question for now, because I'll need to refactor some serialization stuff myself.

I've created a separate issue for the enum base class: #26