Add JsonValue type

[dmontagu]

Adds a JsonValue types with better behavior than the "naive" implementation.

---

In particular, the most naive implementation:

JsonValue: TypeAlias = Union[int, float, str, bool, None, List['JsonValue'], Dict[str, 'JsonValue']]

actually leads to a recursion error:

from typing import Union, List

from typing_extensions import TypeAlias

from pydantic import TypeAdapter

JsonValue: TypeAlias = Union[int, float, str, bool, None, List['JsonValue'], dict[str, 'JsonValue']]

TypeAdapter(JsonValue)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidmontague/.asdf/installs/python/3.12.0/lib/python3.12/typing.py", line 905, in _evaluate
    self.__forward_value__ = _eval_type(
                             ^^^^^^^^^^^
  File "/Users/davidmontague/.asdf/installs/python/3.12.0/lib/python3.12/typing.py", line 407, in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidmontague/.asdf/installs/python/3.12.0/lib/python3.12/typing.py", line 407, in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidmontague/.asdf/installs/python/3.12.0/lib/python3.12/typing.py", line 407, in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidmontague/.asdf/installs/python/3.12.0/lib/python3.12/typing.py", line 407, in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded

While this behavior could perhaps be improved, this is currently expected since there is no ability to know the name of the JsonValue variable at runtime with this approach to defining the type alias.

---

With the next most naive implementation:

JsonValue = TypeAliasType('JsonValue', Union[int, float, str, bool, None, List['JsonValue'], Dict[str, 'JsonValue']])

There are two issues:

• There's a bunch of unnecessary validation overhead when validating from JSON, since anything that can be parsed as JSON will be valid once loaded.
• You get some horrific recursive error messages when something goes wrong.

To expand on this second bullet, let me demonstrate why this is a problem:

from typing import Union, List

from typing_extensions import TypeAliasType

from pydantic import TypeAdapter

JsonValue = TypeAliasType('JsonValue', Union[int, float, str, bool, None, List['JsonValue'], dict[str, 'JsonValue']])

TypeAdapter(JsonValue).validate_python({'a': {'b': ...}})

pydantic_core._pydantic_core.ValidationError: 16 validation errors for nullable[union[int,float,str,bool,list[...],dict[str,...]]]
int
  Input should be a valid integer [type=int_type, input_value={'a': {'b': Ellipsis}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/int_type
float
  Input should be a valid number [type=float_type, input_value={'a': {'b': Ellipsis}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/float_type
str
  Input should be a valid string [type=string_type, input_value={'a': {'b': Ellipsis}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/string_type
bool
  Input should be a valid boolean [type=bool_type, input_value={'a': {'b': Ellipsis}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/bool_type
`list[nullable[union[int,float,str,bool,list[...],dict[str,...]]]]`
  Input should be a valid list [type=list_type, input_value={'a': {'b': Ellipsis}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/list_type
`dict[str,...]`.a.int
  Input should be a valid integer [type=int_type, input_value={'b': Ellipsis}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/int_type
`dict[str,...]`.a.float
  Input should be a valid number [type=float_type, input_value={'b': Ellipsis}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/float_type
`dict[str,...]`.a.str
  Input should be a valid string [type=string_type, input_value={'b': Ellipsis}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/string_type
`dict[str,...]`.a.bool
  Input should be a valid boolean [type=bool_type, input_value={'b': Ellipsis}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/bool_type
`dict[str,...]`.a.`list[nullable[union[int,float,str,bool,list[...],dict[str,...]]]]`
  Input should be a valid list [type=list_type, input_value={'b': Ellipsis}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.4/v/list_type
`dict[str,...]`.a.`dict[str,...]`.b.int
  Input should be a valid integer [type=int_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/int_type
`dict[str,...]`.a.`dict[str,...]`.b.float
  Input should be a valid number [type=float_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/float_type
`dict[str,...]`.a.`dict[str,...]`.b.str
  Input should be a valid string [type=string_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/string_type
`dict[str,...]`.a.`dict[str,...]`.b.bool
  Input should be a valid boolean [type=bool_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/bool_type
`dict[str,...]`.a.`dict[str,...]`.b.`list[nullable[union[int,float,str,bool,list[...],dict[str,...]]]]`
  Input should be a valid list [type=list_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/list_type
`dict[str,...]`.a.`dict[str,...]`.b.`dict[str,...]`
  Input should be a valid dictionary [type=dict_type, input_value=Ellipsis, input_type=ellipsis]
    For further information visit https://errors.pydantic.dev/2.4/v/dict_type

That beast of an error message comes from just trying to validate: {'a': {'b': ...}}!

I was able to address the first bullet by adding an annotation that modifies the core schema to use an any_schema when validating from JSON, so there is no unnecessary overhead.

I was able to address the second bullet through the use of a tagged union with a callable discriminator, where the callable uses the type as the discriminator. With the implementation in this PR, the error you get is just:

from pydantic import TypeAdapter, JsonValue

adapter = TypeAdapter(JsonValue)
adapter.validate_python({'a': {'b': ...}})

pydantic_core._pydantic_core.ValidationError: 1 validation error for json-or-python[json=any,python=tagged-union[list[...],dict[str,...],str,int,float,bool,none]]
dict.a.dict.b
  input was not a valid JSON value [type=invalid-json-type, input_value=Ellipsis, input_type=ellipsis]

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	0b538c0
Status:	✅  Deploy successful!
Preview URL:	https://302e1b02.pydantic-docs2.pages.dev
Branch Preview URL:	https://add-json-value-type.pydantic-docs2.pages.dev

View logs

[dmontagu]

I'll note that 1 validation error for json-or-python[json=any,python=tagged-union[list[...],dict[str,...],str,int,float,bool,none]] is still pretty ugly, but fixing that would require changes in pydantic-core. (Which I think we should make, to be clear.)

[samuelcolvin]

LGTM, does this need more documentation?

[sydney-runkle]

Nice work @dmontagu! Thanks for the detailed context in the description - the horrible error message was helpful in understanding why you implemented this in the way you did 😎.

[alexmojaki]

I'm trying to understand the context since this seems separate from config.JsonValue and there's no linked issue. Is this a new feature for users? Should I be able to find JsonValue somewhere in https://302e1b02.pydantic-docs2.pages.dev/ or https://add-json-value-type.pydantic-docs2.pages.dev/?

[sydney-runkle]

@alexmojaki, looks like I missed an indent on the docs I added, I'll fix that now.
Yeah, this is a new type that'll be available to users in v2.5

[sydney-runkle]

Fix here: #8034
https://docs.pydantic.dev/dev/api/types/#pydantic.types.JsonValue