-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Define a JSON type #182
Comments
I tried to do that but a recursive type alias doesn't work in mypy right now, and I'm not sure how to make it work. In the mean time I use (I'm sure you meant |
You are right about what I meant and I fixed my comment to not confuse anyone in the future. And I understand about the lack of recursive object support. Would this be a better definition? JSONValue = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]]
JSONType = t.Union[t.Dict[str, JSONValue], t.List[JSONValue]] If you read RFC 4627 it says a JSON object must be an object or array at the top level (RFC 7159 loosens that to any JSON value, but it's not an accepted standard). If you want to play it safe with what JSONType = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]] I guess the real question is how far you want to take this, because if you assume that most JSON objects only go, e.g. 4 levels deeps, you could handcraft accuracy to that level: _JSONType_0 = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]]
_JSONType_1 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_0], t.List[_JSONType_0]]
_JSONType_2 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_1], t.List[_JSONType_1]]
_JSONType_3 = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_2], t.List[_JSONType_2]]
JSONType = t.Union[str, int, float, bool, None, t.Dict[str, _JSONType_3], t.List[_JSONType_3]] But then again the union of those objects is pretty broad so this might not be the most useful type hint. :) |
I guess that'll work, but I'm not convinced that it's very useful to The next question is where this would live? Would you add it to |
OK, so As for where it should go, I don't have a good answer unfortunately. It's like the If we put it in |
I think it's best to add it to the json module; we can't keep adding everything to typing.py (even the io and re types there are already questionable). Code that wants to use these in Python 3.5 or earlier can write
(Or they can copy the definition into their own code.) Question: should we name it |
Actually, since the json module consistently uses JSONWhatevs, it should be JSONType. |
PEP 8 says to capitalize abbreviations. Open an issue for the stdlib at http://bugs.python.org/issue26396 and one for typeshed at python/typeshed#84. |
I'm marking this for 3.5.2 so we at least have the discussion. I'm still not sure of the solution -- add it to typeshed/.../json.pyi or to typing.py? It can't appear in the stdlib json module until 3.6 but it could appear in typing.py in 3.5.2 (since typing.py is provisional), but I'm not excited about pushing everything to typing.py. So maybe adding it to typeshed/**/json.pyi now and the stdlib json module in 3.6 would be best? If you want to use it then you'd have to write (I've got a feeling I'm just summarizing where we ended up before but I'm currently in the mood to make the definitive list of things to discuss and/or implement before 3.5.2.) |
I'm not a fan of having a "partial" JSON type that soon degenerates into Having a recursive JSON type seems like a better idea to me, but even then I'd like to see how the type works for real-world code before including it in the PEP. I suspect that the majority of code doing JSON parsing actually doesn't perform enough Anyway, if programmers want to use such as partial type, they can define the alias and use it in their code even without making it official, though they may have to introduce some explicit type annotations when interacting with library code that doesn't use the type. |
Two problems with adding a recursive JSON type to the PEP:
|
The summary @gvanrossum gave of where things left off was accurate. Didn't come up with a recursive type that could work. In response to @JukkaL about usefulness, I view it as useful for specifying what |
Somehow this was closed but we don't even have consensus! |
I forgot to confirm that mypy doesn't support the kinds of recursive types discussed above, and there are no concrete plans to implement them. @brettcannon I agree that JSON values are common in programs, but I'm not convinced that having a precise type would make it easy to type check common code that processes JSON data, because before accessing any value read from a JSON object, the code needs to use As there are many valid ways of processing JSON data, I think that In order to describe types of JSON values precisely, these features would be useful:
(I started writing a proposal for (2) a while ago but got distracted.) Neither are currently defined in PEP 484. The first one would be easy to specify but potentially tricky to implement. The latter would be tricky to specify and implement, and all the use cases are not clear to me yet. I suggest that we wait until a tool implements one or the other and then we can continue this discussion at a more concrete level. |
I am very tempted to drop this idea. In my own code I use this:
which happens to cover perfectly what I'm doing (even though it sounds like Jukka has found some holes :-). That really doesn't reach the threshold for adding it to a stub file to me (the definition exists in exactly two files). |
I'm fine with closing this if recursive types aren't in the pipeline. Obviously the generic JSON type Guido and I came up with is not a very tight definition and so is of limited value. If the preference is only to worry about tight-fitting type boundaries then this doesn't really make sense. |
OK, I'm closing this, because of a combination of things:
Maybe we can just add the non-tight version to 3.6 and worry about tightening it up if/when we ever implement recursive types. |
I suggest that even if we add the type to the module, we wouldn't use it as the return type of the load functions, at least for now (I discussed this above). |
That's a subtlety I missed. Why is `def json.load(stream) -> Any` better
than `def json.load(stream) -> JSONType`? Where would users add an explicit
annotation that's allowed if it returns Any but not if it returns JSONType?
Or are you talking about the situation where object_hook is used and the
return value may in fact contain other values than the ones mentioned in
Brett's union? That's indeed a good point (though one only appreciated by
the few users who actually use that hook).
|
If it returns If a user wants to do the type check they can add an annotation if the return type is
This, for example, would be rejected if the return type is a union, but would be fine if the return type is
|
OK. you've convinced me that |
I'm fine with tossing this whole idea out. I wasn't sure if typing was trying to have something all the time when someone was willing to specify the type or to only when the type matching was tight. It seems like the latter is how you want to treat types which is fine and makes something as variable and loose as JSON not worth worrying about. |
I think a key requirement is that stubs should not reject code that is in fact correct. Better to accept code that's wrong. Unless of course the correct code is very convoluted, but I don't think that using the hook qualifies as convoluted, and there's just too much code around that reads JSON code and dives in as if it knows what's there, accepting random TypeErrors if the JSON data is wrong. |
@gvanrossum You said that you use
Is it correct? from typing import Dict, List, Union, Any
JSONType = Union[
Dict[str, Any],
List[dict, Any],
] |
@pbryan mypy will enable recursive aliases by default in the next release: python/mypy#13516 |
With the newest versions of Python (
Examples:
mixed-type object
nested lists and dictionaries of strings with a slice hidden to throw off the type checker
The type check of all examples together discovers the slice I hid in the last example:
I'm not sure what MyPy does when data passes through several functions.. |
All major type checkers now support recursive type aliases by default, so this should largely work:
Note that because dict is invariant, you might run into some issues e.g. with
Although this can sometimes result in false negatives, for instance, see python/mypy#13786 (comment) As always, if you return a union type, your callers will have to assert the structure of their JSON in order to type check. If this an issue, return e.g. Given that it is trivial to define these aliases, that as you can see they're not necessarily one-size-fits-all, and that explicit is better than implicit, and that they don't need any type checker special casing, I am not in favour of adding these to CPython's typing.py. However if you like importing stuff, we may ship this and some other useful type definitions in a PyPI package at some point; there also appears to be https://github.com/kevinheavey/jsonalias. If you have opinions about this, please see python/typing_extensions#6.
In order to keep this message visible, I may lock this thread. Please strongly consider posting follow-ups at one of the above locations. Thank you to everyone who contributed support for recursive type aliases to type checkers — cheers and happy typing! |
For each dict that we control properly annotate the type of key and values. For JSON controlled by others (i.e. returned by API)... it's pretty hard! For the moment just annotate all of them via a generic `dict[Any, Any]'. (We can define a custom JSON type annotation as documented by <python/typing#182> but we will still have several problems and at the end we will always return Any.) All of this was pointed out via `mypy --strict .'.
For each dict that we control properly annotate the type of key and values. For JSON controlled by others (i.e. returned by API)... it's pretty hard! For the moment just annotate all of them via a generic `dict[Any, Any]'. (We can define a custom JSON type annotation as documented by <python/typing#182> but we will still have several problems and at the end we will always return Any.) All of this was pointed out via `mypy --strict .'.
JSON is such a common interchange format it might make sense to define it as a specific type.
Not sure if this should go into
typing
or be introduces asjson.JSONType
instead (or if it's even worth it considering the variability of the type).The text was updated successfully, but these errors were encountered: