Support 3.7

[Mattwmaster58]

Currently python 3.7 is unsupported. Is it to early to introduce support right now? @ethanhs seems to be making progress, not sure if it's stable.

[emmatyping]

I am working on it. It is not stable yet. I need to finish adding type comment parsing and feature levels and port it to all the supported Python versions (3.4-3.7) and platforms (Linux, macOS, Windows). It is going to miss mypy 0.600, but I expect it will make it into 0.610.

[ddfisher]

In general, my preference is to backport new syntax prior to full release (if support pre-release is desired), and then to do a rewrite "from scratch" on top of the new Python AST code after the official Python release has happened. That said, I think it's rare for the AST module to change much when Python is in beta, so I don't feel too strongly about this.

@ethanhs which approach are you taking now? (And have you seen the update doc? It has some useful info.)

[emmatyping]

@ddfisher I've started a full from scratch update following your guidelines (also the guide is super useful, thanks for that!). You can see my progress here: https://github.com/ethanhs/typed_ast/tree/3.7.

But since there are still 2 months till 3.7 final, you might be right that sticking to backporting makes sense in the short term. Once RC1 is released, no source changes should be made to CPython so that would be a good point to start the from scratch version.

One change that I think would be quite simple to backport is my change here: https://github.com/python/cpython/pull/6460/files#diff-4d35cf8992b795c5e97e9c8b6167cb34 which will fix #50 (it is actually really simple thankfully!).

[JelleZijlstra]

One issue here for typeshed is that 3.7 makes async and await into full keywords, but typeshed's stubs for asyncio (at https://github.com/python/typeshed/blob/master/stdlib/3.4/asyncio/tasks.pyi#L26) use async as an identifier, because asyncio.async used to be an alias for asyncio.ensure_future. This means that parsing typeshed with the 3.7 AST will fail without some sort of workaround.

[gvanrossum]

I propose to just remove that alias.

[gvanrossum]

Note that Astroid (the parsing library used by PyLint and other tools) has switched to typed_ast: pylint-dev/astroid#317 (comment). This does mean we have a responsibility to keep the API stable and we should probably test with Astroid somehow too.

[bharel]

3.7.0 is now stable. @ethanhs, how's your branch doing? 😃

[emmatyping]

@bharel I am going to work on that today actually. I will probably start from scratch again since I want things to be clean, but hopefully I will be able to make some good progress based on previous work. I've started writing cross platform Python scripts to automate much of it.

[ddfisher]

Thanks for working on that, @ethanhs!

We need to decide if we want to support async and await as non-keyword identifiers for people trying to parse Python 3.6 and below (and, if so, how). My inclination is that we need this support, especially because astroid now depends on us.

As for how we might go about doing this, I see two options:

1. Keep the current ast3 parsing code and use it to parse Python 3.6 and below (when requested by the feature_version argument to parse.) This means the structure of the AST returned by parse could eventually differ by feature_version, which is not great.
2. Undo the changes in bpo-30406: Make async and await proper keywords cpython#1669 in typed_ast's parser to bring back the Python 3.6 behavior and add an explicit check to disallow the use of async and await as identifiers in Python 3.7 and above.

I think option 2 is the clear winner here.

[emmatyping]

Yes, it seems to me 2 is the best option. Since the only change to the Grammar is making async/await keywords, and the ASDLs are identitcal, perhaps we should just "backport" the 3.7 changes onto the current branch? (Save python/cpython#1669).

Also ccing @PCManticore from asteriod who might be interested in this.

[graingert]

Also the new Python 3.7 AST does constant folding, will this be in typed-ast?

[edwardcwang]

Would fixing this address python/mypy#5431?

[Mattwmaster58]

@edwardcwang From that same thread:

This is probably an issue with typed-ast rather than mypy.

and

The error is reported by typed_ast and it should be reported there.

[edwardcwang]

Thanks for the pointer. Sorry for being unclear - I was unsure whether or not filing a new issue would be creating a duplicate of this issue.

[bsandrow]

Is this still being worked on? Just curious as the last comment from someone actually working on it is all the way back in June. Thanks in advance for all of the hard work. ^‿^

[gvanrossum]

I don't think it is being worked on right now. This should be done, and the core mypy team at Dropbox ought to make time for it. It is not a fun task. (See also https://bugs.python.org/issue24119 and https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377.)

[JukkaL]

The Dropbox mypy team should be able to work on this in early 2019.

[gvanrossum]

See also this discussion: https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/25

Looks like we may be able to merge typed_ast back into CPython (it would seem to reduce future maintenance, though it doesn't reduce the work for the current port to 3.7 or 3.8).

[gvanrossum]

This is urgent now. I am planning to attempt supporting 3.7 using the steps outlined in update_process.py. I am starting from scratch just so I get a feel for the whole process. Once I have this done I will work on incorporating as many features as possible into the CPython master (which is slated for 3.8).

@ethanhs, do you have any advice on how to do the 3.7 version?

[emmatyping]

@gvanrossum there are a few bumps along the way I ran into, but nothing major. It seems some items were moved to internal headers, so I had to copy those out. Also the update scripts only work on Mac, so I was in the process of re-writing them in Python, but that shouldn't be an issue for you. I agree we should probably undo the changes in python/cpython#1669 and make them conditionally keywords, but I didn't get that far when I was working on this.

Also, I'm happy to make sure things run well on Windows.

Also, do you want to support Python 3.4 with this newer typed_ast? I previously had to make some changes to get the compiler used for Python 3.4 to play nice. Since 3.4 is EOL'ing in 2 months I thought I'd ask.

[gvanrossum]

Agreed that the update scripts are pretty one-off, but I'm not sure they're worth it spending time on. I am now on step 4.5 in update_process.md. Agreed on the async/await keywords, or at least that some kind of hack is needed. I'd be happy to drop support for 3.4 (especially if it will be possible for mypy to use either the old or the new typed_ast, but I don't know if I can pull that off).

[msullivan]

Fixed by #78