PEP 646: Update draft (#1856)

This is the more-or-less final version now. I'll do one last read-through to confirm everything's consistent, then post in typing-sig to confirm it's ready for the Steering Committee.

Semantic changes:
* Remove support for type variable tuples in `Union`. Apparently the implementation would be tricky in Pyre because of special-casing around `Union`. That could be evidence that it would be tricky in Mypy and pytype. Since we don't have a specific use-case in mind, I think this is fine.
* Support concatenation in `Callable`. Pradeep pointed ou that `ParamSpec` doesn't cover all potential use-cases because it doesn't support concatenating suffixes (in turn because `ParamSpec`s can contain keyword arguments, and concatenation is positional).

Readability changes:
* Remove the section on a full `Array` example. We've moved the important thing here - an explicit confirmation that `Array` can be generic in both datatype and shape - to the beginning of the PEP to make it more obvious. We've removed the `Ndim` thing after someone pointed out that e.g. `Ndim[Literal[2]]` would be incompatible with `Shape[Any, Any]`; I just don't think it's going to work. All that's left is the aliases, which we've moved into the Aliases section itself.

Other changes:
* Fill out Backwards Compatibility section.
* Reference initial CPython implementation.

pep-0646.rst

@@ -476,38 +476,14 @@ Type variable tuples can also be used in the arguments section of a
     Process(target=func, args=(0, 'foo'))  # Valid
     Process(target=func, args=('foo', 0))  # Error
 
-However, note that as of this PEP, if a type variable tuple does appear in
-the arguments section of a ``Callable``, it must appear alone.
-That is, `Type Concatenation`_ is not supported in the context of ``Callable``.
-(Use cases where this might otherwise be desirable are likely covered through use
-of either a ``ParamSpec`` from PEP 612, or a type variable tuple in the ``__call__``
-signature of a callback protocol from PEP 544.)
-
-Type Variable Tuples with ``Union``
------------------------------------
-
-Type variable tuples can also be used with ``Union``:
-
-::
-    
-    def f(*args: *Ts) -> Union[*Ts]:
-        return random.choice(args)
-
-    f(1, 'foo')  # Inferred type is Union[int, str]
-
-Here, if the type variable tuple is empty (e.g. if we had ``*args: *Ts``
-and didn't pass any arguments), the type checker should
-raise an error on the ``Union`` (matching the behaviour of ``Union``
-at runtime, which requires at least one type argument).
-
-Other types can also be included in the ``Union``:
+Other types and normal type variables can also be prefixed/suffixed
+to the type variable tuple:
 
 ::
 
-    def f(*args :*Ts) -> Union[int, str, *Ts]: ...
+    T = TypeVar('T')
 
-However, note that as elsewhere, only a single type variable tuple
-may occur within a ``Union``.
+    def foo(f: Callable[[int, *Ts, T], Tuple[T, *Ts]]): ...
 
 Aliases
 -------
@@ -518,33 +494,65 @@ a similar way to regular type variables:
 ::
 
     IntTuple = Tuple[int, *Ts]
+    NamedArray = Tuple[str, Array[*Ts]]
+
     IntTuple[float, bool]  # Equivalent to Tuple[int, float, bool]
+    NamedArray[Height]     # Equivalent to Tuple[str, Array[Height]]
 
 As this example shows, all type parameters passed to the alias are
-bound to the type variable tuple. If no type parameters are given,
-or if an explicitly empty list of type parameters are given,
-type variable tuple in the alias is simply ignored:
+bound to the type variable tuple.
+
+Importantly for our ``Array`` example (see `Summary Examples`_), this
+allows us to define convenience aliases for arrays of a fixed shape
+or datatype:
 
 ::
 
-    # Both equivalent to Tuple[int]
-    IntTuple
-    IntTuple[()]
+    Shape = TypeVarTuple('Shape')
+    DType = TypeVar('DType')
+    class Array(Generic[DType, *Shape]):
 
-Normal ``TypeVar`` instances can also be used in such aliases:
+    # E.g. Float32Array[Height, Width, Channels]
+    Float32Array = Array[np.float32, *Shape]
+
+    # E.g. Array1D[np.uint8]
+    Array1D = Array[DType, Any]
+
+If an explicitly empty type parameter list is given, the type variable
+tuple in the alias is set empty:
 
 ::
 
-    T = TypeVar('T')
-    Foo = Tuple[T, *Ts]
+    IntTuple[()]    # Equivalent to Tuple[int]
+    NamedArray[()]  # Equivalent to Tuple[str, Array[()]]
+
+If the type parameter list is omitted entirely, the alias is
+compatible with arbitrary type parameters:
+
+::
 
-    # T is bound to `int`; Ts is bound to `bool, str`
-    Foo[int, bool, str]
+    def takes_float_array_of_any_shape(x: Float32Array): ...
+    x: Float32Array[Height, Width] = Array()
+    takes_float_array_of_any_shape(x)  # Valid
 
-Note that the same rules for `Type Concatenation`_ apply for aliases.
-In particular, only one ``TypeVarTuple`` may occur within an alias,
-and the ``TypeVarTuple`` must be at the end of the alias.
+    def takes_float_array_with_specific_shape(y: Float32Array[Height, Width]): ...
+    y: Float32Array = Array()
+    takes_float_array_with_specific_shape(y)  # Valid
 
+Normal ``TypeVar`` instances can also be used in such aliases:
+
+::
+
+    T = TypeVar('T')
+    Foo = Tuple[*Ts, T]
+
+    # Ts bound to Tuple[int], T to int
+    Foo[str, int]
+    # Ts bound to Tuple[()], T to int
+    Foo[int]
+    # T bound to Any, Ts to an arbitrary number of Any
+    Foo
+   
 Overloads for Accessing Individual Types
 ----------------------------------------
 
@@ -574,51 +582,6 @@ overloads for each possible rank is, of course, a rather cumbersome
 solution. However, it's the best we can do without additional type
 manipulation mechanisms, which are beyond the scope of this PEP.)
 
-An Ideal Array Type: One Possible Example
-=========================================
-
-Type variable tuples allow us to make significant progress on the
-typing of arrays. However, the array class we have sketched
-out in this PEP is still missing some desirable features. [#typing-ideas]_
-
-The most crucial feature missing is the ability to specify
-the data type (e.g. ``np.float32`` or ``np.uint8``). This is important
-because some numerical computing libraries will silently cast
-types, which can easily lead to hard-to-diagnose bugs.
-
-Additionally, it might be useful to be able to specify the rank
-instead of the full shape. This could be useful for cases where
-axes don't have obvious semantic meaning like 'height' or 'width',
-or where the array is very high-dimensional and writing out all
-the axes would be too verbose.
-
-Here is one possible example of how these features might be implemented
-in a complete array type.
-
-::
-
-    # E.g. Ndim[Literal[3]]
-    Integer = TypeVar('Integer')
-    class Ndim(Generic[Integer]): ...
-
-    # E.g. Shape[Height, Width]
-    # (Where Height and Width are custom types)
-    Axes = TypeVarTuple('Axes')
-    class Shape(Generic[*Axes]): ...
-
-    DataType = TypeVar('DataType')
-    ShapeType = TypeVar('ShapeType', Ndim, Shape)
-
-    # The most verbose type
-    # E.g. Array[np.float32, Ndim[Literal[3]]
-    #      Array[np.uint8, Shape[Height, Width, Channels]]
-    class Array(Generic[DataType, ShapeType]): ...
-
-    # Type aliases for less verbosity
-    # E.g. Float32Array[Height, Width, Channels]
-    Float32Array = Array[np.float32, Shape[*Axes]]
-    # E.g. Array1D[np.uint8]
-    Array1D = Array[DataType, Ndim[Literal[1]]]
 
 Rationale and Rejected Ideas
 ============================
@@ -710,20 +673,32 @@ in how much of their code they wish to annotate, and to enable compatibility
 between old unannotated code and new versions of libraries which do use
 these type annotations.
 
+
 Backwards Compatibility
 =======================
 
-TODO
+The ``Unpack`` version of the PEP should be back-portable to previous
+versions of Python.
 
-* ``Tuple`` needs to be upgraded to support parameterization with a
-  type variable tuple.
+Gradual typing is enabled by the fact that unparameterised variadic classes
+are compatible with an arbitrary number of type parameters. This means
+that if existing classes are made generic, a) all existing (unparameterised)
+uses of the class will still work, and b) parameterised and unparameterised
+versions of the class can be used together (relevant if, for example, library
+code is updated to use parameters while user code is not, or vice-versa).
 
 
 Reference Implementation
 ========================
 
-Two reference implementations exist: one in Pyre, as of TODO, and one in
-Pyright, as of v1.1.108.
+Two reference implementations of type-checking functionality exist:
+one in Pyre, as of TODO, and one in Pyright, as of v1.1.108.
+
+A preliminary implementation of the ``Unpack`` version of the PEP in CPython
+is available in `cpython/23527`_. A preliminary version of the version
+using the star operator, based on an early implementation of PEP 637,
+is also available at `mrahtz/cpython/pep637+646`_.
+
 
 Footnotes
 ==========
@@ -758,6 +733,10 @@ References
 
 .. [#arbitrary_len] Discussion on Python typing-sig mailing list: https://mail.python.org/archives/list/[email protected]/thread/SQVTQYWIOI4TIO7NNBTFFWFMSMS2TA4J/
 
+.. _cpython/23527: https://github.com/python/cpython/pull/24527
+
+.. _mrahtz/cpython/pep637+646: https://github.com/mrahtz/cpython/tree/pep637%2B646
+
 
 Acknowledgements
 ================