Use modular typeshed (unbundle third-party package stubs) (#9973)

Support the new structure of typeshed (python/typeshed#2491) and only
bundle stdlib stubs with mypy. 

Most stubs for third-party packages now need to be installed using pip (for 
example, `pip install types-requests`). Add stubs for `typing_extensions` 
and `mypy_extensions` as mypy dependencies, since these are 
needed for basic operation.

Suggest a pip command to run if we encounter known missing stubs.

Add `--install-types` option that installs all missing stub packages. This
can be used as `mypy --install-types` to install missing stub packages from
the previous mypy run (no need to provide a full mypy command line).

This also replaces the typeshed git submodule with a partial copy of the
typeshed that only includes stdlib stubs. 

Add a script to sync stubs from typeshed (`misc/sync-typeshed.py`). 
This is still incomplete since typeshed hasn't actually migrated to the new 
structure yet.

Work towards #9971.

.gitignore

@@ -2,7 +2,6 @@ build/
 __pycache__
 *.py[cod]
 *~
-@*
 /build
 /env*/
 docs/build/

.travis.yml

@@ -93,7 +93,7 @@ install:
 # means that tox picks up the mypy from the source directories instead of
 # the version it installed into a venv. This is also *why* we need to do this,
 # since if we arranged for tox to build with mypyc, pytest wouldn't use it.
-- if [[ $TEST_MYPYC == 1 ]]; then pip install -r mypy-requirements.txt; CC=clang MYPYC_OPT_LEVEL=0 python3 setup.py --use-mypyc build_ext --inplace; fi
+- if [[ $TEST_MYPYC == 1 ]]; then pip install -r test-requirements.txt; CC=clang MYPYC_OPT_LEVEL=0 python3 setup.py --use-mypyc build_ext --inplace; fi
 
 script:
 - tox --skip-pkg-install -- $EXTRA_ARGS

docs/source/command_line.rst

@@ -743,12 +743,14 @@ in developing or debugging mypy internals.
 
 .. option:: --custom-typeshed-dir DIR
 
-    This flag specifies the directory where mypy looks for typeshed
+    This flag specifies the directory where mypy looks for standard library typeshed
     stubs, instead of the typeshed that ships with mypy.  This is
     primarily intended to make it easier to test typeshed changes before
     submitting them upstream, but also allows you to use a forked version of
     typeshed.
 
+    Note that this doesn't affect third-party library stubs.
+
 .. _warn-incomplete-stub:
 
 .. option:: --warn-incomplete-stub
@@ -841,6 +843,26 @@ format into the specified directory.
 Miscellaneous
 *************
 
+.. option:: --install-types
+
+    This flag causes mypy to install known missing stub packages for
+    third-party libraries using pip.  It will display the pip command
+    line to run, and expects a confirmation before installing
+    anything.
+
+    If you use this option without providing any files or modules to
+    type check, mypy will install stub packages suggested during the
+    previous mypy run. If there are files or modules to type check,
+    mypy first type checks those, and proposes to install missing
+    stubs at the end of the run, but only if any missing modules were
+    detected.
+
+    .. note::
+
+        This is new in mypy 0.900. Previous mypy versions included a
+        selection of third-party package stubs, instead of having them
+        installed separately.
+
 .. option:: --junit-xml JUNIT_XML
 
     Causes mypy to generate a JUnit XML test result document with

docs/source/getting_started.rst

@@ -329,10 +329,11 @@ Library stubs and typeshed
 Mypy uses library *stubs* to type check code interacting with library
 modules, including the Python standard library. A library stub defines
 a skeleton of the public interface of the library, including classes,
-variables and functions, and their types. Mypy ships with stubs from
-the `typeshed <https://github.com/python/typeshed>`_ project, which
-contains library stubs for the Python builtins, the standard library,
-and selected third-party packages.
+variables and functions, and their types. Mypy ships with stubs for
+the standard library from the `typeshed
+<https://github.com/python/typeshed>`_ project, which contains library
+stubs for the Python builtins, the standard library, and selected
+third-party packages.
 
 For example, consider this code:
 
@@ -344,11 +345,40 @@ Without a library stub, mypy would have no way of inferring the type of ``x``
 and checking that the argument to :py:func:`chr` has a valid type.
 
 Mypy complains if it can't find a stub (or a real module) for a
-library module that you import. Some modules ship with stubs that mypy
-can automatically find, or you can install a 3rd party module with
-additional stubs (see :ref:`installed-packages` for details).  You can
-also :ref:`create stubs <stub-files>` easily. We discuss ways of
-silencing complaints about missing stubs in :ref:`ignore-missing-imports`.
+library module that you import. Some modules ship with stubs or inline
+annotations that mypy can automatically find, or you can install
+additional stubs using pip (see :ref:`fix-missing-imports` and
+:ref:`installed-packages` for the details). For example, you can install
+the stubs for the ``requests`` package like this:
+
+.. code-block::
+
+  python3 -m pip install types-requests
+
+The stubs are usually packaged in a distribution named
+``types-<distribution>``.  Note that the distribution name may be
+different from the name of the package that you import. For example,
+``types-PyYAML`` contains stubs for the ``yaml`` package. Mypy can
+often suggest the name of the stub distribution:
+
+.. code-block:: text
+
+  prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8)
+  prog.py:1: note: Hint: "python3 -m pip install types-PyYAML"
+  ...
+
+.. note::
+
+   Starting in mypy 0.900, most third-party package stubs must be
+   installed explicitly. This decouples mypy and stub versioning,
+   allowing stubs to updated without updating mypy. This also allows
+   stubs not originally included with mypy to be installed. Earlier
+   mypy versions included a fixed set of stubs for third-party
+   packages.
+
+You can also :ref:`create
+stubs <stub-files>` easily. We discuss ways of silencing complaints
+about missing stubs in :ref:`ignore-missing-imports`.
 
 Configuring mypy
 ****************

docs/source/installed_packages.rst

@@ -3,54 +3,92 @@
 Using installed packages
 ========================
 
-:pep:`561` specifies how to mark a package as supporting type checking.
-Below is a summary of how to create PEP 561 compatible packages and have
-mypy use them in type checking.
-
-Using PEP 561 compatible packages with mypy
-*******************************************
-
-Generally, you do not need to do anything to use installed packages that
-support typing for the Python executable used to run mypy. Note that most
-packages do not support typing. Packages that do support typing should be
-automatically picked up by mypy and used for type checking.
-
-By default, mypy searches for packages installed for the Python executable
-running mypy. It is highly unlikely you want this situation if you have
-installed typed packages in another Python's package directory.
-
-Generally, you can use the :option:`--python-version <mypy --python-version>` flag and mypy will try to find
-the correct package directory. If that fails, you can use the
-:option:`--python-executable <mypy --python-executable>` flag to point to the exact executable, and mypy will
-find packages installed for that Python executable.
-
-Note that mypy does not support some more advanced import features, such as zip
-imports and custom import hooks.
-
-If you do not want to use typed packages, use the :option:`--no-site-packages <mypy --no-site-packages>` flag
-to disable searching.
-
-Note that stub-only packages (defined in :pep:`PEP 561: Stub-only Packages
-<561#stub-only-packages>`) cannot be used with ``MYPYPATH``. If you want mypy
-to find the package, it must be installed. For a package ``foo``, the name of
-the stub-only package (``foo-stubs``) is not a legal package name, so mypy
-will not find it, unless it is installed.
-
-Making PEP 561 compatible packages
-**********************************
-
-:pep:`561` notes three main ways to distribute type information. The first is a
-package that has only inline type annotations in the code itself. The second is
-a package that ships :ref:`stub files <stub-files>` with type information
-alongside the runtime code. The third method, also known as a "stub only
-package" is a package that ships type information for a package separately as
-stub files.
-
-If you would like to publish a library package to a package repository (e.g.
-PyPI) for either internal or external use in type checking, packages that
-supply type information via type comments or annotations in the code should put
-a ``py.typed`` file in their package directory. For example, with a directory
-structure as follows
+Packages installed with pip can declare that they support type
+checking. For example, the `aiohttp
+<https://docs.aiohttp.org/en/stable/>`_ package has built-in support
+for type checking.
+
+Packages can also provide stubs for a library. For example,
+``types-requests`` is a stub-only package that provides stubs for the
+`requests <https://requests.readthedocs.io/en/master/>`_ package.
+Stub packages are usually published from `typeshed
+<https://github.com/python/typeshed>`_, a shared repository for Python
+library stubs, and have a name of form ``types-<library>``. Note that
+many stub packages are not maintained by the original maintainers of
+the package.
+
+The sections below explain how mypy can use these packages, and how
+you can create such packages.
+
+.. note::
+
+   :pep:`561` specifies how a package can declare that it supports
+   type checking.
+
+Using installed packages with mypy (PEP 561)
+********************************************
+
+Typically mypy will automatically find and use installed packages that
+support type checking or provide stubs. This requires that you install
+the packages in the Python environment that you use to run mypy.  As
+many packages don't support type checking yet, you may also have to
+install a separate stub package, usually named
+``types-<library>``. (See :ref:`fix-missing-imports` for how to deal
+with libraries that don't support type checking and are also missing
+stubs.)
+
+If you have installed typed packages in another Python installation or
+environment, mypy won't automatically find them. One option is to
+install another copy of those packages in the environment in which you
+use to run mypy. Alternatively, you can use the
+:option:`--python-executable <mypy --python-executable>` flag to point
+to the target Python executable, and mypy will find packages installed
+for that Python executable.
+
+Note that mypy does not support some more advanced import features,
+such as zip imports and custom import hooks.
+
+If you don't want to use installed packages that provide type
+information at all, use the :option:`--no-site-packages <mypy
+--no-site-packages>` flag to disable searching for installed packages.
+
+Note that stub-only packages cannot be used with ``MYPYPATH``. If you
+want mypy to find the package, it must be installed. For a package
+``foo``, the name of the stub-only package (``foo-stubs``) is not a
+legal package name, so mypy will not find it, unless it is installed
+(see :pep:`PEP 561: Stub-only Packages <561#stub-only-packages>` for
+more information).
+
+Creating PEP 561 compatible packages
+************************************
+
+.. note::
+
+  You can generally ignore this section unless you maintain a package on
+  PyPI, or want to publish type information for an existing PyPI
+  package.
+
+:pep:`561` describes three main ways to distribute type
+information:
+
+1. A package has inline type annotations in the Python implementation.
+
+2. A package ships :ref:`stub files <stub-files>` with type
+   information alongside the Python implementation.
+
+3. A package ships type information for another package separately as
+   stub files (also known as a "stub-only package").
+
+If you want to create a stub-only package for an existing library, the
+simplest way is to contribute stubs to the `typeshed
+<https://github.com/python/typeshed>`_ repository, and a stub package
+will automatically be uploaded to PyPI.
+
+If you would like to publish a library package to a package repository
+yourself (e.g. on PyPI) for either internal or external use in type
+checking, packages that supply type information via type comments or
+annotations in the code should put a ``py.typed`` file in their
+package directory. For example, here is a typical directory structure:
 
 .. code-block:: text
 
@@ -60,7 +98,7 @@ structure as follows
         lib.py
         py.typed
 
-the ``setup.py`` might look like
+The ``setup.py`` file could look like this:
 
 .. code-block:: python
 
@@ -80,7 +118,7 @@ the ``setup.py`` might look like
    ``setup()``, or mypy will not be able to find the installed package.
 
 Some packages have a mix of stub files and runtime files. These packages also
-require a ``py.typed`` file. An example can be seen below
+require a ``py.typed`` file. An example can be seen below:
 
 .. code-block:: text
 
@@ -91,7 +129,7 @@ require a ``py.typed`` file. An example can be seen below
         lib.pyi
         py.typed
 
-the ``setup.py`` might look like:
+The ``setup.py`` file might look like this:
 
 .. code-block:: python
 
@@ -121,7 +159,7 @@ had stubs for ``package_c``, we might do the following:
         __init__.pyi
         lib.pyi
 
-the ``setup.py`` might look like:
+The ``setup.py`` might look like this:
 
 .. code-block:: python
 
@@ -134,3 +172,8 @@ the ``setup.py`` might look like:
         package_data={"package_c-stubs": ["__init__.pyi", "lib.pyi"]},
         packages=["package_c-stubs"]
     )
+
+If you have separate stubs for Python 2 and Python 3, you can place
+the Python 2 stubs in a directory with the suffix ``-python2-stubs``.
+We recommend that Python 2 and Python 3 stubs are bundled together for
+simplicity, instead of distributing them separately.

docs/source/running_mypy.rst

@@ -127,17 +127,21 @@ The third outcome is what mypy will do in the ideal case. The following
 sections will discuss what to do in the other two cases.
 
 .. _ignore-missing-imports:
+.. _fix-missing-imports:
 
 Missing imports
 ***************
 
-When you import a module, mypy may report that it is unable to
-follow the import.
+When you import a module, mypy may report that it is unable to follow
+the import.
 
-This can cause errors that look like the following::
+This can cause errors that look like the following:
 
-    main.py:1: error: Skipping analyzing 'django': found module but no type hints or library stubs
-    main.py:2: error: Cannot find implementation or library stub for module named 'this_module_does_not_exist'
+.. code-block:: text
+
+    main.py:1: error: Library stubs not installed for "requests" (or incompatible with Python 3.8)
+    main.py:2: error: Skipping analyzing 'django': found module but no type hints or library stubs
+    main.py:3: error: Cannot find implementation or library stub for module named "this_module_does_not_exist"
 
 If you get any of these errors on an import, mypy will assume the type of that
 module is ``Any``, the dynamic type. This means attempting to access any
@@ -153,6 +157,36 @@ attribute of the module will automatically succeed:
 
 The next sections describe what each error means and recommended next steps.
 
+Library stubs not installed
+---------------------------
+
+If mypy can't find stubs for a third-party library, and it knows that stubs exist for
+the library, you will get a message like this:
+
+.. code-block:: text
+
+    main.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8)
+    main.py:1: note: Hint: "python3 -m pip install types-PyYAML"
+    main.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
+
+You can resolve the issue by running the suggested pip command or
+commands. Alternatively, you can use :option:`--install-types <mypy
+--install-types>` to install all known missing stubs:
+
+.. code-block:: text
+
+    mypy --install-types
+
+This installs any stub packages that were suggested in the previous
+mypy run. You can also use your normal mypy command line with the
+extra :option:`--install-types <mypy --install-types>` option to
+install missing stubs at the end of the run (if any were found).
+
+You can also get this message if the stubs only support Python 3 and
+your target Python version is Python 2, or vice versa. In this case
+follow instructions in
+:ref:`missing-type-hints-for-third-party-library`.
+
 .. _missing-type-hints-for-third-party-library:
 
 Missing type hints for third party library