From 66c050030d2166daebff043ece58be96746a338a Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Mon, 3 Jun 2019 18:17:57 -0500 Subject: [PATCH 01/10] Add manylinux2014 draft --- pep-9999.rst | 317 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 317 insertions(+) create mode 100644 pep-9999.rst diff --git a/pep-9999.rst b/pep-9999.rst new file mode 100644 index 00000000000..efc73b53563 --- /dev/null +++ b/pep-9999.rst @@ -0,0 +1,317 @@ +PEP: 9999 +Title: The manylinux2014 Platform Tag +Version: $Revision$ +Last-Modified: $Date$ +Author: Dustin Ingram +BDFL-Delegate: Paul Moore +Discussions-To: Distutils SIG +Status: Active +Type: Informational +Content-Type: text/x-rst +Created: 29-April-2019 +Post-History: 29-April-2019 + + +Abstract +======== + +This PEP proposes the creation of a ``manylinux2014`` platform tag to +succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also +proposes that PyPI and ``pip`` both be updated to support uploading, +downloading, and installing ``manylinux2014`` distributions on +compatible platforms. + +Rationale +========= + +CentOS 6 is now the oldest supported CentOS release, and will receive +maintenance updates through November 30th, 2020, [1]_ at which point +it will reach end-of-life, and no further updates such as security +patches will be made available. All wheels built under the +``manylinux2010`` images will remain at obsolete versions after that +point. + +Therefore, we propose the continuation of the existing manylinux +standard, and that a new PEP 425-style [2]_ platform tag called +``manylinux2014`` be derived from CentOS 7 and that the ``manylinux`` +toolchain, PyPI, and ``pip`` be updated to support it. + +Similar to how :pep:`571` and :pep:`513` drew allowed shared +libraries and their symbol versions from CentOS 5.11 and CentOS 6, +respectively, a ``manylinux2014`` platform tag will draw its libraries +and symbol versions from CentOS 7, which will reach end-of-life on +June 30th, 2024. [1]_ + +The ``manylinuxYYYY`` pattern has a number of advantages that motivate +continuing with the current status quo: + +- Well-defined Docker images with clearly specified compatible + libraries; +- No need to survey for compatibility issues across multiple releases; +- A single build image and ``auditwheel`` profile per architecture. + +There are also some disadvantages: + +- Requires drafting a new PEP for every new standard; +- Requires adding the new platform tag to installers (e.g., ``pip``); +- Installers are unable to install a platform tag which predates a + given release. + +There are also challenges which would exist for any proposal, +including the time and effort it takes to define, prepare and release +the Docker images and corresponding ``auditwheel`` profiles. These +challenges were experienced in the long rollout period for +``manylinux2010``, which took approximately 1 year from PEP acceptance +to compatible build environment published. [3]_ + +However, if this PEP can be an indicator, the process is now +well-defined and easily repeatable, which should increase the timeline +for rollout of a newer, updated platform tag. + +The ``manylinux2014`` policy +============================ + +The following criteria determine a ``linux`` wheel's eligibility for +the ``manylinux2014`` tag: + +1. The wheel may only contain binary executables and shared objects + compiled for one of the following architectures supported by CentOS + 7, or a CentOS 7 compatible base image (such as ubi7): [4]_ :: + + x86_64 + i686 + aarch64 + armhfp + ppc64 + ppc64le + s390x + + This list adds support for ARM (aarch64, armhfp) and PowerPC + (ppc64, ppc64le) architectures supported by the CentOS Alternative + Architecture Special Interest Group, as well as the IBM Z (s390x) + architecture. [5]_ + +2. The wheel's binary executables or shared objects may not link + against externally-provided libraries except those in the following + list: :: + + libgcc_s.so.1 + libstdc++.so.6 + libm.so.6 + libdl.so.2 + librt.so.1 + libcrypt.so.1 + libc.so.6 + libnsl.so.1 + libutil.so.1 + libpthread.so.0 + libresolv.so.2 + libX11.so.6 + libXext.so.6 + libXrender.so.1 + libICE.so.6 + libSM.so.6 + libGL.so.1 + libgobject-2.0.so.0 + libgthread-2.0.so.0 + libglib-2.0.so.0 + + This list is identical to the externally-provided libraries + allowed for ``manylinux2010``. ``libpythonX.Y`` remains ineligible + for inclusion for the same reasons outlined in :pep:`513`. + + On Debian-based systems, these libraries are provided by the + packages: + + ============ ======================================================= + Package Libraries + ============ ======================================================= + libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, + libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, + libnsl.so.1 + libgcc1 libgcc_s.so.1 + libgl1 libGL.so.1 + libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 + libice6 libICE.so.6 + libsm6 libSM.so.6 + libstdc++6 libstdc++.so.6 + libx11-6 libX11.so.6 + libxext6 libXext.so.6 + libxrender1 libXrender.so.1 + ============ ======================================================= + + On RPM-based systems, they are provided by these packages: + + ============ ======================================================= + Package Libraries + ============ ======================================================= + glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 + glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, + libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, + libc.so.6 + libICE libICE.so.6 + libX11 libX11.so.6 + libXext: libXext.so.6 + libXrender libXrender.so.1 + libgcc: libgcc_s.so.1 + libstdc++ libstdc++.so.6 + mesa libGL.so.1 + ============ ======================================================= + +3. If the wheel contains binary executables or shared objects linked + against any allowed libraries that also export versioned symbols, + they may only depend on the following maximum versions:: + + GLIBC_2.17 + CXXABI_1.3.7 + GLIBCXX_3.4.19 + GCC_4.8.5 + + As an example, ``manylinux2014`` wheels may include binary + artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``, + because this an earlier version than the maximum of ``GLIBC_2.17``. +4. If a wheel is built for any version of CPython 2 or CPython + versions 3.0 up to and including 3.2, it *must* include a CPython + ABI tag indicating its Unicode ABI. A ``manylinux2014`` wheel + built against Python 2, then, must include either the ``cpy27mu`` + tag indicating it was built against an interpreter with the UCS-4 + ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2 + ABI. [6]_ [7]_ +5. A wheel *must not* require the ``PyFPE_jbuf`` symbol. This is + achieved by building it against a Python compiled *without* the + ``--with-fpectl`` ``configure`` flag. + +Compilation of Compliant Wheels +=============================== + +Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014`` +platform tags to ``linux`` wheels built by ``pip wheel`` or +``bdist_wheel`` in a ``manylinux2014`` Docker container. + +Docker Images +------------- + +A ``manylinux2014`` Docker image based on CentOS 7 x86_64 is provided +for building binary ``linux`` wheels that can reliably be converted to +``manylinux2014`` wheels. [8]_ This image comes with a full compiler +suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.5) as well as +the latest releases of Python and ``pip``. + +Compatibility with kernels that lack ``vsyscall`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Similar to CentOS 6, CentOS 7 includes a version of ``glibc`` that +depends on the ``vsyscall`` page. As described in :pep:`571`, this +breaks the assumption that a Docker containers userland is compatible +wtih its host's kernel. Similar to ``manylinux2010``, Docker images +for ``manylinux2014`` require patching ``glibc`` to remove all +dependencies on ``vsyscall`` in the version of ``glibc`` included with +our Docker image. + +Auditwheel +---------- + +The ``auditwheel`` tool has also been updated to produce +``manylinux2014`` wheels. [9]_ Its behavior and purpose are otherwise +unchanged from :pep:`513`. + +Platform Detection for Installers +================================= + +Platforms may define a ``manylinux2014_compatible`` boolean attribute +on the ``_manylinux`` module described in :pep:`513`. A platform is +considered incompatible with ``manylinux2014`` if the attribute is +``False``. + +If the ``_manylinux`` module is not found, or it does not have the +attribute ``manylinux2014_compatible``, tools may fall back to +checking for glibc. If the platform has glibc 2.17 or newer, it is +assumed to be compatible unless the ``_manylinux`` module says +otherwise. + +Specifically, the algorithm we propose is:: + + def is_manylinux2014_compatible(): + # Only Linux, and only supported architectures + from distutils.util import get_platform + + if get_platform() not in [ + "linux-x86_64", + "linux-i686", + "linux-aarch64", + "linux-armhfp", + "linux-ppc64", + "linux-ppc64le", + "linux-s390x", + ]: + return False + + # Check for presence of _manylinux module + try: + import _manylinux + + return bool(_manylinux.manylinux2014_compatible) + except (ImportError, AttributeError): + # Fall through to heuristic check below + pass + + # Check glibc version. CentOS 7 uses glibc 2.17. + # PEP 513 contains an implementation of this function. + return have_compatible_glibc(2, 17) + +Backwards compatibility with ``manylinux2010`` wheels +===================================================== + +As explained in :pep:`513`, the specified symbol versions for +``manylinux1`` allowed libraries constitute an *upper bound*. The +same is true for the symbol versions defined for ``manylinux2014`` in +this PEP. As a result, ``manylinux1`` and ``manylinux2010`` wheels +are considered ``manylinux2014`` wheels. A ``pip`` that recognizes +the ``manylinux2014`` platform tag will thus install ``manylinux2010`` +wheels for ``manylinux2014`` platforms -- even when explicitly set -- +when no ``manylinux2014`` wheels are available. + +PyPI Support +============ + +PyPI should permit wheels containing the ``manylinux2014`` platform +tag to be uploaded in the same way that it permits ``manylinux2010``. +It should not attempt to verify the compatibility of ``manylinux2014`` +wheels. + +References +========== + +.. [1] CentOS Product Specifications + (https://wiki.centos.org/About/Product) +.. [2] PEP 425 -- Compatibility Tags for Built Distributions + (https://www.python.org/dev/peps/pep-0425/) +.. [3] Tracking issue for manylinux2010 rollout + (https://github.com/pypa/manylinux/issues/179) +.. [4] Red Hat Universal Base Image 7 + (https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7) + (https://wiki.centos.org/SpecialInterestGroup/AltArch) +.. [5] The CentOS Alternative Architecture Special Interest Group + (https://wiki.centos.org/SpecialInterestGroup/AltArch) +.. [6] PEP 3149 + https://www.python.org/dev/peps/pep-3149/ +.. [7] SOABI support for Python 2.X and PyPy + https://github.com/pypa/pip/pull/3075 +.. [8] manylinux2014 Docker images + (https://quay.io/repository/pypa/manylinux2014_x86_64) +.. [9] auditwheel + (https://github.com/pypa/auditwheel/) + +Copyright +========= + +This document has been placed into the public domain. + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 94b97edfe024aaf3eab07f6982a0f71787989ed9 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Wed, 10 Jul 2019 23:54:42 +0200 Subject: [PATCH 02/10] Remove libcrypt.so.1 --- pep-9999.rst | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index efc73b53563..9db001bffd5 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -100,7 +100,6 @@ the ``manylinux2014`` tag: libm.so.6 libdl.so.2 librt.so.1 - libcrypt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 @@ -117,8 +116,10 @@ the ``manylinux2014`` tag: libglib-2.0.so.0 This list is identical to the externally-provided libraries - allowed for ``manylinux2010``. ``libpythonX.Y`` remains ineligible - for inclusion for the same reasons outlined in :pep:`513`. + originally allowed for ``manylinux2010``, with one exception: + ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30. + ``libpythonX.Y`` remains ineligible for inclusion for the same + reasons outlined in :pep:`513`. On Debian-based systems, these libraries are provided by the packages: @@ -127,8 +128,7 @@ the ``manylinux2014`` tag: Package Libraries ============ ======================================================= libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, - libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1, - libnsl.so.1 + libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1 libgcc1 libgcc_s.so.1 libgl1 libGL.so.1 libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0 @@ -147,8 +147,7 @@ the ``manylinux2014`` tag: ============ ======================================================= glib2 libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0 glibc libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, - libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, - libc.so.6 + libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6 libICE libICE.so.6 libX11 libX11.so.6 libXext: libXext.so.6 From 3ebb4a2117e2d468dd86325272af7b6b86e3f81a Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Thu, 11 Jul 2019 00:13:25 +0200 Subject: [PATCH 03/10] Consolidate email addresses across PEPs --- pep-0566.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0566.rst b/pep-0566.rst index 74924e04230..a3aa46e399d 100644 --- a/pep-0566.rst +++ b/pep-0566.rst @@ -2,7 +2,7 @@ PEP: 566 Title: Metadata for Python Software Packages 2.1 Version: $Revision$ Last-Modified: $Date$ -Author: Dustin Ingram +Author: Dustin Ingram BDFL-Delegate: Daniel Holth Discussions-To: distutils-sig Status: Final From d4d730488e6713096f0a10b9248395b7504f3941 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Thu, 11 Jul 2019 00:54:48 +0200 Subject: [PATCH 04/10] Update Discussions-To to point to discourse thread --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 9db001bffd5..7a5c3adcccc 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -4,7 +4,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Dustin Ingram BDFL-Delegate: Paul Moore -Discussions-To: Distutils SIG +Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ Status: Active Type: Informational Content-Type: text/x-rst From df1f13db679dc87b9cf59c64dfeb01c425e7b543 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Thu, 18 Jul 2019 11:50:41 -0500 Subject: [PATCH 05/10] Update copyright --- pep-9999.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 7a5c3adcccc..1109fe1094d 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -304,7 +304,8 @@ References Copyright ========= -This document has been placed into the public domain. +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: From d8209f3e101f6b04c7b6350c08ac51a672316bc3 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Fri, 19 Jul 2019 11:25:36 -0500 Subject: [PATCH 06/10] Use future tense --- pep-9999.rst | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 1109fe1094d..dca2eeb92e1 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -190,11 +190,11 @@ platform tags to ``linux`` wheels built by ``pip wheel`` or Docker Images ------------- -A ``manylinux2014`` Docker image based on CentOS 7 x86_64 is provided -for building binary ``linux`` wheels that can reliably be converted to -``manylinux2014`` wheels. [8]_ This image comes with a full compiler -suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.5) as well as -the latest releases of Python and ``pip``. +A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be +provided for building binary ``linux`` wheels that can reliably be +converted to ``manylinux2014`` wheels. This image will come with a +full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` +4.8.5) as well as the latest releases of Python and ``pip``. Compatibility with kernels that lack ``vsyscall`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -210,9 +210,9 @@ our Docker image. Auditwheel ---------- -The ``auditwheel`` tool has also been updated to produce -``manylinux2014`` wheels. [9]_ Its behavior and purpose are otherwise -unchanged from :pep:`513`. +The ``auditwheel`` tool will also be updated to produce +``manylinux2014`` wheels. [8]_ Its behavior and purpose will be +otherwise unchanged from :pep:`513`. Platform Detection for Installers ================================= @@ -296,9 +296,7 @@ References https://www.python.org/dev/peps/pep-3149/ .. [7] SOABI support for Python 2.X and PyPy https://github.com/pypa/pip/pull/3075 -.. [8] manylinux2014 Docker images - (https://quay.io/repository/pypa/manylinux2014_x86_64) -.. [9] auditwheel +.. [8] auditwheel (https://github.com/pypa/auditwheel/) Copyright From 9d65b9e6a2bbbf129f4878eb507980b0a8f6e360 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Fri, 19 Jul 2019 11:29:47 -0500 Subject: [PATCH 07/10] Status is 'Draft' --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index dca2eeb92e1..aa18af6468d 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -5,7 +5,7 @@ Last-Modified: $Date$ Author: Dustin Ingram BDFL-Delegate: Paul Moore Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ -Status: Active +Status: Draft Type: Informational Content-Type: text/x-rst Created: 29-April-2019 From 6897708777c29d1ade2d587760005217e5946b2a Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Fri, 19 Jul 2019 17:31:17 -0500 Subject: [PATCH 08/10] Add Paul Moore as sponsor --- pep-9999.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/pep-9999.rst b/pep-9999.rst index aa18af6468d..c1351d5e07d 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -4,6 +4,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Dustin Ingram BDFL-Delegate: Paul Moore +Sponsor: Paul Moore Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ Status: Draft Type: Informational From ed9fecb713a1ee0b4e837e751e40a4d29faaf4af Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sat, 20 Jul 2019 21:23:23 +1000 Subject: [PATCH 09/10] Update and rename pep-9999.rst to pep-0599.rst --- pep-9999.rst => pep-0599.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename pep-9999.rst => pep-0599.rst (99%) diff --git a/pep-9999.rst b/pep-0599.rst similarity index 99% rename from pep-9999.rst rename to pep-0599.rst index c1351d5e07d..7ea82d6298e 100644 --- a/pep-9999.rst +++ b/pep-0599.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 599 Title: The manylinux2014 Platform Tag Version: $Revision$ Last-Modified: $Date$ From c2436fe74c34c89da40314ddcd1d78cc1a830cb8 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sat, 20 Jul 2019 21:27:41 +1000 Subject: [PATCH 10/10] Fix header order --- pep-0599.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0599.rst b/pep-0599.rst index 7ea82d6298e..59d83cc6d4e 100644 --- a/pep-0599.rst +++ b/pep-0599.rst @@ -3,8 +3,8 @@ Title: The manylinux2014 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Dustin Ingram -BDFL-Delegate: Paul Moore Sponsor: Paul Moore +BDFL-Delegate: Paul Moore Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/ Status: Draft Type: Informational