PEP 730: Revisions and clarifications following community discussion

peps/pep-0730.rst

@@ -124,7 +124,7 @@ Distribution
 ------------
 
 Adding iOS as a Tier 3 platform only requires adding support for compiling an
-iOS-compatibile code with an unpatched CPython code checkout. It does not
+iOS-compatible code with an unpatched CPython code checkout. It does not
 require production of officially distributed iOS artefacts for use by end-users.
 
 If/when iOS is updated to Tier 2 or 1 support, there should be a process for
@@ -176,30 +176,27 @@ devices.
 * ``"iphonesimulator-arm64"`` for ARM64 simulators
 * ``"iphonesimulator-x86_64"`` for x86_64 simulators
 
-``sys.implementation`` will also have an additional attribute - ``_simulator`` -
-storing a Boolean that is ``True`` if the device running the app is a simulator.
-This attribute would not exist on non-iOS platforms.
-
 ``platform``
 ''''''''''''
 
-Platform will be used as the primary mechanism for retrieving OS and device
-details.
-
-* ``platform.system()`` - ``"iOS"``
+Platform be modified to support returning iOS-specific details. Most of the
+values returned by the platform module will match those returned by
+``os.uname()``, with the exception of:
 
-* ``platform.node()`` - the user-provided name of the device, as returned by the
-  ``[[UIDevice currentDevice] systemName]`` system call (e.g.,
-  ``"Janes-iPhone"``). For simulated devices, this will be the name of the
-  development computer running the simulator.
+* ``platform.system()`` - ``"iOS"``, instead of the default ``"Darwin"``
 
-* ``platform.release()`` - the iOS version number, as a string (e.g., ``"16.6.1"``)
+* ``platform.release()`` - the iOS version number, as a string (e.g.,
+  ``"16.6.1"``), instead of the Darwin kernel version.
 
-* ``platform.machine()`` - The device model returned by ``[[UIDevice
-  currentDevice] model]`` (e.g., ``"iPhone13,2"``); or ``"iPhoneSimulator"`` for
-  simulated devices.
+In addition, a ``platform.ios_ver()`` method will be added. This mirrors
+``platform.mac_ver()``, which can be used to provide macOS version information.
+``ios_ver()`` will return a namedtuple that contains the following:
 
-All other values will be as returned by ``os.uname()``
+* ``version`` - the iOS version, as a string (e.g., ``"16.6.1"``).
+* ``min_version`` - the minimum supported iOS version, as a string (e.g., ``"12.0"``)
+* ``model`` - the model identifier of the device (e.g., ``iPhone13,2``). On simulators, this
+  will return ``iPhoneSimulator``.
+* ``is_simulator`` - a boolean indicating if the device is a simulator.
 
 ``os``
 ''''''
@@ -211,11 +208,14 @@ result in the following values:
 
 * ``release`` - The Darwin kernel version (e.g., ``"22.6.0"``)
 
+This approach treats the ``os`` module as a "raw" interface to system APIs, and
+``platform`` as a higher-level API providing more generally useful values.
+
 ``sysconfig``
 '''''''''''''
 
 The ``sysconfig`` module will use the minimum iOS version as part of
-``sysconfig.get_platform()`` identifier (e.g., ``"iOS-12.0-iphoneos-arm64"``).
+``sysconfig.get_platform()`` identifier (e.g., ``"ios-12.0-iphoneos-arm64"``).
 The ``sysconfigdata_name`` and Config makefile will follow the same patterns as
 existing platforms (using ``sys.platform``, ``sys.implementation._multiarch``
 etc.) to construct identifiers.
@@ -237,41 +237,42 @@ into a Framework location. This finder will only be installed if ``sys.platform
 == "ios"``.
 
 This finder will convert a Python module name (e.g., ``foo.bar._whiz``) into a
-unique Framework name by replacing the dots with underscores (i.e.,
-``foo_bar__whiz.framework``). A framework is a directory; the finder will look
+unique Framework name by using the full Python module as the framework name (i.e.,
+``foo.bar._whiz.framework``). A framework is a directory; the finder will look
 for ``_whiz.dylib`` in that directory.
 
 CI resources
 ------------
 
+`Anaconda <https://anaconda.com>`__ has offered to provide physical hardware to
+run iOS Buildbots.
+
 GitHub Actions is able to host iOS simulators on their macOS machines, and the
 iOS simulator can be controlled by scripting environments. The free tier
-currently only provides x86_64 macOS machines; however ARM64 runners `have
-recently become available on paid plans <https://github.blog/
+currently only provides x86_64 macOS machines; however ARM64 runners `recently
+became available on paid plans <https://github.blog/
 2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/>`__.
-
-If GitHub Actions resources are insufficient or not viable for cost reasons,
-Anaconda has offered to provide resources to support CI requirements.
+However, in order to avoid exhausting macOS runner resources, a Github Actions
+run for iOS will not be added as part of the standard CI configuration.
 
 Packaging
 ---------
 
 iOS will not provide a "universal" wheel format. Instead, wheels will be
-provided for each ABI-arch combination. At present, no binary merging is
-required. There is only one on-device architecture; and simulator binaries are
-not considered to be distributable artefacts, so only one architecture is needed
-to build a simulator.
+provided for each ABI-arch combination.
 
 iOS wheels will use tags:
 
-* ``iOS_12_0_iphoneos_arm64``
-* ``iOS_12_0_iphonesimulator_arm64``
-* ``iOS_12_0_iphonesimulator_x86_64``
+* ``ios_12_0_iphoneos_arm64``
+* ``ios_12_0_iphonesimulator_arm64``
+* ``ios_12_0_iphonesimulator_x86_64``
 
-In these tags, "12.0" is the minimum supported iOS version. The choice of
-minimum supported iOS version is a decision of whoever compiles CPython for iOS.
-At time of writing, iOS 12.0 exposes most significant iOS features, while
-reaching near 100% of devices.
+In these tags, "12.0" is the minimum supported iOS version. As with macOS, the
+wheel tag will incorporate the minimum iOS version that is selected when CPython
+is compiled; a wheel compiled with a minimum iOS version of 15.0 would use the
+``ios_15_0_iphone*`` tags. At time of writing, iOS 12.0 exposes most significant
+iOS features, while reaching near 100% of devices; this will be used as a floor
+for iOS version matching.
 
 These wheels can include binary modules in-situ (i.e., co-located with the
 Python source, in the same way as wheels for a desktop platform); however, they
@@ -340,15 +341,133 @@ a test suite that is executed on the iOS simulator using GitHub Actions.
 Rejected Ideas
 ==============
 
-``sys.implementation._simulator`` availability
-----------------------------------------------
+Simulator identification
+------------------------
+
+Earlier versions of this PEP suggested the inclusion of
+``sys.implementation._simulator`` attribute to identify when code is running on
+device, or on a simulator. This was rejected due to the use of a protected name
+for a public API, plus the pollution of the ``sys`` namespace with an
+iOS-specific detail.
+
+Another proposal during discussion was to include a generic
+``platform.is_emulator()`` API that could be implemented by any platform - for
+example to differentiate running on x86_64 code on ARM64 hardware, or when
+running in QEMU or other virtualization methods. This was rejected on the basis
+that it wasn't clear what a consistent interpretation of "emulator" would be, or
+how an emulator would be detected outside of the iOS case.
+
+The decision was made to keep this detail iOS-specific, and include it on the
+``platform.ios_ver()`` API.
+
+GNU compiler triples
+--------------------
+
+``autoconf`` requires the use of a GNU compiler triple to identify build and
+host platforms. However, the ``autoconf`` toolchain doesn't provide native
+support for iOS simulators, so we are left with the task of working out how to
+squeeze iOS hardware into GNU's naming regimen.
+
+This can be done (with some patching of ``config.sub``), but it leads to 2 major
+sources of naming inconsistency:
+
+* ``arm64`` vs ``aarch64`` as an identifier of 64-bit ARM hardware; and
+* What identifier is used to represent simulators.
+
+Apple's own tools use ``arm64`` as the architecture, but appear to be tolerant
+of ``aarch64`` in some cases. The device platform is identified as ``iphoneos``
+and ``iphonesimulator``.
+
+Rust toolchains uses ``aarch64`` as the architecture, and use
+``aarch64-apple-ios`` and ``aarch64-apple-ios-sim`` to identify the device
+platform; however, they use ``x86_64-apple-ios`` to represent iOS *simulators*
+on x86_64 hardware.
+
+The decision was made to use ``arm64-apple-ios`` and
+``arm64-apple-ios-simulator`` because:
+
+1. The ``autoconf`` toolchain already contains support for ``ios`` as a platform
+   in ``config.sub``; it's only the simulator that doesn't have a representation.
+2. The third part of the host triple is used as ``sys.platform``.
+3. When Apple's own tools reference CPU architecture, they use ``arm64``, and the
+   GNU tooling usage of the architecture isn't visible outside the build process.
+4. When Apple's own tools reference simulator status independent of the OS
+   (e.g., in the naming of Swift submodules), they use a ``-simulator`` suffix.
+5. While *some* iOS packages will use Rust, *all* iOS packages will use Apple's
+   tooling.
+
+"Universal" wheel format
+------------------------
+
+macOS currently supports 2 CPU architectures. To aid the end-user development
+experience, Python defines a "universal2" wheel format that incorporates both
+x86_64 and ARM64 binaries.
+
+It would be conceptually possible to offer an analogous "universal" iOS wheel
+format. However, this PEP does not use this approach, for 2 reasons.
+
+Firstly, the experience on macOS, especially in the numerical Python ecosystem,
+as been that universal wheels can be exceedingly difficult to accomodate. While
+native macOS libraries maintain strong multi-platform support, and Python itself
+has been updated, the vast majority of upstream non-Python libraries do not
+provide multi-architecture build support. As a result, compiling universal
+wheels inevitably requires multiple compilation passes, and complex decisions
+over how to distribute header files for different architectures. As a result of
+this complexity, many popular projects (including NumPy and Pillow) do not
+provide universal wheels at all, instead providing separate ARM64 and x86_64
+wheels.
+
+Secondly, historical experience is that iOS would require a much more fluid
+"universal" definition. In the last 10 years, there have been *at least* 5
+different possible interpretations of "universal" that would apply to iOS,
+including various combinations of armv6, armv7, armv7s, arm64, x86 and and
+x86_64 architectures, on device and simulator. If defined right now,
+"universal-iOS" would likely include x86_64 and arm64 on simulator, and arm64 on
+device; however, the pending deprecation of x86_64 hardware would add another
+interpretation; and there may be a need to add arm64e as a new device
+architecture in the future. Specifying iOS wheels as single-platform-only means
+the Python core team can avoid an ongoing standardization discussion about the
+updated "universal" formats.
+
+It also means wheel publishers are able to make per-project decisions over which
+platforms are feasible to support. For example, a project may choose to drop
+x86_64 support, or adopt a new architecture earlier than other parts of the
+Python ecosystem. Using platform-specific wheels means this decision can be left
+to individual package publishers.
+
+This decision comes at cost of making deployment more complicated. However,
+deployment on iOS is already a complicated process that is best aided by tools.
+At present, no binary merging is required, as there is only one on-device
+architecture, and simulator binaries are not considered to be distributable
+artefacts, so only one architecture is needed to build an app for a simulator.
+
+Interactive/REPL mode
+---------------------
 
-The ``_simulator`` attribute could be provided on *all* platforms, returning
-``False``. However, the attribute has no use outside of an iOS context.
+A traditional ``python.exe`` command line experience isn't really viable on
+mobile devices, because mobile devices don't have a command line. iOS apps don't
+have a stdout, stderr or stdin; and while you can redirect stdout and stderr to
+the system log, there's no source for stdin that exists that doesn't also
+involve building a very specific user-facing app that would be closer to an
+IDLE-style IDE experience. Therefore, the decision was made to only focus on
+"embedded mode" as a target for mobile distribution.
 
 Open Issues
 ===========
 
+x86_64 buildbot availability
+----------------------------
+
+Apple no longer sells x86_64 hardware. As a result, commissioning an x86_64
+buildbot may not be possible. It is possible to run macOS binaries in x86_64
+compatibility mode on ARM64 hardware; however, this isn't ideal for testing
+purposes.
+
+If native x86_64 Mac hardware cannot be sourced for buildbot purposes, it may be
+necessary to exclude the x86_64 simulator platform in Tier 3. Given the
+anticipated deprecation of x86_64 as a macOS development platform, this doesn't
+pose a significant impediment to adoption or long term maintenance.
+
 On-device testing
 -----------------