hardcoded MB_PYTHON_OSX_VER

[robbuckley]

it seems a bit clumsy from a user PoV to have to remember that the default macOS version is hardcoded to 10.6 (10.9 for python 3.8). I've got a PR that derives a sensible default from MB_PYTHON_VERSION -- namely the newest build for which python.org provides a build on https://www.python.org/downloads/mac-osx/. OK to submit @matthew-brett ?

[matthew-brett]

Sure. As of a few days ago, the default behavior is to use 10.9 by default for Python 3.8 and later - is that what you are suggesting?

[robbuckley]

A bit further than that - it would be to use 10.9 by default for all python versions where it is available - 2.7.15+ and 3.6+. So for normal use cases it would not be neccessary to use MB_PYTHON_OSX_VER in users' .travis.yml files

[robbuckley]

ping @matthew-brett - I have the MR for this and 2 others pending, any chance they will be merged soon? pls let me know if i need to change anything

[anthrotype]

I'm not sure what's the recommended MB_PYTHON_VERSION now. Say a project of mine only supports Python 3.6+ (and I don't care about 32 bit), should I set the MB_PYTHON_VERSION=10.9 for the py3.6 and py3.7 builds?
I can see that it is still defaulting to 10.6.
What is the implication of selecting 10.9 when one has a choice?

[matthew-brett]

I guess the issue with 10.6 vs 10.9 as the default installer is that the 10.6 wheels will at least appear to be intel compatible, meaning combined 32 / 64 bit binaries, which is what system Python requires, in order to install the wheel, and this is also what Python.org installations from 10.6 installers need. So, I think, by default, wheels built with 10.9 installers won't install into a quite a few Python installations.

[anthrotype]

I see, thanks. So I guess it makes sense to continue building for 10.6 "intel" wherever possible, since it's going to work on either 32 or 64-bit pythons.

[matthew-brett]

@robbuckley - sorry - I didn't check - but do your changed default to 10.6 where possible? I thought they default to 10.9 ?

[robbuckley]

@matthew-brett @anthrotype I wouldnt reccomend 10.6 as a default, as it can have problems with newer macOS SDK versions - see pandas-dev/pandas#23424

The default should be 10.9 / x86_64, with 10.6 / intel to support older python versions where needed (e.g. 3.5 and before, or anything before 2.7.15)

With https://github.com/matthew-brett/multibuild/pull/279 the reccomendation for user projects is not to set MB_PYTHON_OSX_VER and to rely on the defaults. These are to target the newest available macOS SDK for the python version built for - 10.9 for 2.7.15+ and 3.6.5+ and 10.6 for older pythons.

The end user can override that by setting MB_PYTHON_OSX_VER, provided there is a python download available for that combination of MB_PYTHON_VER and MB_PYTHON_OSX_VER at https://www.python.org/downloads/mac-osx/

In practise that would only be needed if you want to target 10.6 where a 10.9 build is available (for example for 3.6.8). The only use case i could think of for that is if there is a system python which targets 10.6 where python.org provides a 10.9 build for the same version.

FYI my system python (macOS 10.5.1 with Xcode 11.2.1) is 2.7.16 with distutils tag macosx-10.15-x86_64

[robbuckley]

Say a project of mine only supports Python 3.6+ (and I don't care about 32 bit), should I set the MB_PYTHON_VERSION=10.9 for the py3.6 and py3.7 builds?

@anthrotype if you are targetting MB_PYTHON_VER=3.6 it should default to 10.9, you dont need to set it see https://github.com/robbuckley/multibuild/blob/0ecd047469cb4af0563c9bb4beef4f2fce9dbb90/osx_utils.sh#L109

[matthew-brett]

@robbuckley - do you agree, that the 10.9 shift will also result in reduced compatibility of the wheels, in that they will be 64-bit only, and won't install to 32/64-bit Pythons, which are still pretty common?

Am I right in thinking that the problem with building for 10.6 is that it generates a requirement for libstdc++, which is not installed by recent Xcodes?

[robbuckley]

@matthew-brett I've previously found pip rules to be quite lax: https://github.com/matthew-brett/multibuild/pull/224#issue-250238516:

recent versions of pip I've tried (18.1, 19.0.1) seem to flag both the 10.6-based [tagged as 32/64b] and 10.9-based [tagged as 64b] wheels I've built as compatible with my [64b] system with both single arch [64b] and dual-arch [32/64b] pythons.

Unfortunately I cant seem to replicate that now - all my python environments report x86_64 and not intel, even 3.5 and 3.6 and I cant seem to install any dual arch 32/64b python as the system python. Maybe this is because i've upgraded to 10.15 and 32bit isnt supported at all anymore.

I may have access to an older mac system over christmas, which i could try this on. Or i could try to set up a test on travis.

And yes the issue Im aware of with running 10.6 builds on current systems is that since xcode 10, libstdc++ insnt installed anymore. Its been deprecated since xcode 8, which was released around 3 years ago. Its a potentially nasty issue, as most python C extensions dont link against the C++ std lib, and even for those which do, it may still work depending on the history of the system.

[matthew-brett]

My 10.14 system Python is Intel. I think you'll find that installing the 10.6 Python 3.7 from Python.org will give you an Intel Python too - that's what I get installing the latest Python 3.7 from Python.org on my 10.15.1 system. I think that means that there are a lot of intel installations out there, and that will likely be true for a while.

At the moment, I think we're in a situation where it's easier to make a less-useful wheel by mistake, using the 10.9 default.

[robbuckley]

I managed to install the 10.6 intel flavour of python 3.7.3 from python.org. This seems quite happy to install a 10.9 x86_64 wheel:

$ python3.7
Python 3.7.4 (v3.7.4:e09359112e, Jul  8 2019, 14:36:03) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.util; print(distutils.util.get_platform())
macosx-10.6-intel
>>> exit()
$ python3.7 -m pip install --user ~/Downloads/pandas-0.25.3-cp37-cp37m-macosx_10_9_x86_64.whl
Processing ./Downloads/pandas-0.25.3-cp37-cp37m-macosx_10_9_x86_64.whl
Collecting numpy>=1.13.3 (from pandas==0.25.3)
  Downloading https://files.pythonhosted.org/packages/60/9a/a6b3168f2194fb468dcc4cf54c8344d1f514935006c3347ede198e968cb0/numpy-1.17.4-cp37-cp37m-macosx_10_9_x86_64.whl (15.1MB)
    100% |████████████████████████████████| 15.1MB 1.6MB/s 
Collecting pytz>=2017.2 (from pandas==0.25.3)
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting python-dateutil>=2.6.1 (from pandas==0.25.3)
  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2.6.1->pandas==0.25.3)
  Downloading https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Installing collected packages: numpy, pytz, six, python-dateutil, pandas
Successfully installed numpy-1.17.4 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3 six-1.13.0
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python3.7
Python 3.7.4 (v3.7.4:e09359112e, Jul  8 2019, 14:36:03) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'0.25.3'

checking on tags that pip claims to support:

>>> tags=pip._internal.pep425tags.get_supported()
>>> [print(t) for t in tags if t[2].startswith('macosx') and t[1]=='cp37m']
('cp37', 'cp37m', 'macosx_10_15_x86_64')
('cp37', 'cp37m', 'macosx_10_15_intel')
('cp37', 'cp37m', 'macosx_10_15_fat64')
('cp37', 'cp37m', 'macosx_10_15_fat32')
('cp37', 'cp37m', 'macosx_10_15_universal')
('cp37', 'cp37m', 'macosx_10_14_x86_64')
('cp37', 'cp37m', 'macosx_10_14_intel')
('cp37', 'cp37m', 'macosx_10_14_fat64')
('cp37', 'cp37m', 'macosx_10_14_fat32')
('cp37', 'cp37m', 'macosx_10_14_universal')
('cp37', 'cp37m', 'macosx_10_13_x86_64')
('cp37', 'cp37m', 'macosx_10_13_intel')
('cp37', 'cp37m', 'macosx_10_13_fat64')
('cp37', 'cp37m', 'macosx_10_13_fat32')
('cp37', 'cp37m', 'macosx_10_13_universal')
('cp37', 'cp37m', 'macosx_10_12_x86_64')
('cp37', 'cp37m', 'macosx_10_12_intel')
('cp37', 'cp37m', 'macosx_10_12_fat64')
('cp37', 'cp37m', 'macosx_10_12_fat32')
('cp37', 'cp37m', 'macosx_10_12_universal')
('cp37', 'cp37m', 'macosx_10_11_x86_64')
('cp37', 'cp37m', 'macosx_10_11_intel')
('cp37', 'cp37m', 'macosx_10_11_fat64')
('cp37', 'cp37m', 'macosx_10_11_fat32')
('cp37', 'cp37m', 'macosx_10_11_universal')
('cp37', 'cp37m', 'macosx_10_10_x86_64')
('cp37', 'cp37m', 'macosx_10_10_intel')
('cp37', 'cp37m', 'macosx_10_10_fat64')
('cp37', 'cp37m', 'macosx_10_10_fat32')
('cp37', 'cp37m', 'macosx_10_10_universal')
('cp37', 'cp37m', 'macosx_10_9_x86_64')
('cp37', 'cp37m', 'macosx_10_9_intel')
('cp37', 'cp37m', 'macosx_10_9_fat64')
('cp37', 'cp37m', 'macosx_10_9_fat32')
('cp37', 'cp37m', 'macosx_10_9_universal')
('cp37', 'cp37m', 'macosx_10_8_x86_64')
('cp37', 'cp37m', 'macosx_10_8_intel')
('cp37', 'cp37m', 'macosx_10_8_fat64')
('cp37', 'cp37m', 'macosx_10_8_fat32')
('cp37', 'cp37m', 'macosx_10_8_universal')
('cp37', 'cp37m', 'macosx_10_7_x86_64')
('cp37', 'cp37m', 'macosx_10_7_intel')
('cp37', 'cp37m', 'macosx_10_7_fat64')
('cp37', 'cp37m', 'macosx_10_7_fat32')
('cp37', 'cp37m', 'macosx_10_7_universal')
('cp37', 'cp37m', 'macosx_10_6_x86_64')
('cp37', 'cp37m', 'macosx_10_6_intel')
('cp37', 'cp37m', 'macosx_10_6_fat64')
('cp37', 'cp37m', 'macosx_10_6_fat32')
('cp37', 'cp37m', 'macosx_10_6_universal')
('cp37', 'cp37m', 'macosx_10_5_x86_64')
('cp37', 'cp37m', 'macosx_10_5_intel')
('cp37', 'cp37m', 'macosx_10_5_fat64')
('cp37', 'cp37m', 'macosx_10_5_fat32')
('cp37', 'cp37m', 'macosx_10_5_universal')
('cp37', 'cp37m', 'macosx_10_4_intel')
('cp37', 'cp37m', 'macosx_10_4_fat32')
('cp37', 'cp37m', 'macosx_10_4_universal')
('cp37', 'cp37m', 'macosx_10_3_fat32')
('cp37', 'cp37m', 'macosx_10_3_universal')
('cp37', 'cp37m', 'macosx_10_2_fat32')
('cp37', 'cp37m', 'macosx_10_2_universal')
('cp37', 'cp37m', 'macosx_10_1_fat32')
('cp37', 'cp37m', 'macosx_10_1_universal')
('cp37', 'cp37m', 'macosx_10_0_fat32')
('cp37', 'cp37m', 'macosx_10_0_universal')

basically, everything. Looking through the pip source, I think it defaults to declaring support for macOS wheels up to the system version which its currently running on (10.15 here) rather than distutils.util.get_platform() which is the minimum macOS SDK the python build is targetted for (10.6 in this case)

see:
https://github.com/pypa/pip/blob/a8de4eb49a45035aefd9a122b6d1d67915663d5f/src/pip/_internal/pep425tags.py#L286
and
https://github.com/pypa/pip/blob/a8de4eb49a45035aefd9a122b6d1d67915663d5f/src/pip/_internal/wheel.py#L722

[mattip]

This has to do with the MACOSX_DEPLOYMENT_TARGET set when building CPython. What do you get when you try sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET')? I think there are some subtle expectations around system calls: when using a too-low target some calls are blocked when building the c-extension module. I seem to remember something around clock_gettime only available after 10.12, leading to bugs like this

[robbuckley]

@matthew-brett I am just catching up with this so apologies as its probably all old news to you. pip since version 8.0 allows mac wheels to be installed where the wheel's macOS SDK tag <= that of the system on which its running. Prior to this change pypa/pip#3232 it was much more restrictive, only permitting wheels with tag <= python interpreter.

So a wheel tagged macosx_10_9_x86_64 should install on any 64-bit system running macOS 10.9 or later, regardless wether the installed python is built for 10.6 intel or 10.9 x86_64, if i understand correctly.

@mattip The OpenCV issue you link to doesnt state how it was installed -- that kind of issue should be avoided if we make sure wheels are correctly tagged with a macOS version >= that of MACOSX_DEPLOYMENT_TARGET when the wheel is built. Then pip will refused to install on a too-old system. multibuild ensures this is so.

[matthew-brett]

Yes - I did know about the 8.0 change, allowing you to install e.g 10.12 wheels on an e.g. 10.13 system, regardless of Python SDK. That was the reason we removed the wheel renaming a little while ago in a198d0e

I'm really surprised, though, that pip allows installing an x86_64 wheel on an intel system. If you really were using 32-bit - I would have thought this would cause all sorts of breakage. But I agree - in my own test just now, that is what happens.

That being so, I agree, changing to 10.9 by default is the way to go.

[robbuckley]

I'm really surprised, though, that pip allows installing an x86_64 wheel on an intel system. If you really were using 32-bit - I would have thought this would cause all sorts of breakage. But I agree - in my own test just now, that is what happens.

I think that works because pip's platform checking doesnt get the arch for compatibility purposes from the interpreter (distutils.util.get_platform()) but from platform.mac_ver(), which gets it from os.uname().machine, which is probably a passthrough to uname -m so equal to the mode the kernel is running in -- UNLESS python is running in 32bit mode when it should report i386:

https://github.com/pypa/pip/blob/b8c16a0dc86519e283a94b14591f9ddae27f9c55/src/pip/_internal/pep425tags.py#L145

On my system pip reports 64bit:

>>> platform.mac_ver()
('10.15.1', ('', '', ''), 'x86_64')
>>> distutils.util.get_platform()
'macosx-10.6-intel'
>>> pip._internal.pep425tags.get_platform()
'macosx_10_15_x86_64'

What I think this means is that pip will only refuse to install a x86_64 wheel due to arch incompatibility if either the kernel or python are running in 32bit mode. 32bit mode for the kernel was removed a loong time ago (10.8?). If anyone is really building wheels for installation on an 10.6-intel python running in 32bit mode, they always have the option of specifying MB_PYTHON_OSX_VER=10.6 to build a 10_6_intel wheel. Except from python3.8 where only 64b builds are available

[matthew-brett]

I'm totally OK with the wheels failing for Python in 32-bit mode - I just don't think there's anyone using that nowadays.

[matthew-brett]

OK - in it goes. We live in hope.