pyyaml=6.0 python=3.8 is broken with osx-arm64

[erykoff]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

pyyaml 6.0 is broken on osx-arm64, as it seems to have been built for the wrong architecture.

To reproduce:

conda create -n yaml6 python=3.8 pyyaml=6
conda activate yaml6
python
>>> import yaml.cyaml

This gives:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/erykoff/miniforge3/envs/yaml6/lib/python3.8/site-packages/yaml/cyaml.py", line 7, in <module>
    from yaml._yaml import CParser, CEmitter
ImportError: dlopen(/Users/erykoff/miniforge3/envs/yaml6/lib/python3.8/site-packages/yaml/_yaml.cpython-38-darwin.so, 2): no suitable image found.  Did find:
	/Users/erykoff/miniforge3/envs/yaml6/lib/python3.8/site-packages/yaml/_yaml.cpython-38-darwin.so: mach-o, but wrong architecture
	/Users/erykoff/miniforge3/envs/yaml6/lib/python3.8/site-packages/yaml/_yaml.cpython-38-darwin.so: mach-o, but wrong architecture

This does work with pyyaml=5.4.1.

Environment (conda list):

$ conda list
# Name                    Version                   Build  Channel
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
libcxx                    12.0.1               h168391b_0    conda-forge
libffi                    3.4.2                hbdafb3b_4    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
ncurses                   6.2                  h9aa5885_4    conda-forge
openssl                   3.0.0                h3422bc3_1    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
python                    3.8.12          hd949e87_2_cpython    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pyyaml                    6.0              py38hea4295b_1    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                58.4.0           py38h10201cd_1    conda-forge
sqlite                    3.36.0               h72a2b83_2    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
yaml                      0.2.5                h642e427_0    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : yaml6
    active env location : /Users/erykoff/miniforge3/envs/yaml6
            shell level : 1
       user config file : /Users/erykoff/.condarc
 populated config files : /Users/erykoff/miniforge3/.condarc
                          /Users/erykoff/.condarc
          conda version : 4.10.0
    conda-build version : not installed
         python version : 3.9.2.final.0
       virtual packages : __osx=11.6=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/erykoff/miniforge3  (writable)
      conda av data dir : /Users/erykoff/miniforge3/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/erykoff/miniforge3/pkgs
                          /Users/erykoff/.conda/pkgs
       envs directories : /Users/erykoff/miniforge3/envs
                          /Users/erykoff/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.10.0 requests/2.25.1 CPython/3.9.2 Darwin/20.6.0 OSX/11.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[jakirkham]

Can you please try build number 0 as well?

[erykoff]

Good question! Also broken. And looking at either the history of this feedstock or pyyaml upstream from 5.4.1 to today I don't see any obvious smoking guns.

[erykoff]

Ah, this is interesting. Python 3.9 and 3.10 work, it's just 3.8 that doesn't seem to be working.

[jakirkham]

Probably the next thing to do is look at the build log and see if there are any noticeable differences

[erykoff]

Okay, looking at this recent CI log building 3.8 here: https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/401148/logs/123 shows the compile line:

arm64-apple-darwin20.0.0-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I$BUILD_PREFIX/include -arch x86_64 -I$BUILD_PREFIX/include -arch x86_64 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/pyyaml-6.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_FORTIFY_SOURCE=2 -isystem $PREFIX/include -mmacosx-version-min=11.0 -I$PREFIX/include -I$PREFIX/include/python3.8 -c yaml/_yaml.c -o build/temp.macosx-10.9-x86_64-3.8/yaml/_yaml.o

Which has ... -arch x86_64. Which looks like a problem! (This isn't there in the 3.10 build, for example). I have no idea where this is being picked up, though. Everything else I see that should be relevant looks the same. @conda-forge/help-osx-arm64 any thoughts?

Further down the build is the warning:

warning: ignoring file /Users/runner/miniforge3/conda-bld/pyyaml_1635830692683/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/libyaml.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

[erykoff]

Okay, some spelunking leads me to believe that this line in pyyaml is the problem: https://github.com/yaml/pyyaml/blob/master/setup.py#L69-L70 which triggers a bug in distutils: pypa/setuptools#2257 which was fixed in main but not backported to py3.8.

So I'll do a PR which patches this line out when building on py3.8 and see if that works.

[jakirkham]

Sounds good. Thanks for all the detailed debugging here 😀