macos arm64 build #53

isuruf · 2021-01-23T22:32:05Z

depends on https://github.com/matthew-brett/multibuild/pull/384

tylerjereddy

Cool! For some reason, I was under the impression that the access to the OS/plat combination in question was somewhat limited for now.

In case you didn't know (you probably do)--our Travis CI credits are pretty limited for this organization (i.e., if experimentation is possible on github actions).

mattip · 2021-01-24T05:34:18Z

@tylerjereddy I think this does atraget github actions, just the script is named travis-ci/build_steps.sh for historical reasons. At some point we should rename the directory.

mattip

Thanks for this. It looks good to me, just a few minor comments.

mattip · 2021-01-24T05:39:24Z

.github/workflows/multibuild.yml

+          - os: macos-latest
+            PLAT: arm64
+            INTERFACE64: ''
+            platform: [x64]


I am not sure we need platform at all in this file, it might be some cruft left over from the transition to github actions?

Maybe add a comment to explain that this entry of the build matrix is using cross-compilation to build an macos/arm64 on a macos/x86_64 host (typically on github actions that does not provide native macos/arm64 runners at this time).

travis-ci/build_steps.sh

mattip · 2021-01-24T05:49:39Z

This is cross-compiling using x86_64 hardware to compile the arm64 library.

ogrisel · 2021-01-29T14:46:17Z

travis-ci/build_steps.sh

        s390x)
            local bitness=64
            ;;
        ppc64le)
            local bitness=64
+            local target_flags="TARGET=POWER8"


I wonder if this change will impact the future numpy / scipy wheels built for ppc64le. Those wheels are typically built on a PPC host (without cross-compilation) so maybe this won't change a thing (but I agree it's best to make the target architecture explicit).

Those wheels are typically built on a PPC host (without cross-compilation) so maybe this won't change a thing

It will if the host becomes a power9 or power10 some day.

ogrisel

The new entry in the build matrix fails after a bunch of ld warnings:

021-01-23T23:07:18.9595570Z ld: warning: object file (sblat1.o) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:18.9597490Z ld: warning: dylib (/opt/gfortran-darwin-arm64/lib/gcc/arm64-apple-darwin20.0.0/10.2.1/libgfortran.dylib) was built for newer macOS version (11.0) than being linked (10.9)
[...]
2021-01-23T23:07:18.9611620Z ld: warning: object file (../libopenblasp-r0.3.13.dev.a(isamax.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:18.9613540Z ld: warning: object file (../libopenblasp-r0.3.13.dev.a(sasum.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:18.9615450Z ld: warning: object file (../libopenblasp-r0.3.13.dev.a(saxpy.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:18.9617340Z ld: warning: object file (../libopenblasp-r0.3.13.dev.a(scopy.o)) was built for newer macOS version (11.0) than being linked (10.9)
[...]
2021-01-23T23:07:50.0315770Z ld: warning: object file (/opt/gfortran-darwin-arm64/lib/gcc/arm64-apple-darwin20.0.0/10.2.1/libgcc.a(_udivmoddi4.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:50.0317750Z ld: warning: object file (/opt/gfortran-darwin-arm64/lib/gcc/arm64-apple-darwin20.0.0/10.2.1/libgcc.a(_udiv_w_sdiv.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:50.0319920Z ld: warning: object file (/opt/gfortran-darwin-arm64/lib/gcc/arm64-apple-darwin20.0.0/10.2.1/libgcc.a(enable-execute-stack.o)) was built for newer macOS version (11.0) than being linked (10.9)
2021-01-23T23:07:50.0321110Z Undefined symbols for architecture arm64:
2021-01-23T23:07:50.0321630Z   "___chkstk_darwin", referenced from:
2021-01-23T23:07:50.0322620Z       _sgemv_ in libopenblasp-r0.3.13.dev.a(sgemv.o)
2021-01-23T23:07:50.0323660Z       _sger_ in libopenblasp-r0.3.13.dev.a(sger.o)
2021-01-23T23:07:50.0324750Z       _cblas_sgemv in libopenblasp-r0.3.13.dev.a(cblas_sgemv.o)
2021-01-23T23:07:50.0325860Z       _cblas_sger in libopenblasp-r0.3.13.dev.a(cblas_sger.o)
2021-01-23T23:07:50.0326930Z       _dgemv_ in libopenblasp-r0.3.13.dev.a(dgemv.o)
2021-01-23T23:07:50.0327970Z       _dger_ in libopenblasp-r0.3.13.dev.a(dger.o)
2021-01-23T23:07:50.0329050Z       _cblas_dgemv in libopenblasp-r0.3.13.dev.a(cblas_dgemv.o)
2021-01-23T23:07:50.0329590Z       ...
2021-01-23T23:07:50.0329980Z ld: symbol(s) not found for architecture arm64
2021-01-23T23:07:50.0330510Z collect2: error: ld returned 1 exit status
2021-01-23T23:07:50.0331500Z make[1]: *** [libopenblasp-r0.3.13.dev.dylib] Error 1
2021-01-23T23:07:50.0332080Z make: *** [shared] Error 2
2021-01-23T23:07:50.0339920Z ##[error]Process completed with exit code 2.

Potential fix below:

.github/workflows/multibuild.yml

mattip · 2021-01-30T18:17:42Z

The new run is failing after building in a switch statement with echo Did not recognize plat arm64. It seems one of the bash commands is missing set -x and there is somewhere without arm64

isuruf · 2021-01-30T18:18:56Z

See MacPython/gfortran-install#5

mattip · 2021-01-30T22:16:39Z

s390x is failing, everything else passes. This now creates a tarball openblas-v0.3.13-62-gaf2b0d02-macosx_11_0_arm64-gf_f10e307.tar.gz. How do we know it performs correctly without testing it?

isuruf · 2021-01-30T22:51:38Z

How do we know it performs correctly without testing it?

We don't.

ogrisel · 2021-01-31T14:44:06Z

How do we know it performs correctly without testing it?

While waiting for CI-providers to provide M1 support, we can run the tests manually from time to time on people's laptop. @mattip let me know if you want me to run the numpy tests before a release for instance.

ogrisel · 2021-01-31T15:07:47Z

The s390x builds on travis only fail when trying to install anaconda-client to upload the resulting artifacts:

+++pip install -q git+https://github.com/Anaconda-Platform/[email protected]
  WARNING: Building wheel for clyent failed: [Errno 13] Permission denied: '/home/travis/.cache/pip/wheels/a8'
  WARNING: Building wheel for PyYAML failed: [Errno 13] Permission denied: '/home/travis/.cache/pip/wheels/30'
  WARNING: Building wheel for pyrsistent failed: [Errno 13] Permission denied: '/home/travis/.cache/pip/wheels/34'
ERROR: Could not build wheels for PyYAML which use PEP 517 and cannot be installed directly

Nothing in the current PR seem to be able to cause this, beyond the multibuild update.

mattip · 2021-01-31T20:30:14Z

OK, I guess this is required to even think about numpy wheels for M1, so let's merge it.

mattip · 2021-01-31T20:32:52Z

Thanks @isuruf. The next step for NumPy will be to add arm64 to the accepted platforms in tool/openblas_support.py so that people with a M1 can do python tool/openblas_support.py to download the artifact.

ogrisel · 2021-02-01T17:19:21Z

@mattip @isuruf ok the build was successful:

I downloaded the tarball with:

curl -L --output openblas.tar.gz https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.13-62-gaf2b0d02/download/openblas-v0.3.13-62-gaf2b0d02-macosx_11_0_arm64-gf_f10e307.tar.gz

Then I unzipped it in /opt/arm64-builds/. This is important because this path is hardcoded in the dylib and importing numpy will fail otherwise.

My numpy site.cfg had therefore the following lines:

[openblas]
libraries = openblas
library_dirs = /opt/arm64-builds/lib
include_dirs = /opt/arm64-builds/include
runtime_library_dirs = /opt/arm64-builds/lib

Then I could build numpy (pip install -e . --no-build-isolation -v) and I can import numpy and it dynlinks to openblas successfully:

% python -m threadpoolctl -i numpy
[
  {
    "filepath": "/opt/arm64-builds/lib/libopenblasp-r0.3.13.dev.dylib",
    "prefix": "libopenblas",
    "user_api": "blas",
    "internal_api": "openblas",
    "version": "0.3.13.dev",
    "num_threads": 8,
    "threading_layer": "pthreads"
  }
]

and I confirm that this is the arm64 build:

% file /opt/arm64-builds/lib/libopenblasp-r0.3.13.dev.dylib
/opt/arm64-builds/lib/libopenblasp-r0.3.13.dev.dylib: Mach-O 64-bit dynamically linked shared library arm64

and for Python:

% python -c "import platform; print(platform.platform())"
macOS-11.1-arm64-arm-64bit

I can then run the numpy tests almost successfully. There are only 6 failures and those are the same as those failing with the conda-forge macos/arm64 build of numpy + openblas. Here are the details:

https://gist.github.com/ogrisel/faa61675b490dcf749a60e26840938fb

Note that the first time I had downloaded the archive with Firefox, but when I try to import numpy I get a pop-up that said:

“libopenblasp-r0.3.13.dev.dylib” can’t be opened because Apple cannot check it for malicious software.

This problem does not happen if you download the tarball with curl apparently...

ogrisel · 2021-02-01T17:22:50Z

So I guess the next step would be to adapt https://github.com/MacPython/numpy-wheels to try build the macos/arm64 numpy wheel in a cross-compiler environment (using crossenv) and this OpenBLAS tarball.

Also the openblas_support.py script will need an update because it's confusing OS info and architecture info. It would be better to have platform info as the concatenation of OS identifier and architecture identifier.

isuruf · 2021-02-01T20:28:21Z

So I guess the next step would be to adapt https://github.com/MacPython/numpy-wheels to try build the macos/arm64 numpy wheel in a cross-compiler environment (using crossenv) and this OpenBLAS tarball.

Yes, multibuild has support now. So, once openblas_support.py is fixed, this should be good to go.

Also the openblas_support.py script will need an update because it's confusing OS info and architecture info. It would be better to define the platform info as the concatenation of an OS identifier and an architecture identifier.

Agreed. It's a mess because of mixing OS and architecture info. Anybody wants to fix that? @mattip, @rgommers, @ogrisel ?

ogrisel · 2021-02-02T10:02:18Z

How do we know it performs correctly without testing it?

@mattip https://www.scaleway.com/en/hello-m1/ is now providing (costly) M1 hosting for continuous integration. Maybe it would be possible to ask them for a CI free runner for github actions for the scipy ecosystem. Or we can just wait for the github and Azure to provide M1 runners by default for free (which is bound to happen at some point I think).

Agreed. It's a mess because of mixing OS and architecture info. Anybody wants to fix that? @mattip, @rgommers, @ogrisel ?

Not in the coming days. But I can give it a try a bit later if nobody did it in the mean time.

rgommers · 2021-02-02T10:39:34Z

It's 10 cents an hour, that's okay. It's just a hosted service though, where you can use SSH/VPN in (https://www.scaleway.com/en/docs/apple-silicon-as-a-service-quickstart/) - there's no GitHub Actions or other CI provider integration as far as I can tell.

If someone wants to work on this connect to the wheel build repos as CI, we can pay for the hosting costs - it'll be 75 euro/hr until we can drop it because free CI providers gain support (maybe 6 months or so?).

ogrisel · 2021-02-02T17:32:10Z

Unfortunately, it appears that it's not trivial to register an Apple M1 machine as a github actions runner: actions/runner#805.

rgommers · 2021-02-02T19:17:27Z

That's too bad.

The opinions on M1 support diverge in interesting ways. From "we want this yesterday, it's the best thing since sliced bread" to "why would we care unless there's CI support or Apple does or pays for all the work".

Could also say it's good advertisement for conda-forge, which has superior infrastructure, so let's point people at it for now (that's what we have to say today anyway).

Guess we'll end up being pragmatic and do manual testing + upload to PyPI.

ogrisel · 2021-02-03T14:37:11Z

"we want this yesterday, it's the best thing since sliced bread"

I really like this hardware. It's really a step up in terms of computational performance - battery autonomy - no noise - low waste heat / $.

Could also say it's good advertisement for conda-forge, which has superior infrastructure, so let's point people at it for now (that's what we have to say today anyway).

I agree but. I really had a smooth experience with conda-forge (especially with the faster mamba), but apparently that won't prevent users to complain and ask for pip-installable wheels for M1 machines.

Guess we'll end up being pragmatic and do manual testing + upload to PyPI.

I guess so :) Finalizing the infra (e.g. fixing openblas_support.py) for this will not be wasted because it will be reused once github actions / azure pipelines and co start to natively host M1 runners.

ilayn · 2021-02-03T20:09:35Z

"why would we care unless there's CI support or Apple does or pays for all the work".

That would be me. I wholeheartedly agree with the efforts but covering apple's back is something I resent a lot given how precious maintainers' time is. Especially if the next model is going to be yet another "revolutionary upgrade" this will all go to waste which is pretty much possible given their track record.

Not to mention there are a huge population of Python users who won't or can't use conda so this needs a bit of a pushback towards apple.

matthew-brett · 2021-02-04T10:43:40Z

The divergence of opinions doesn't surprise me - I bet it tracks closely with the divergence of opinions on Apple and Macs generally.

The key point for me is that I want Numpy and scientific Python to be the natural best choice for scientific computing. In practice, a lot of scientists are going to own M1 Macs, soon, and they will feel strongly about getting best performance out of their machines. If Numpy is fairly well behind, or perceived as being fairly well behind the best performance, on these machines, that will be a problem for the ecosystem.

And for the maintainer time - I don't own an M1 Mac personally, but I would be suprised if they weren't fairly common as developer machines in a few months.

mattip · 2021-02-04T12:23:24Z

@matthew-brett I am not sure what you are proposing. I do not feel comfortable releasing software we have not tested. Can some of these people urging us to release packaging prompt Apple and the CI vendors to make runners available?

matthew-brett · 2021-02-04T12:32:33Z

My suggestion above was only that we can't afford to dismiss M1 as a major platform for Numpy and scientific computing, whatever we think of Apple or its engagement with the open source community. It's really a matter of tone. At the moment I think we're giving off a vague impression of "why should we help you to use your fancy and exotic hardware" - rather than "excellent, we're planning to support this well, as soon as we can, here's how you can help".

ilayn · 2021-02-04T12:39:44Z

That's very true but I think there is a balance here. There is a difference between "OK here is one more architecture we have to support, there are no tools for it yet but it's popular" and "can we contact apple about this? we would love to have it but currently that's too much risk."

Similarly we ignored the whole windows land if it wasn't for the fantastic efforts of Gohlke's builds. It is almost identical to that situation now. So I find this apple adventure a bit too enthusiastic.

Other than that fully agree with the tone part

rgommers · 2021-02-04T13:49:03Z

Similarly we ignored the whole windows land if it wasn't for the fantastic efforts of Gohlke's builds

Not quite right - we spent a ton of effort building .exe MSI installers (on Linux, via Wine) before Appveyor, wheels etc. came into existence.

"excellent, we're planning to support this well, as soon as we can, here's how you can help".

That sounds about right to me.

ilayn · 2021-02-04T18:08:51Z

Well for more than a decade we couldn't have pip install for SciPy on windows I don't think that counts as supporting it

isuruf · 2021-02-04T20:56:42Z

@ogrisel numpy/numpy#18331

isuruf added 13 commits January 14, 2021 02:07

try new gfortran install

e76439e

build for arm64

c042088

use python 3.9

c173bcc

use 3.9 in build_steps.sh

93f5531

update gfortran-install

ca85acd

skip linux builds for now

34f3564

verbose

4a09007

debug

19ba74a

use wrap_wheel_builder

cf836ed

use macos-11.0 image

72e6b5f

use macos-latest with new xcode

3e16738

remove debug

060da29

guard xcode-select

a4cf419

tylerjereddy reviewed Jan 23, 2021

View reviewed changes

mattip reviewed Jan 24, 2021

View reviewed changes

ogrisel reviewed Jan 29, 2021

View reviewed changes

.github/workflows/multibuild.yml Show resolved Hide resolved

update multibuild and make the switch labels clearer

571bdeb

Update gfortran-install

e1abae8

ogrisel mentioned this pull request Jan 31, 2021

BLD: fail to build on Apple M1 numpy/numpy#17807

Closed

mattip merged commit c70ac71 into MacPython:master Jan 31, 2021

isuruf deleted the arm64 branch January 31, 2021 20:31

rgommers mentioned this pull request Nov 14, 2021

Build for Apple M1 silicon #49

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macos arm64 build #53

macos arm64 build #53

isuruf commented Jan 23, 2021

tylerjereddy left a comment

mattip commented Jan 24, 2021

mattip left a comment

mattip Jan 24, 2021

ogrisel Jan 29, 2021

mattip commented Jan 24, 2021

ogrisel Jan 29, 2021

isuruf Jan 29, 2021

ogrisel left a comment

mattip commented Jan 30, 2021

isuruf commented Jan 30, 2021

mattip commented Jan 30, 2021

isuruf commented Jan 30, 2021

ogrisel commented Jan 31, 2021

ogrisel commented Jan 31, 2021

mattip commented Jan 31, 2021

mattip commented Jan 31, 2021

ogrisel commented Feb 1, 2021 •

edited

Loading

ogrisel commented Feb 1, 2021

isuruf commented Feb 1, 2021 •

edited by ogrisel

Loading

ogrisel commented Feb 2, 2021

rgommers commented Feb 2, 2021

ogrisel commented Feb 2, 2021

rgommers commented Feb 2, 2021

ogrisel commented Feb 3, 2021 •

edited

Loading

ilayn commented Feb 3, 2021 •

edited

Loading

matthew-brett commented Feb 4, 2021

mattip commented Feb 4, 2021

matthew-brett commented Feb 4, 2021

ilayn commented Feb 4, 2021

rgommers commented Feb 4, 2021

ilayn commented Feb 4, 2021 •

edited

Loading

isuruf commented Feb 4, 2021

macos arm64 build #53

macos arm64 build #53

Conversation

isuruf commented Jan 23, 2021

tylerjereddy left a comment

Choose a reason for hiding this comment

mattip commented Jan 24, 2021

mattip left a comment

Choose a reason for hiding this comment

mattip Jan 24, 2021

Choose a reason for hiding this comment

ogrisel Jan 29, 2021

Choose a reason for hiding this comment

mattip commented Jan 24, 2021

ogrisel Jan 29, 2021

Choose a reason for hiding this comment

isuruf Jan 29, 2021

Choose a reason for hiding this comment

ogrisel left a comment

Choose a reason for hiding this comment

mattip commented Jan 30, 2021

isuruf commented Jan 30, 2021

mattip commented Jan 30, 2021

isuruf commented Jan 30, 2021

ogrisel commented Jan 31, 2021

ogrisel commented Jan 31, 2021

mattip commented Jan 31, 2021

mattip commented Jan 31, 2021

ogrisel commented Feb 1, 2021 • edited Loading

ogrisel commented Feb 1, 2021

isuruf commented Feb 1, 2021 • edited by ogrisel Loading

ogrisel commented Feb 2, 2021

rgommers commented Feb 2, 2021

ogrisel commented Feb 2, 2021

rgommers commented Feb 2, 2021

ogrisel commented Feb 3, 2021 • edited Loading

ilayn commented Feb 3, 2021 • edited Loading

matthew-brett commented Feb 4, 2021

mattip commented Feb 4, 2021

matthew-brett commented Feb 4, 2021

ilayn commented Feb 4, 2021

rgommers commented Feb 4, 2021

ilayn commented Feb 4, 2021 • edited Loading

isuruf commented Feb 4, 2021

ogrisel commented Feb 1, 2021 •

edited

Loading

isuruf commented Feb 1, 2021 •

edited by ogrisel

Loading

ogrisel commented Feb 3, 2021 •

edited

Loading

ilayn commented Feb 3, 2021 •

edited

Loading

ilayn commented Feb 4, 2021 •

edited

Loading