azure-cli-ml installation fails on MacOS arm

[EdAbati]

This is autogenerated. Please review and update as needed.

Describe the bug

Hi I am trying to install azure-cli-ml using the command below, but unfortunately it fails. Please see error below.

Command Name
az extension add

Errors:
It looks like pip fails to install some libraries:

An error occurred. Pip failed with status code 1. Use --debug for more information.

When running --debug and scrolling through the log, I noticed that probably the problem is cryptography. Its wheel cannot be built:

Building wheels for collected packages: cryptography
  Created temporary directory: [some-temp-path]
  Destination directory: [some-temp-path]
  Building wheel for cryptography (pyproject.toml): started
  Running command Building wheel for cryptography (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  running egg_info
  writing src/cryptography.egg-info/PKG-INFO
  writing dependency_links to src/cryptography.egg-info/dependency_links.txt
  writing requirements to src/cryptography.egg-info/requires.txt
  writing top-level names to src/cryptography.egg-info/top_level.txt
  reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs/_build'
  warning: no previously-included files found matching 'vectors'
  warning: no previously-included files matching '*' found under directory 'vectors'
  warning: no previously-included files matching '*' found under directory '.github'
  warning: no previously-included files found matching 'release.py'
  warning: no previously-included files found matching '.coveragerc'
  warning: no previously-included files found matching 'codecov.yml'
  warning: no previously-included files found matching '.readthedocs.yml'
  warning: no previously-included files found matching 'dev-requirements.txt'
  warning: no previously-included files found matching 'rtd-requirements.txt'
  warning: no previously-included files found matching 'tox.ini'
  warning: no previously-included files matching '*' found under directory '.zuul.d'
  warning: no previously-included files matching '*' found under directory '.zuul.playbooks'
  adding license file 'LICENSE'
  adding license file 'LICENSE.APACHE'
  adding license file 'LICENSE.BSD'
  adding license file 'LICENSE.PSF'
  adding license file 'AUTHORS.rst'
  running build_ext
  generating cffi module 'build/temp.macosx-12-arm64-cpython-310/_padding.c'
  generating cffi module 'build/temp.macosx-12-arm64-cpython-310/_openssl.c'
  building '_openssl' extension
  build/temp.macosx-12-arm64-cpython-310/_openssl.c:575:10: fatal error: 'openssl/opensslv.h' file not found
  #include <openssl/opensslv.h>
           ^~~~~~~~~~~~~~~~~~~~
  1 error generated.

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      =============================DEBUG ASSISTANCE=============================

  error: command '/usr/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for cryptography (pyproject.toml) did not run successfully.

Is it a known bug with MacOS with arm? Do you know how I could fix it?
Thanks :)

To Reproduce:

Steps to reproduce the behavior.

• brew update && brew install azure-cli
• az login
• az extension add -n azure-cli-ml --debug

Expected Behavior

azure-cli-ml installed successfully.

Environment Summary

macOS-12.3.1-arm64-arm-64bit, Darwin 21.4.0
Python 3.10.2
Installer: HOMEBREW

azure-cli 2.35.0

Additional Context

[yonzhan]

route to CXP team

[navba-MSFT]

@EdAbati Thanks for reaching out to us and reporting this issue. We are looking into this. I will provide an update on this issue once I have more details.

[navba-MSFT]

@EdAbati Could you please check if you are able to run the below command ? Awaiting your reply.

az extension add -n ml -y

[EdAbati]

Hi @navba-MSFT , thank you for the quick reply.
I run that command (with --debug ) and still fails in the same exact way. There is still a problem with cryptography

[Riezebos]

Hi! I had the same problem yesterday and today had some time to look into it. I have an M1 macbook. I was able to get the installation to work using:

brew install [email protected]
export LDFLAGS="-L/opt/homebrew/opt/[email protected]/lib"
export CPPFLAGS="-I/opt/homebrew/opt/[email protected]/include"
az extension add -n ml -y

(based on https://stackoverflow.com/questions/29651258/install-openssl-devel-on-mac)

I'm 99% sure this will also solve the issue for azure-cli-ml since the underlying problem is the same.

I also tested the installation of the cryptography package in a fresh venv created with [email protected]. The results are:
>=3.3.2,<=3.4.5: Error (sometimes an error saying I should have rust installed)
>=3.4.6: No issues (a wheel is found and used starting with this version).

The workaround above can be used as a quick fix for people who find this issue, to make it work automatically it might be best to upgrade cryptography to 3.4.6 or higher.

[navba-MSFT]

Thanks @Riezebos for sharing this.

@EdAbati Could you please try the above suggestions and check if that helps ? Awaiting your reply.

[EdAbati]

Awesome, thanks @Riezebos for the workaround! :)

@navba-MSFT , it worked for now, but as @Riezebos said I also think the best/long term solution would be to upgrade the requirements so azure-cli-ml uses a newer version of cryptography. If I understood correctly from the logs the cryptography version is fixed to <=3.3.2 (or similar)

[navba-MSFT]

@EdAbati Thanks for getting back. I am glad that the issue is resolved now. Since the error is when building cryptography, This doc talks about the dependencies like rust and openssl which are required.

Also refer this #16858, where there is discussion on pinning the Cryptography to 3.3.2 in AzCLI. I am doing some more research on this and I will update this thread and provide more info regarding the azure-cli-ml using newer version of cryptography once I have the details.

[navba-MSFT]

@EdAbati Could you please confirm if you are using a Mac with an M1 chip ? Awaiting your reply.

[Riezebos]

The problem in #16858 should only be a problem when the package needs to be built from source. If a wheel is available for your system type the installation should work without having rust. This is why it starts working on mac-arm64 starting with 3.4.6 as I mentioned above. Looking at the available wheels in 36.0.2 compared to 3.4, I would expect that #16858 is not an issue anymore.

[navba-MSFT]

@Riezebos Thanks for your comment. @EdAbati Regarding the issue reported in this github thread this is indeed a known issue and we’ll investigate a solution and get back with a timeline.

@EdAbati Could you please confirm if you are using a Mac with an M1 chip ? Awaiting your reply.

[EdAbati]

Hi @navba-MSFT , Yes I confirm that I have a Mac with M1.
And I fully agree with @Riezebos .
Thank you both for the help!

[navba-MSFT]

@EdAbati The fix for this issue is checked in for the v2 CLI (ml extension), and it will go out in the next release on 5/23. We recommend our customers to use the ml extension. If you face any issues while migration, please raise an issue, we would be happy to help.

Thanks again @EdAbati and @Riezebos for reporting this issue.