Error while installing dev dependencies for Apple Silicon

[xssChauhan]

System Info

• transformers version: 4.24.0
• Platform: macOS-13.2.1-arm64-arm-64bit
• Python version: 3.10.10
• Huggingface_hub version: 0.10.1
• PyTorch version (GPU?): 1.12.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Follow contribution guidelines as outlined here, at the pip install -e ".[dev]" step and results in the following output and error:

$ pip install -e ".[dev]"                     

Obtaining file:///Users/eipizero/Documents/Code/transformers
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting tqdm>=4.27
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting pyyaml>=5.1
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)
Collecting huggingface-hub<1.0,>=0.11.0
  Using cached huggingface_hub-0.13.3-py3-none-any.whl (199 kB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting filelock
  Using cached filelock-3.10.7-py3-none-any.whl (10 kB)
Collecting numpy>=1.17
  Using cached numpy-1.24.2-cp310-cp310-macosx_11_0_arm64.whl (13.9 MB)
Collecting regex!=2019.12.17
  Using cached regex-2023.3.23-cp310-cp310-macosx_11_0_arm64.whl (288 kB)
Collecting optax>=0.0.8
  Using cached optax-0.1.4-py3-none-any.whl (154 kB)
Collecting pyctcdecode>=0.4.0
  Using cached pyctcdecode-0.5.0-py2.py3-none-any.whl (39 kB)
Collecting nltk
  Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Collecting torch!=1.12.0,>=1.9
  Using cached torch-2.0.0-cp310-none-macosx_11_0_arm64.whl (55.8 MB)
Collecting jax!=0.3.2,<=0.3.6,>=0.2.8
  Using cached jax-0.3.6.tar.gz (936 kB)
  Preparing metadata (setup.py) ... done
Collecting pytest
  Using cached pytest-7.2.2-py3-none-any.whl (317 kB)
Collecting kenlm
  Using cached kenlm-0.1.tar.gz (424 kB)
  Preparing metadata (setup.py) ... done
Collecting GitPython<3.1.19
  Using cached GitPython-3.1.18-py3-none-any.whl (170 kB)
Collecting sudachipy>=0.6.6
  Using cached SudachiPy-0.6.7-cp310-cp310-macosx_10_12_universal2.whl (2.4 MB)
ERROR: Could not find a version that satisfies the requirement decord==0.6.0; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for decord==0.6.0; extra == "dev"

Expected behavior

Development dependencies should be installed without error.

[sgugger]

cc @amyeroberts and @nateraw

[nateraw]

Hmm maybe its the Python version? Some issue mentioning that here: dmlc/gluon-cv#1539

[xssChauhan]

The issue is mainly because of Apple Silicon. decord does not provide any built wheels for apple silicon, and hence cannot be found using pip. I had to build it from source and then install the python bindings.

Similar issue arises for tensorflow-text since it also does not provide any built wheels for apple silicon, and has to be built from scratch. I used a community built wheel from here.

I think the docs should be updated to account for these issues.

[nateraw]

I see. we should add a note then that in some cases you may need to install decord from source, and link to any related issues.

Or, perhaps we migrate fully to pyav at this point, given we started to do that here: #21572 (since decord is no longer being actively maintained and these issues will never go away).

WDYT?

[mayankagarwals]

My dev setup on apple silicon failed with

ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement jaxlib<=0.3.6,>=0.1.65; extra == "dev" (from transformers[dev]) (from versions: 0.3.24, 0.3.25, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.6, 0.4.7)
ERROR: No matching distribution found for jaxlib<=0.3.6,>=0.1.65; extra == "dev"

After a bit of hunting found success by installing jaxlib through conda : jax-ml/jax#5501 (comment)

Maybe it helps someone

[xssChauhan]

@nateraw Migrating fully to pyav is indeed the correct thing to do since the migration has already begun.

There are still other issues with setting up dev env on apple silicon, and setting it up correctly should be part of docs. It took me some time to correctly install the entire dev env. Following is the list of issues and solutions that worked for me for python 3.9:

• decord
  • Problem: No prebuilt wheels for apple silicon.
  • Solution: Building locally, and installing python bindings.
  • Action: Complete migration to pyav.
• tensorflow, tensorflow-*
  • Problem: tensorflow, tensorflow-* are not directly installable for macOS.
  • Solution: Need to install tensorflow-deps from conda apple channel. This has already been highlighted in a previous issue Unable to set up developer environment on Mac M1 #18355. Install tensorflow-macos( instead of tensorflow).
  • Action:
    - setup.pyshould be able to detect if the dev env is being setup on apple silicon, and install tensorflow-macosinstead of tensorflow.
    - Docs should account for setting up dev env on apple silicon.
• pip
  • Problem: pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 20000
    • Not sure if others have faced this issue, but this could also just be my machine.
    • pip was unable to resolve the dev dependencies in about 6 hours, and failed with the above error.
    • This could be because some package versions are not aligned for apple silicon correctly.
  • Solution:
    • Follow the steps mentioned in issue Unable to set up developer environment on Mac M1 #18355 .
    • install and build record
    • install tensorflow-text from here
    • Run pip install -e ".[dev]" --use-deprecated=legacy-resolver
    • Run pip3 install lxml
    • Some package versions were still not correctly resolved, and tests were failing, along with the make fixup, etc commands. So, I had to install specific versions of the following packages from pip: jax==0.4.7, numpy==1.23.
    • onnx had to be installed using instructions here.
  • Action:
    • I need some sanity check for the pip errors. Am I the only one who faced this or is this reproducible for other M2users.
    • Possible update of setup.py to account for apple silicon, and guides in docs.

After these steps, the dev environment is finally working for me along with tests, and other commands. But, this took way too long to get working. I also tried setting up vscode devcontainer for dev dependencies, but jaxlib still does not provide wheels for manylinux aarch64 wheels yet.

How can we proceed here? I want to actively contribute towards solving these issues :)

[xssChauhan]

@mayankagarwals Did you face the issues highlighted in above comment? Link

[sgugger]

For most contributions, you only need to run pip install -e .["quality"], but we do need TensorFlow and Jax for the complete quality checks (as we have many models in both those frameworks). But if you make contributions that do not require them (e.g. you're not touching a TensorFlow or Flax model) you will be fine.

[xssChauhan]

@nateraw I could already start working on completing the migration from decord to pyav .

What do you think about the other set of problems I point out?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[tanaymeh]

Hi @sgugger, I tried installing transformers using pip install -e '.[quality]' and it failed to build the wheel for safetensors.

I am facing this on M1 Pro, Python 3.10

[sgugger]

cc @Narsil

[Narsil]

@tanaymeh

Can you try doing pip install -U safetensors to confirm the bug occurs there ?
Version 0.3.2 should be precompiled for Python 3.10 macos 13 on ARM (m1)...
https://pypi.org/project/safetensors/0.3.2/#files

If the command does work, you should just use that.
If that doesn't work, I'll investigate why your environment is not picking it up.

[jamesbraza]

As of transformers==4.36.2, on macOS 13.5.2 on a MacBook Pro with an M1 chip with Python 3.11.7, I was able to:

pip install -e .[quality,testing,docs_specific,sentencepiece,torch]

This is enough to run some of the pytest cases without ImportErrors

I am still unable to install tensorflow-text as of its latest v2.15.0. tensorflow/text#823 links to some instructions on how to manually build, for those who need it