AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'linux_aarch64')

[RomanGurinovich]

Experiencing this error when pip install numpy and cryptography in Docker container on Raspberry. While images built in May and June of 2020 worked well. Tried and it didn't work:

• downgrading versions of wheel and packages,

• vice versa, upgrading pip, wheel and packages to the very latest versions

• pip install cryptography --no-binary cryptography

Maybe it is an arm64 issue. For instance, cryptography doesn't list this platform as supported https://cryptography.io/en/latest/installation/#build-on-linux

---

Workaround

pip install -e git+https://github.com/pyca/cryptography.git#egg=cryptography

---

Details

      File "/tmp/pip-build-env-n8ueeulb/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag
        assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
    AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'linux_aarch64')
  ...
  ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly

OS : Raspbian GNU/Linux 10 (buster)
Docker: 19.03.12
Base: python:3.8-buster

[agronholm]

Please paste the output of the following script (this requires the packaging library installed):

from packaging import tags
from pprint import pprint

pprint(list(tags.sys_tags()))

[RomanGurinovich]

@agronholm, here you are

pprint(list(tags.sys_tags()))
[<cp38-cp38-linux_armv7l @ 4146360008>,
 <cp38-abi3-linux_armv7l @ 4146423816>,
 <cp38-none-linux_armv7l @ 4146423784>,
 <cp37-abi3-linux_armv7l @ 4146357992>,
 <cp36-abi3-linux_armv7l @ 4146423944>,
 <cp35-abi3-linux_armv7l @ 4145553704>,
 <cp34-abi3-linux_armv7l @ 4145553768>,
 <cp33-abi3-linux_armv7l @ 4145556008>,
 <cp32-abi3-linux_armv7l @ 4145556104>,
 <py38-none-linux_armv7l @ 4145556360>,
 <py3-none-linux_armv7l @ 4145556200>,
 <py37-none-linux_armv7l @ 4145556424>,
 <py36-none-linux_armv7l @ 4145556520>,
 <py35-none-linux_armv7l @ 4145556616>,
 <py34-none-linux_armv7l @ 4145556712>,
 <py33-none-linux_armv7l @ 4145556808>,
 <py32-none-linux_armv7l @ 4145556872>,
 <py31-none-linux_armv7l @ 4145556968>,
 <py30-none-linux_armv7l @ 4145557064>,
 <py38-none-any @ 4145557160>,
 <py3-none-any @ 4145557256>,
 <py37-none-any @ 4145557416>,
 <py36-none-any @ 4145555752>,
 <py35-none-any @ 4145598600>,
 <py34-none-any @ 4145598728>,
 <py33-none-any @ 4145598856>,
 <py32-none-any @ 4145598984>,
 <py31-none-any @ 4145599112>,
 <py30-none-any @ 4145599240>]

[agronholm]

If you have previously built wheels around, is their platform tag also linux_aarch64?

[RomanGurinovich]

seems so

  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.8.5-cp38-cp38-linux_aarch64.whl

[agronholm]

It looks like wheel has been using the wrong platform tag all along for this particular case (64-bit ARM running a 32-bit OS). It's the one corner case not covered by the logic in wheel's get_platform().

[agronholm]

Please test the fix in the associated PR.

[RomanGurinovich]

Alex, thank you. Unfortunately, the error persists. Exact steps:

1. On host

$ docker run -it python:3.8-buster

2. Getting inside container

$ docker exec -it container_id /bin/bash

3. Installing new wheel inside the container

$ pip install git+https://github.com/pypa/wheel.git@b07333d62d333063e8d060fe66f6d3d26378b357
$ pip show wheel
Version: 0.35.1

4. Installing any package

$ pip install cffi
    File "/usr/local/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 283, in get_tag
      assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
  AssertionError: would build wheel with unsupported tag ('cp38', 'cp38', 'linux_aarch64')

---

By the way, both host and container OS are reported as 64 bit, if I interpret it correctly

$ uname -m
aarch64

[agronholm]

Are you suggesting that I could reproduce this without ARM hardware?

[agronholm]

I remember that I have a bunch of raspberry pis at my disposal, but they're running Raspbian 9. They report distutils.util.get_platform() == 'linux-armv7l' as expected. I'm thoroughly confused now. What do you get from that, and also sys.maxsize?

[RomanGurinovich]

turns out hyphen, not underscore ) Could you please PR? (Tried, insufficient permissions)

>>> import distutils.util
>>> result = distutils.util.get_platform()
>>> result
'linux-aarch64'

[agronholm]

I didn't understand anything about your last message. Are you running a 64 bit OS or not? If you are, then it may be that packaging.tags has a bug.

[RomanGurinovich]

there is check elif result == "linux_aarch64" while distutils.util.get_platform() returns linux-aarch64 with hyphen between terms

[agronholm]

Got it. Try again now please.

[RomanGurinovich]

Tried, assertion error didn't go, unfortunately. Then tried to move up from the assert tag in supported_tags line.
I didn't dig down to the bottom, please, treat the next experiments as a fast kludge.

1. For sure the latest change made get_platform() to return tag linux-armv7l as intended.

2. But. Somehow at the very beginning of theget_tag() function
   self.plat_name is linux-aarch64
   while
   get_platform(self.bdist_dir) is linux-armv7l
   thus added explicit assigning to make sure that plat_name is set

if self.plat_name and not self.plat_name.startswith("macosx"):
    self.plat_name = get_platform(self.bdist_dir)
    plat_name = self.plat_name

With this very strange fix cryptography was built finally when running ~/cryptography# python3 setup.py bdist_wheel.
Does it make sense?

[mattip]

What computer are you using? Is this anyhow connected to qemu or a docker image? When you run uname -a; python3 -c "import sys; print(sys.version); print('platform', sys.platform)" what is the output?

[RomanGurinovich]

yes, Docker on Raspberry

On host

Linux raspberrypi 4.19.75v64 #1 SMP PREEMPT Sat Oct 5 03:06:01 CEST 2019 aarch64 GNU/Linux
3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0]
platform linux

In container

Linux 53888f3410a6 4.19.75v64 #1 SMP PREEMPT Sat Oct 5 03:06:01 CEST 2019 aarch64 GNU/Linux
3.8.5 (default, Aug  4 2020, 14:33:56) 
[GCC 8.3.0]
platform linux

[mattip]

It seems both inside and outside the docker you are using aarch64. What does python3 -c "import sys, distutils.util; print(sys.maxsize); print(distutils.util.get_platform()); "

[RomanGurinovich]

2147483647
linux-aarch64

---

by the way, noticed that similar command pip3 install xyz uses different wheel for i.e. numpy and cffi, even after reinstalling of the wheel

numpy

File "/tmp/pip-build-env-850ieytl/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 278, in get_tag

cffi

File "/usr/local/lib/python3.8/site-packages/wheel-0.35.1-py3.8.egg/wheel/bdist_wheel.py", line 283, in get_tag

Maybe that is why only some packages failed?

I'll be back tomorrow

[agronholm]

Why is there an .egg of wheel in site-packages?

[agronholm]

FWIW, on Raspberry Pi 3 + Raspbian 9, distutils.util.get_platform() reports linux-armv7l as expected, at least on Python 3.7.3.

[mattip]

So you are using a 32-bit python on 64-bit arm.

@agronholm this is on a raspberrypi 4 which supports 64 bit OSes. BTW, in the comment "Return a different tag for a 64-bit CPU running a 32-bit OS" it should be "Return a different tag for a 64-bit OS running a 32-bit python"

[agronholm]

Raspberry Pi 3 also has a 64-bit CPU, but Raspbian is only available in 32 bits. And yes, that would seem to be a more accurate comment.

[agronholm]

So the difference seems to be that the OS is 64-bit but the Python binary is 32-bit?

[agronholm]

If that is the case, this begs the question: why does distutils.util.get_platform() return an incorrect platform tag?

[mattip]

Because it parses uname -a, not any sysconfig variable. packaging.tags has a _platform_tag() but unfortunately it is not public. I opened an issue with packaging

[agronholm]

@RomanGurinovich since I do not currently have access to an aarch64 machine with a 64-bit OS, I need you to figure out where the aarch64 platform name is coming from. Can you do that?

[RomanGurinovich]

let me try

[nioncode]

I was able to work around it by setting _PYTHON_HOST_PLATFORM=linux_armv7l

Just to clarify for others that might be struggling with where to set this: this should be set with ENV in the Dockerfile before doing the pip install step, e.g.:

ENV _PYTHON_HOST_PLATFORM linux_armv7l

This was required to build some wheels in Docker on CoreELEC, which uses a 64bit kernel with 32bit userspace as well.

[agronholm]

So where did we end up with this? I don't own the necessary hardware to test this.

[RomanGurinovich]

it builds with @nioncode instruction

[agronholm]

I now have a rpi3 in my possession so I could try to figure out if there's anything wheel can do about this.

[agronholm]

I can't reproduce this. Can you @RomanGurinovich repro it now? For me, python:3.8-buster downloads the aarch64 build of Python and it successfully builds aarch64 wheels.

[mattip]

I think the original issue was about building 32-bit wheels inside a docker, so it should not be building aarch64 wheels.

[agronholm]

I followed OP's instructions but could not reproduce the error. I don't know what to do.

[nioncode]

@agronholm are you running a 64bit kernel with 32bit userspace? Last time I checked (with CoreELEC and docker) this was broken for me as described in #367 (comment)

[agronholm]

I'm running 64-bit Raspberry Pi OS beta. It has 64-bit kernel and userspace. I guess I need this specific CoreELEC distro then...?

[agronholm]

Or maybe I can repro this by running the linux/arm/v7 variant of the python docker image.

[nioncode]

If you run 64bit userspace on a 64bit kernel, everything works fine (this is more or less the default configuration similar to amd64, just on arm). Maybe you could reproduce with a 32bit raspbian, but I cannot guarantee that, since I only worked with CoreELEC before, which is a bit more work to set up I guess.

[agronholm]

Well, I tried running the armv7 variant of Python there (checked that sys.maxsize indicated 32-bit) and to my surprise it still built an aarch64 wheel, no errors. The library worked fine too.

[agronholm]

OP said the problem occurred on Raspbian 10 so I'll try with that.

[RomanGurinovich]

Alex, I'll have access to the same device next week and will try to reproduce

[agronholm]

This is what I did on Raspbian 10:

docker run --rm python:3.8 pip install cbor2

It built a linux_armv7l wheel as expected. I could not build a cryptography wheel due to the lack of a Rust compiler.

[agronholm]

Oh, this was Raspbian 9, not 10. Does that make a difference?

[agronholm]

To make sure, I repeated this process with Raspbian 10, and got the same results. Everything works as expected.

[agronholm]

I even downgraded wheel to v0.35.1 which was the current version at the time, and it still worked as expected.

[RomanGurinovich]

Good news, everyone,

Just setting FROM python:3.9-buster explicitly solves the issue. Most likely, in conjunction with the recent changes in the packages. I didn't dive deep into what else changed since that time.

In this setup _PYTHON_HOST_PLATFORM=linux_armv7l is not needed anymore.
Side note, for the current cryptography one might set envvar CRYPTOGRAPHY_DONT_BUILD_RUST=1 pyca/cryptography#5771 (comment).

At the same time, running previous setup failed as before.

Nothing else changed on the host or Docker

$ hostnamectl
  Operating System: Raspbian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.75v64
      Architecture: arm64

$ docker --version
Docker version 19.03.12, build 48a6621

$ lscpu
Architecture:        aarch64
Model name:          Cortex-A72

$ python3
>>> platform.architecture()
('32bit', 'ELF')

Thus for the combination of the recent versions of the language and packages everything is fine

[agronholm]

Where did you get this OS image? This is what I have:

  Operating System: Raspbian GNU/Linux 10 (buster)
            Kernel: Linux 5.4.83-v7+
      Architecture: arm

[agronholm]

I installed the 64-bit beta of Raspbian 10, but even that has both 64-bit kernel and 64-bit userland.

[agronholm]

I tried piCorePlayer too but I have no clue on how to operate it.

[agronholm]

I would really welcome a clear set of repro instructions. It's been over 6 months since this was opened and I still haven't been able to repro the problem.

[mattip]

It seems like the people who reported this issue are not interested in helping @agronholm pursue a reproducer for it, so perhaps the issue should be closed.

[n8henrie]

If anyone is interested in re-opening, I've been struggling with this for a week or so and documented what is (for me) a reproducible issue here.

With current versions of pip and wheel, running on Buster with 64b kernel and 32b userland, python 3.9.4, I'm able to build an aarch64 wheel (for numpy) but unable to install it.