Unable to import module on M1 Mac

[colindean]

When I tried to run a very simple "run this Starlark file":

$ python3 test.py .vela.star
Traceback (most recent call last):
  File "/Users/colin/myprojects/python-poetry/test.py", line 3, in <module>
    from starlark_go import Starlark
  File "/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/__init__.py", line 9, in <module>
    from starlark_go.starlark_go import (  # pyright: reportMissingModuleSource=false
ImportError: dlopen(/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so, 0x0002): tried: '/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))

When I check the library file, I can see that it's compiled in x86_64 mode only:

$ file /Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so
/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so: Mach-O 64-bit dynamically linked shared library x86_64

If it was universal, I'd expect to see Mach-O universal binary with 2 architectures, as alluded by running file on every shared object I've got installed in every Python version I've got installed:

$ cd ~/.pyenv/versions
$ fd '\.so' . --exec file | cut -d : -f 2 | sort | uniq -c
   2 	Mach-O 64-bit bundle arm64
   2 	Mach-O 64-bit bundle x86_64
1189  Mach-O 64-bit bundle arm64
   2  Mach-O 64-bit bundle x86_64
   1  Mach-O 64-bit dynamically linked shared library x86_64
   2  Mach-O universal binary with 2 architectures

Any other Mach-O 64-bit dynamically linked shared library x86_64?

$ fd '\.so' . --exec file | rg 'Mach-O 64-bit dynamically linked shared library x86_64'
./3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so: Mach-O 64-bit dynamically linked shared library x86_64

No, just starlark_go.

Only others I've got installed shipping universal are MarkupSafe and uvloop:

$ fd '\.so' . --exec file | rg 'Mach-O universal binary with 2 architectures'
./3.10.2/envs/spiderbot/lib/python3.10/site-packages/uvloop/loop.cpython-310-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64]
./3.10.2/envs/spiderbot/lib/python3.10/site-packages/markupsafe/_speedups.cpython-310-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64Mach-O 64-bit bundle arm64]

The universal wheel for version 0.1.1 was installed:

$ pip install starlark-go
Looking in indexes: <internal repo URLs redacted>
Collecting starlark-go
  Downloading https://binrepo.mycompany.com/artifactory/api/pypi/pypi-remote/packages/packages/35/1d/7a3104a0038e5a77b20faec9cd2b110053304f8706369f6aa3db57ee0d11/starlark_go-0.1.1-cp39-cp39-macosx_10_9_universal2.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 2.1 MB/s eta 0:00:00
Installing collected packages: starlark-go
Successfully installed starlark-go-0.1.1

My test runner

I'm not sure is even correct because of the import error:

import sys

from starlark_go import Starlark

s = Starlark()

with open(sys.argv[1], 'r') as input_file:
    s.exec(input_file.read())

print(s.eval("main([])"))

Versions:

$ sw_vers 
ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115
$ uname -a
Darwin mycomputer 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64
$ python --version
Python 3.9.11

Possible remedy

It looks like uvloop is using the ~same tool starlark-go is using to build, so maybe it can be adapted.

[colindean]

Unfortunately, it looks like v0.1.2 didn't fix the problem:

Updating:

$ pip install -U starlark-go
Looking in indexes: https://binrepo.mycompany.com/artifactory/api/pypi/pypi-remote/simple, https://binrepo.mycompany.com/artifactory/api/pypi/tgt-python/simple
Requirement already satisfied: starlark-go in /Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages (0.1.1)
Collecting starlark-go
  Downloading https://binrepo.mycompany.com/artifactory/api/pypi/pypi-remote/packages/packages/5d/76/14817499732da00a0781ad26755ecd5c2697aa41e8a20b10bb6620ed8de4/starlark_go-0.1.2-cp39-cp39-macosx_10_9_universal2.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 5.6 MB/s eta 0:00:00
Installing collected packages: starlark-go
  Attempting uninstall: starlark-go
    Found existing installation: starlark-go 0.1.1
    Uninstalling starlark-go-0.1.1:
      Successfully uninstalled starlark-go-0.1.1
Successfully installed starlark-go-0.1.2

Running my test runner again, unchanged since ☝️:

$ python test.py .vela.star 
Traceback (most recent call last):
  File "/Users/colin/Source/MyCompany/MyCompanyOSS/python-poetry/test.py", line 3, in <module>
    from starlark_go import Starlark
  File "/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/__init__.py", line 9, in <module>
    from starlark_go.starlark_go import (  # pyright: reportMissingModuleSource=false
ImportError: dlopen(/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so, 0x0002): tried: '/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so' (no such file), '/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

File inspection:

/Users/colin/.pyenv/versions/3.9.11/lib/python3.9/site-packages/starlark_go/starlark_go.cpython-39-darwin.so: Mach-O 64-bit dynamically linked shared library x86_64

Looking at the build history, I'd expect to see something here affirming the compilation mode but I admit I don't know enough about Go compilation to understand if something in there is implying universal build versus compile platform architecture build.

[jordemort]

Oh hm, yeah that probably bears more thought; with the Go stuff I'm not sure if we can actually use universal2, or if we need to build separate arch wheels after all. Unfortunately I've started a new contract this week that's going to keep me pretty busy for a while; I'll try to loop back to it if I have time but it could be a bit.

[colindean]

It seems from the cibuildwheel documentation that it may be necessary to set CIBW_ARCHS_MACOS explicitly in order to get a universal binary

[colindean]

It seems from the cibuildwheel documentation that it may be necessary to set CIBW_ARCHS_MACOS explicitly in order to get a universal binary. I put up #131 as an attempt to fix this by switching to OS-specific arch lists.