Document and fix Python installation problems on Big Sur #22497

armenzg · 2020-12-07T17:17:56Z

This issue is to track all known issues to doing development of Sentry on Big Sur (Mac's latest OS update). I will update the description of this issue as issues are documented and the associated workaround. If possible, I will look to file upstream issues where appropriate.

Known issues

pyenv cannot install Python 3.6.10 w/o custom command. See comment
- LANDED
- Here's the command: make setup-pyenv
  - Equivalent to:

# pyenv 1.2.22 and greater (no need for LDFLAGS
pyenv install --skip-existing --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# Before pyenv 1.2.22
`LDFLAGS="-L$(xcrun --show-sdk-path)/usr/lib ${LDFLAGS}" pyenv install --skip-existing --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)`

brew install librdkafka seems to be necessary. See comment
- LANDED

If you don't use the Python version installed via pyenv as we recommend you might encounter this issue:

Failing to install psycopg2-binary
- Workaround SYSTEM_VERSION_COMPAT=1 pip install psycopg2-binary

Upstream issues

(pypa/packaing) [MERGED] add support for macos 11.0, arm64, universal2 #319
- Released as part of 20.5 or 20.7
[Waiting for release to remove hack] Add LDFLAGS for zlib on macOS >= 1100 #1711
psycopg2 issue

The text was updated successfully, but these errors were encountered:

armenzg · 2020-12-07T21:20:19Z

I've installed the Python 3.9 release candidate and it fails in the building of the wheels (as I recalled).

I will determine which packages are having trouble.

Here's the output of one of the failures (this one seems to think we're on mac 10.9; am I reading this right?):

Building wheels for collected packages: confluent-kafka, exam, googleapis-common-protos, grpc-google-iam-v1, hiredis, lxml, maxminddb, mmh3, msgpack, openapi-core, parse, parsimonious, petname, phabricator, Pillow, pyrsistent, python-u2flib-server, PyYAML, rb, setproctitle, simplejson, strict-rfc3339, toronado, uwsgi, wrapt, xmlsec
  Building wheel for confluent-kafka (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/armenzg/code/sentry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/setup.py'"'"'; __file__='"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-wheel-qskfoqi8
       cwd: /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/
  Complete output (54 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.9
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/serializing_producer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  copying confluent_kafka/deserializing_consumer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/avro.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  copying confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/schema_registry
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/serialization
  copying confluent_kafka/serialization/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/serialization
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/admin
  copying confluent_kafka/admin/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/admin
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/error.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  copying confluent_kafka/avro/load.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/kafkatest
  creating build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.9-x86_64-3.9/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.macosx-10.9-x86_64-3.9
  creating build/temp.macosx-10.9-x86_64-3.9/private
  creating build/temp.macosx-10.9-x86_64-3.9/private/var
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka
  creating build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Users/armenzg/code/sentry/.venv/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.c -o build/temp.macosx-10.9-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.o
  In file included from /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/Admin.c:17:
  /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-7knl61vt/confluent-kafka_54b9cd4f587f425db719898dcdffc9ee/confluent_kafka/src/confluent_kafka.h:22:10: fatal error: 'librdkafka/rdkafka.h' file not found
  #include <librdkafka/rdkafka.h>
           ^~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for confluent-kafka

armenzg · 2020-12-07T21:26:52Z

I've been told that another engineer on Big Sur all they had to do is:

brew install librdkafka
brew install zlib
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

I will verify this.

wedamija · 2020-12-07T22:16:14Z

Here's the output of one of the failures (this one seems to think we're on mac 10.9; am I reading this right?):

@taylangocmen Had similar output to that, I don't remember if we figured out why.

armenzg · 2020-12-08T15:57:48Z

With the OS' Python 3:

❯ /usr/bin/python3 --version
Python 3.8.2
❯ /usr/bin/python3 -m pip install virtualenv
❯ /usr/bin/python3 -m pip install --user --upgrade pip
❯ /usr/bin/python3 -m virtualenv .venv
❯ source .venv/bin/activate
❯ make install-py-dev

It fails to build a wheel for uwsgi

clang: error: no such file or directory: '/Users/armenzg/code/sentry/.venv/lib/python3.8/config-3.8/libpython3.8.a'
  *** error linking uWSGI ***
  ----------------------------------------
  ERROR: Failed building wheel for uwsgi

Having installed the Python 3.9 RC1 installation gets on the way (this shows that having multiple ways of installing Python can get on the way and each developer can end up on a different unique situation):

sentry on  big_sur_wheels [!] via 🐍 v3.8.2 (.venv)
❯ python-config --ldflags
-L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation

sentry on  big_sur_wheels [!] via 🐍 v2.7.16
❯ python3-config --ldflags
-L/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

I had to unistall it from /Applications and sudo mv /Library/Frameworks/Python.framework Library_Frameworks_Python.framework.

I decided to use brew to get me Python and go from there:

❯ which python3
/usr/local/bin/python3
❯ python3 --version
Python 3.9.0
❯ python3 -m virtualenv .venv
❯ source .venv/bin/activate
❯ pip install uwsgi
# This worked! Let's track the cflags and ldflags
❯ python3-config --cflags
-I/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers

❯ python3-config --ldflags
-L/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/lib/python3.9/config-3.9-darwin -ldl -framework CoreFoundation

Using the brewed Python version I got a lot further:

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/Users/armenzg/code/sentry/.venv/include -I/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.c -o build/temp.macosx-11-x86_64-3.9/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.o
    In file included from /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/Admin.c:17:
    /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/confluent_kafka/src/confluent_kafka.h:22:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/armenzg/code/sentry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/setup.py'"'"'; __file__='"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-5yzfoz7n/confluent-kafka_cc39abbaae5547579487ad6f3891b466/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-record-3_x6_1nh/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/sentry/.venv/include/site/python3.9/confluent-kafka Check the logs for full command output.

To resolve this running brew install librdkafka seemed sufficient, however, I got this message:

DEPRECATION: confluent-kafka was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.

Conclusion:

Use Python from brew (so far I've tried with Python 3.9)
brew install librdkafka

I need to test with Python 3.6 and see what the situation is for pyenv.

armenzg · 2020-12-08T21:12:28Z

Before I started, the following comment I removed all packages installed via brew to start from a known state. I then installed pyenv through brew.

Pyenv

Install recommended build dependencies:

brew install openssl readline sqlite3 xz zlib

Build Python 3.6.10 with pyenv

armenzg@Armens-MacBook-Pro sentry % pyenv install 3.6.10
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.6.10.tar.xz...
-> https://www.python.org/ftp/python/3.6.10/Python-3.6.10.tar.xz
Installing Python-3.6.10...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.0.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/python-build.20201208134740.78464
Results logged to /var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/python-build.20201208134740.78464.log

Last 10 log lines:
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10432:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/armenzg/.pyenv/versions/3.6.10/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include -I/Users/armenzg/.pyenv/versions/3.6.10/include   -DPy_BUILD_CORE  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

This can be resolved by using CLAGS, LDFLAGS with --patch (step 1 from this comment is not necessary). I also had to brew install zlib since I have not run brew bundle :

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.6.10 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
# There's also this warning:
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?

This set up seems to work.

Brew installation

# We can't get to 3.6.10 anymore
brew install [email protected]
/usr/local/Cellar/[email protected]/3.7.9_2/bin/python3 -m pip install virtualenv
/usr/local/Cellar/[email protected]/3.7.9_2/bin/python3 -m virtualenv .venv
source .venv/bin/activate
make install-py-dev

This set up seems to work.

The platform for each Python installed

pyenv, System & brew installed Python's platform:

% /usr/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit
% /usr/local/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit
% ~/.pyenv/shims/python3 -c "import platform; print(platform.platform())"
Darwin-20.1.0-x86_64-i386-64bit

joshuarli · 2020-12-09T19:53:09Z

I highly recommend focusing on getting pyenv to work, with a high tolerance for workarounds. Brew python's burned us in the past, especially when they removed 2.7, and you can't get patch version granularity. I'd also avoid system python, you absolutely are at the mercy of their version too and in general I don't mess around with system python because it's easy to break in silent + unexpected ways.

With pyenv we have full control over versions and can easily upgrade people with the tooling revolving around .python-version.

evanpurkhiser · 2020-12-09T20:13:30Z

Agreed with josh. pyenv is the way to go. If there are problems with it on Big Sur I would work to resolve those, even if it means contributing to pyenv.

Installing Python via pyenv on Big Sur for older Python versions requires: * Setting LDFLAGS for zlib (pyenv/pyenv#1711) * Patching the source code Once a new release of pyenv is released and we use a newer Python 3 version we will not need all these hacks and all Mac versions will be able to use the same code paths. Fixes #22497

armenzg · 2020-12-09T20:53:29Z

Remove all brew dependencies:

brew remove --force $(brew list --formula) --ignore-dependencies
brew remove --cask --force $(brew list --cask) --ignore-dependencies

Remove venv and pyenv:

rm -rf ~/.pyenv && rm -rf .venv
Install dependencies:
brew bundle

Installing pyenv with make setup-pyenv (with local patch) seems to work, however, there's no symlink at ~/.pyenv/bin.

For the curious:

% echo $PATH
/Users/armenzg/.pyenv/bin:/Users/armenzg/.pyenv/shims:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/armenzg/.volta/bin:/Users/armenzg/.pyenv/bin:/Users/armenzg/.pyenv/shims

Meanwhile, while I figure out the above:

~/.pyenv/versions/3.6.10/bin/python3 -m pip install --upgrade pip
~/.pyenv/versions/3.6.10/bin/python3 -m pip install virtualenv
~/.pyenv/versions/3.6.10/bin/python3 -m virtualenv .venv
source .venv/bin/activate
make install-py-dev
make bootstrap
sentry devserver --workers

librdkafka did not seem necessary.

armenzg · 2020-12-09T21:01:02Z

Note: The PR is a hack until a new pyenv is released and we upgrade the Python version we use.

Installing Python via pyenv on Big Sur for older Python versions requires: * Setting LDFLAGS for zlib (pyenv/pyenv#1711) * Patching the source code Once a new release of pyenv is released and we use a newer Python 3 version we will not need all these hacks and all Mac versions will be able to use the same code paths. Fixes #22497

armenzg · 2020-12-10T15:00:54Z

I've discovered why pyenv was not fully setting the Python versions correctly. Unless all Python versions specified within .python-version are defined, it won't change the version.

I've updated the PR to install Python 2.7 as well.

Here's the verification of the steps:

make setup-pyenv
python3 -m pip install --upgrade pip
python3 -m pip install virtualenv
python3 -m virtualenv .venv
source .venv/bin/activate
make install-py-dev
make bootstrap
sentry devserver --workers

Installing Python via pyenv on Big Sur for older Python versions requires: * Setting LDFLAGS for zlib (pyenv/pyenv#1711) * Patching the source code Once a new release of pyenv is released and we use a newer Python 3 version we will not need all these hacks and all Mac versions will be able to use the same code paths. Fixes #22497

joshuarli · 2020-12-11T20:56:56Z

@armenzg librdkafka is indeed needed, you probably fixed it and it was installed from cache.

In my case, pip 20.3.1 thinks a cached 10_6 wheel is sufficient. pypi doesn't yet have big sur wheels for confluent-kafka.

To see this issue you have to force wheel compilation from remote sources:

pip install --no-cache-dir --no-binary :all: confluent-kafka

So, for now, you should add librdkafka to the Brewfile.

armenzg · 2020-12-11T21:05:54Z

Thanks to your tip I decided to run this pip install --no-cache-dir --no-binary :all: -e ".[dev]"

I will investigate:

Collecting psycopg2-binary<2.9.0,>=2.7.0
  Downloading psycopg2-binary-2.8.6.tar.gz (384 kB)
     |████████████████████████████████| 384 kB 7.1 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /Users/armenzg/code/sentry/.venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-4uwq9hfc/psycopg2-binary_b19e432a261845248a25c80873844183/setup.py'"'"'; __file__='"'"'/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-4uwq9hfc/psycopg2-binary_b19e432a261845248a25c80873844183/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6
         cwd: /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-install-4uwq9hfc/psycopg2-binary_b19e432a261845248a25c80873844183/
    Complete output (23 lines):
    running egg_info
    creating /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6/psycopg2_binary.egg-info
    writing /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6/psycopg2_binary.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6/psycopg2_binary.egg-info/dependency_links.txt
    writing top-level names to /private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6/psycopg2_binary.egg-info/top_level.txt
    writing manifest file '/private/var/folders/zr/vhvlh_t57vz0_qkfwh9fmlm00000gn/T/pip-pip-egg-info-gyx3vjg6/psycopg2_binary.egg-info/SOURCES.txt'
    
    Error: pg_config executable not found.
    
    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:
    
        python setup.py build_ext --pg-config /path/to/pg_config build ...
    
    or with the pg_config option in 'setup.cfg'.
    
    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

joshuarli · 2020-12-11T21:11:22Z

@armenzg You don't need to investigate that. Let pip prefer remote wheels.

$ pip install --no-cache-dir psycopg2-binary
Collecting psycopg2-binary
  Downloading psycopg2_binary-2.8.6-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 4.0 MB/s
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.8.6

Actually there isn't an 11 wheel here... does postgres work with this? Most tests involve postgres.

joshuarli · 2020-12-11T21:13:04Z

Yeah so, tests/sentry/db/postgres/test_base.py works for me.

I'd go ahead and take a guess that most 10_x wheels still work on Big Sur, so don't worry about this.

armenzg · 2020-12-11T21:14:34Z

I'm wondering if this would affect developers starting from a clean slate.
I've found SYSTEM_VERSION_COMPAT=1 as a way to work around it.

armenzg · 2020-12-11T21:21:48Z

sentry on  more_big_sur [!] via ⬢ v12.19.0 via 🐍 v3.6.10
❯ /usr/bin/python3  -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit

sentry on  more_big_sur [!] via ⬢ v12.19.0 via 🐍 v3.6.10
❯ SYSTEM_VERSION_COMPAT=1 /usr/bin/python3  -c "import platform; print(platform.platform())"
macOS-10.16-x86_64-i386-64bit

There's currently no wheel for confluent-kafka, thus, we need librdkafka to build it. This can be removed once there's a wheel for it. Fixes #22497

There's currently no wheel for confluent-kafka, thus, we need librdkafka to build it from the source. This can be removed once there's a wheel for it. Fixes #22497 Co-authored-by: josh <[email protected]>

armenzg self-assigned this Dec 8, 2020

armenzg added the Component: Developer Environment This covers issues related to setting up a developer's environment label Dec 8, 2020

armenzg mentioned this issue Dec 9, 2020

build(feat): Support pyenv/Python installation on Big Sur #22257

Merged

armenzg changed the title ~~Document BigSur development environment issues~~ Fix pyenv 3.6.10 installation on Big Sur Dec 9, 2020

armenzg closed this as completed in #22257 Dec 11, 2020

armenzg reopened this Dec 11, 2020

armenzg changed the title ~~Fix pyenv 3.6.10 installation on Big Sur~~ Document and fix Python installation problems on Big Sur Dec 11, 2020

armenzg added a commit that referenced this issue Dec 11, 2020

build(fix): Include librdkafka in order to support Big Sur

a0504df

There's currently no wheel for confluent-kafka, thus, we need librdkafka to build it. This can be removed once there's a wheel for it. Fixes #22497

armenzg mentioned this issue Dec 14, 2020

build(fix): Include librdkafka in order to support Big Sur #22645

Merged

armenzg closed this as completed in #22645 Dec 14, 2020

github-actions bot locked and limited conversation to collaborators Jan 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document and fix Python installation problems on Big Sur #22497

Document and fix Python installation problems on Big Sur #22497

armenzg commented Dec 7, 2020 •

edited

Loading

armenzg commented Dec 7, 2020

armenzg commented Dec 7, 2020

wedamija commented Dec 7, 2020

armenzg commented Dec 8, 2020

armenzg commented Dec 8, 2020

joshuarli commented Dec 9, 2020

evanpurkhiser commented Dec 9, 2020

armenzg commented Dec 9, 2020

armenzg commented Dec 9, 2020

armenzg commented Dec 10, 2020

joshuarli commented Dec 11, 2020

armenzg commented Dec 11, 2020

joshuarli commented Dec 11, 2020

joshuarli commented Dec 11, 2020

armenzg commented Dec 11, 2020

armenzg commented Dec 11, 2020

Document and fix Python installation problems on Big Sur #22497

Document and fix Python installation problems on Big Sur #22497

Comments

armenzg commented Dec 7, 2020 • edited Loading

Known issues

Upstream issues

armenzg commented Dec 7, 2020

armenzg commented Dec 7, 2020

wedamija commented Dec 7, 2020

armenzg commented Dec 8, 2020

armenzg commented Dec 8, 2020

Pyenv

Brew installation

The platform for each Python installed

joshuarli commented Dec 9, 2020

evanpurkhiser commented Dec 9, 2020

armenzg commented Dec 9, 2020

armenzg commented Dec 9, 2020

armenzg commented Dec 10, 2020

joshuarli commented Dec 11, 2020

armenzg commented Dec 11, 2020

joshuarli commented Dec 11, 2020

joshuarli commented Dec 11, 2020

armenzg commented Dec 11, 2020

armenzg commented Dec 11, 2020

armenzg commented Dec 7, 2020 •

edited

Loading