Python fails to build core extension modules on Xcode-only 10.11 #41085

mikestecker · 2015-06-25T18:34:18Z

I know I'm running a beta, but wanted to report this anyway.

Python fails during the postinstall phase:

==> /usr/local/Cellar/python/2.7.10/bin/python -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/2.7.10/
  File "/usr/local/Cellar/python/2.7.10/libexec/setuptools/setuptools/command/bdist_egg.py", line 463, in make_zipfile
    z = zipfile.ZipFile(zip_filename, mode, compression=compression)
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 736, in __init__
    "Compression requires the (missing) zlib module"
RuntimeError: Compression requires the (missing) lib module

I have the latest Xcode beta installed:

/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
xcode-select version 2342

Here is the output of running brew gist-logs python https://gist.github.com/dc7662dba10b070d2336

[edit @tdsmith Sep 14 2015: Installing the Command Line Tools with xcode-select --install should let you work around this. I would like this to work without the CLT package but won't have time to investigate for a bit; community efforts are welcome!]

The text was updated successfully, but these errors were encountered:

shaneburrell · 2015-06-25T18:35:59Z

Same here.

tdsmith · 2015-06-25T18:53:36Z

What command did you use to install Python? What's xcode-select -p say? The Python build process is failing to build any of the extension modules that depend on the OS X SDK.

shaneburrell · 2015-06-25T18:55:13Z

first brew reinstall python then uninstall and install

command output = /Applications/Xcode-beta.app/Contents/Developer

tdsmith · 2015-06-25T19:07:44Z

@shaneburrell, can you paste logs with brew gist-logs python to be sure we're looking at the same problem?

Does switching the active Xcode configuration to the Command Line Tools (xcode-select -s /Library/Developer/CommandLineTools) help?

shaneburrell · 2015-06-25T19:12:13Z

I only have Xcode 7 installed.

gist-log to follow in a min or two.

tdsmith · 2015-06-25T19:15:31Z

It looks like python's setup.py is failing to detect the libraries and is refusing to even attempt to build them. It would be nice to be able to send a patch for this upstream.

shaneburrell · 2015-06-25T19:15:48Z

https://gist.github.com/e70ac4c2f8bdd1afa756

mikestecker · 2015-06-25T19:17:34Z

When switching to the command line tools using xcode-select -s /Library/Developer/CommandLineTools I get the same errors as I did with the Xcode beta path

tdsmith · 2015-06-25T20:00:24Z

I don't see this on a system with freshly installed CLT and Xcode 7 Beta 2. I'll try Xcode-only later.

shaneburrell · 2015-06-25T20:19:39Z

I'll try with Beta 2 in a bit I was using Beta 1.

mikestecker · 2015-06-25T20:21:56Z

I could have sworn I had Beta 2 installed already... Downloading now and will report back once installed.

mikestecker · 2015-06-25T20:49:00Z

ok confirmed for me, same issues with Beta 2

Hs-Yeah · 2015-07-11T10:55:04Z

[edit: my belief is that this is not true -@tdsmith]

Dive into

/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib

you can see the filename extension of libz* is "tbd"~~, which means "to be done"~~. The same in MacOSX10.10.sdk.

So we should switch to the stable version of CLT and reinstall python:

sudo xcode-select -s /Applications/Xcode.app
brew reinstall python

mattieb · 2015-07-11T20:50:49Z

I was able to build 2.7.10_1 on a system

running 10.10.4
upgraded to the first 10.11 public beta
with Xcode 7 beta 3 and CLT for Xcode 7 beta 3
with xcode-select -s /Library/Developer/CommandLineTools

dstaley · 2015-07-11T22:17:54Z

I was able to just brew install python on a fresh install of 10.11 Public Beta.

tdsmith · 2015-07-11T22:21:21Z

@mikestecker: Are you still having trouble?

erhudy · 2015-07-12T18:19:18Z

I was having this same problem when using Xcode 7 beta 7A121I with the current 10.11 beta (15A216g). After upgrading to Xcode 7 beta 7A152u, I was able to successfully build and install both Python 2.7.10_1 and Python 3.4.3_1.

Hs-Yeah · 2015-07-13T03:45:56Z

@tdsmith ~~I thought the reason of the issue is what I commented above, the zlib module is not finished in Xcode.~~

I found my Xcode 7 is beta 2 (7A121I) ~~and the libz*.dylib is not finished~~. I'm downloading the beta 3 (7A152u) and I will check for the zlib module in it after I finish downloading.

tdsmith · 2015-07-13T04:23:40Z

The .tbd files are functional; see https://forums.developer.apple.com/message/8609#8609. Many people have successfully built the Python extension modules with Xcode 7 on El Capitan. I haven't seen anything pointing to an Xcode problem or insufficiency.

Hs-Yeah · 2015-07-13T12:33:05Z

@tdsmith Thanks for the link, learned something new from the forum. :) So there is no libz.dylib in it Xcode anymore after Xcode 6, right?

Yesterday I ran python, which was updated a few day ago (built with Xcode 7 beta CTL), and it told me No module named zlib.
I found the result of xcode-select -p was /Applications/Xcode-beta.app/Contents/Developer
Then I dived into

/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib

and found no libz.dylib in there.

So I switch to the CTL for Xcode 6.4, in which I can find the libz.dylib. And after reinstalling python, everything went well.

Today I updated Xcode 7 to beta 3 (with no libz.dylib in it too) and ran

sudo xcode-select -s /Applications/Xcode-beta.app

Then I made a reinstallation of python using homebrew reinstall python and the building was successful.
But when I import zlib in python, it told me No module named zlib again.

I think the libz.dylib is needed for the extension modules while building python.

jedisct1 · 2015-07-15T11:02:38Z

Same issue here, with Xcode 7 beta 3.

mattieb · 2015-07-15T12:59:43Z

Did any of you having zlib trouble try installing the Xcode 7 beta 3 CLT and xcode-selecting it?

$ xcode-select -p
/Library/Developer/CommandLineTools
$ otool -L /usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so:
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1223.0.0)
$ python
Python 2.7.10 (default, Jul 11 2015, 16:43:09)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.57.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import zlib
>>> zlib.crc32('foo')
-1938594527

For the record, the xcode-select --install bit @vyazovoi mentions below installs the CLT I'm speaking of here. You can also get it from the Developer site.

paulelms · 2015-07-17T20:00:09Z

Solution (Xcode 7 beta 3):

sudo xcode-select -s /Applications/Xcode-beta.app
xcode-select --install
brew reinstall python

xcode-select --install

bric3 · 2015-07-18T12:43:49Z

Solution of @vyazovoi looks like it works well.

mdarby · 2015-07-21T16:18:21Z

Thanks @vyazovoi!

tdsmith · 2015-08-11T18:04:04Z

This "should" work without the CLT installed but I'm not sure why it fails; I haven't had a chance to build a VM against the most recent 10.11 beta.

mattrobenolt · 2015-09-13T16:51:02Z

Just throwing my two cents in here, I can't get this to work at all with the El Capitan GM paired either with the latest command line tools installed with xcode-select --install or the latest Xcode 7.0 GM seed.

Simsey · 2015-09-13T17:30:14Z

Same here with 10.11 GM + Xcode 7.0 GM.

I also tried to locate the file with

brew cat python | grep zlib
args << "CPPFLAGS=-I#{MacOS.sdk_path}/usr/include" # find lib

..if i jump to the folder there is a /usr/include/zlib.h

jontheophilus · 2015-09-13T21:25:34Z

Same here w/ 10.11 GM + Xcode 7.1 beta.

ColtonProvias · 2015-09-14T17:24:09Z

10.11 Beta 6 + Xcode 7.1 beta also failing here, except for python3 (3.5.0).

https://gist.github.com/anonymous/6d3caeeb0e48c7182fb3

However, it does succeed after the @paulelms solution posted above.

tdsmith · 2015-09-14T17:53:18Z

Added a note to the top comment for easy discoverability:

Installing the Command Line Tools with xcode-select --install should let you work around this. I would like this to work without the CLT package but won't have time to investigate for a bit; community efforts are welcome!

mattrobenolt · 2015-09-14T18:07:09Z

As stated in my comment: #41085 (comment)

It doesn't work for me using the CLT package. :(

tdsmith · 2015-09-14T18:09:33Z

I saw, but without more insight into the root cause it seemed helpful to point out the CLT installation as a workaround because there's positive signal that it helps some (most?) people. :/

Can you post the output of brew gist-logs python? (or python3, if that's your jam)

mattrobenolt · 2015-09-14T18:14:04Z

Will do when I get back to my El Capitan computer. :)

Simsey · 2015-09-14T18:32:02Z

well, I'm sure I also tried it yesterday, but today (now on Xcode 7.1):

brew update
Updated Homebrew from b4537f12 to 8a6e49d9.
==> New Formulae
gxml
==> Updated Formulae
bitrise      jetty    libgee       nghttp2      py3cairo     python3
ffmpeg       keepassc     libtiff      pastebinit   pyenv    treeline
gnutls       lensfun      mypy         ponysay      pyqt5    xonsh
==> Deleted Formulae
honeyd

python -V
Python 2.7.10

Thanks

mattrobenolt · 2015-09-15T06:46:58Z

Ok, yeah, weird. Today it works for me too using the CLT packages. 💻

tdsmith · 2015-09-16T00:37:53Z

The problem is that python's setup.py wants to find the library on disk before it tries to build anything, it only looks inside the Xcode sysroot (if a sysroot is specified), it's looking for a .dylib, and the SDK only has the .tbd file.

This patch extends distutils to find the stubs: https://gist.github.com/tdsmith/015536712a6c0e4b2e5d

tdsmith · 2015-09-16T01:04:33Z

Upstream report: https://bugs.python.org/issue25136

Fixes Homebrew#41085.

Fixes #41085. Closes #44245. Signed-off-by: Tim D. Smith <[email protected]>

DomT4 added python 10.11 labels Jun 25, 2015

mikestecker mentioned this issue Jun 25, 2015

Pango install fail on 10.11 #41086

Closed

DomT4 mentioned this issue Jun 26, 2015

10.11 [Tracking Issue] #40837

Closed

tdsmith changed the title ~~Python fails during postinstall on 10.11~~ Python fails to build core extension modules on 10.11 Jun 29, 2015

zmwangx mentioned this issue Aug 4, 2015

python ImportError: No module named _curses #42411

Closed

helmi72 mentioned this issue Sep 8, 2015

error install python #43720

Closed

tdsmith changed the title ~~Python fails to build core extension modules on 10.11~~ Python fails to build core extension modules on Xcode-only 10.11 Sep 14, 2015

This was referenced Sep 14, 2015

pip3 is not working on OS X 10.11 with Python 3.5 pypa/pip#3110

Closed

pip3 can’t update itself after upgrading to Python 3.5 #43952

Closed

tdsmith mentioned this issue Sep 22, 2015

python3 - pip3 fails on missing zlib #44242

Closed

tdsmith added a commit to tdsmith/homebrew that referenced this issue Sep 22, 2015

python: recognize .tbd stubs

375be3d

Fixes Homebrew#41085.

tdsmith added a commit to tdsmith/homebrew that referenced this issue Sep 22, 2015

python3: recognize .tbd stubs

7eeb8e0

Fixes Homebrew#41085.

tdsmith mentioned this issue Sep 22, 2015

Convince cpythons to recognize .tbd stubs #44245

Closed

tdsmith closed this as completed in 3f76265 Sep 22, 2015

tdsmith added a commit that referenced this issue Sep 22, 2015

python3: recognize .tbd stubs

797990a

Fixes #41085. Closes #44245. Signed-off-by: Tim D. Smith <[email protected]>

Homebrew locked and limited conversation to collaborators Jul 10, 2016

Python fails to build core extension modules on Xcode-only 10.11 #41085

Python fails to build core extension modules on Xcode-only 10.11 #41085

Comments

mikestecker commented Jun 25, 2015

shaneburrell commented Jun 25, 2015

tdsmith commented Jun 25, 2015

shaneburrell commented Jun 25, 2015

tdsmith commented Jun 25, 2015

shaneburrell commented Jun 25, 2015

tdsmith commented Jun 25, 2015

shaneburrell commented Jun 25, 2015

mikestecker commented Jun 25, 2015

tdsmith commented Jun 25, 2015

shaneburrell commented Jun 25, 2015

mikestecker commented Jun 25, 2015

mikestecker commented Jun 25, 2015

Hs-Yeah commented Jul 11, 2015

mattieb commented Jul 11, 2015

dstaley commented Jul 11, 2015

tdsmith commented Jul 11, 2015

erhudy commented Jul 12, 2015

Hs-Yeah commented Jul 13, 2015

tdsmith commented Jul 13, 2015

Hs-Yeah commented Jul 13, 2015

jedisct1 commented Jul 15, 2015

mattieb commented Jul 15, 2015

paulelms commented Jul 17, 2015

bric3 commented Jul 18, 2015

mdarby commented Jul 21, 2015

tdsmith commented Aug 11, 2015

mattrobenolt commented Sep 13, 2015

Simsey commented Sep 13, 2015

jontheophilus commented Sep 13, 2015

ColtonProvias commented Sep 14, 2015

tdsmith commented Sep 14, 2015

mattrobenolt commented Sep 14, 2015

tdsmith commented Sep 14, 2015

mattrobenolt commented Sep 14, 2015

Simsey commented Sep 14, 2015

mattrobenolt commented Sep 15, 2015

tdsmith commented Sep 16, 2015

tdsmith commented Sep 16, 2015