Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

macos moduleset updates for 3.0 #7

Closed
totaam opened this issue Aug 12, 2019 · 23 comments
Closed

macos moduleset updates for 3.0 #7

totaam opened this issue Aug 12, 2019 · 23 comments

Comments

@totaam
Copy link
Collaborator

totaam commented Aug 12, 2019

See Xpra-org/xpra#1985 for 2.5, and pull from ​[https://github.com/GNOME/gtk-osx]

Since v3 will be supported for years, we have to try to stay close to the upstream stable moduleset.
@totaam
Copy link
Collaborator Author

totaam commented Aug 12, 2019 •
edited
Loading

After merging from upstream in r23490 + r23491 + r23492 + r23493, jhbuild did the usual thing: fail with an utterly unhelpful message, very much like Xpra-org/xpra#1392.
Why are they so keen on swallowing exceptions everywhere?
Adding print statements in various places, it turns out that the real cause is this:

failed to parse https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules: unknown module type meson

@totaam
Copy link
Collaborator Author

totaam commented Aug 12, 2019 •
edited
Loading

The meson issue is because we have to re-bootstrap gtk-osx... More info here: https://gitlab.gnome.org/GNOME/gtk-osx/tree/master

Also, to workaround the https issues with the old version of wget found in macos, just install a newer one (TODO: move this to moduleset):

curl -O https://ftp.gnu.org/gnu/wget/wget-1.20.3.tar.gz
tar -zxvf wget-1.20.3.tar.gz
cd wget-1.20.3
./configure --with-ssl=openssl --prefix=$JHBUILD_PREFIX
make
make install

Then we also have to set CA_CERTIFICATE.

@totaam
Copy link
Collaborator Author

totaam commented Aug 12, 2019 •
edited
Loading

To make this work (PITA):

  • had to pull a new version of jhbuild by hand and install it - why?
  • had to install meson by hand - why?
    That's OK for the python3 / gtk3 variant, but for the python2 / gtk2 environment, there is no python3 there and meson requires it...
    During setup the gtk-osx-setup.sh script said something about installing a pipenv with python3, somewhere? who knows?
    Do I have to restore the old modules and make a GTK2 URL for it?

@totaam
Copy link
Collaborator Author

totaam commented Aug 12, 2019 •
edited
Loading

Then later, glib complained that ninja was not installed.. Yay, yet another build system to deal with. (looking at the setup script, it should have been pulled automatically, oh well)
And this one requires a weird incantation to install (and still complained about something):

python3 ./configure.py --bootstrap
cp ninja $JHBUILD_PREFIX/bin

@totaam
Copy link
Collaborator Author

totaam commented Aug 14, 2019 •
edited
Loading

Other things that broke: JHBUILD_PREFIX/lib/charset.alias went MIA.

Starting again from scratch on a clean new 10.11 "El Capitan" VM - first for GTK2:

  • sh gtk-osx-setup.sh - mentions that we can activate the virtualenv using pipenv shell.
  • alias jhbuild="PATH=.new_local/bin:$PATH jhbuild"
  • jhbuild bootstrap-gtk-osx
  • stick our jhbuild-customrc config in ~/.config (is that even documented anywhere? the gtk-osx home page still refers to the old location..)
  • disable cloudflare for xpra.org because jhbuild fails miserably otherwise.. (SSL errors, 403 errors, etc)
  • run jhbuild build and watch numerous modules fail starting with gtk..(then pygtk, gtk-mac-integration-python, gtkglext, pygtkglext, etc)
    But also:
  • python-lzo
  • rencode
  • pyobjc requires xcode and not just the command line tools... so first install that
  • gtkglext directory was empty - the hack to apply the patches before building no longer works because the path has changed.. yay, fixing it by hand fails later during compilation: libtool: compile: unable to infer tagged configuration
  • gtk-mac-bundler fails
  • bomutils
  • serf

The gtk error is caused by the newer pango version missing some macros, simply re-adding them to the pango.h fixes things, but then pygtk still fails later.
So r23506 (+r23507 fixup) downgrades pango back to version 1.42.4 - doesn't gtk-osx still support gtk2?
r23508 also reverts the harfbuzz + freetype-no-harfbuzz changes needed by the older version of pango but keeps the newer freetype version.
(it builds as long as you skip the tests - TODO: make a patch or use a different branch for GTK2 / GTK3?)

Next, try GTK3...

@totaam
Copy link
Collaborator Author

totaam commented Aug 14, 2019 •
edited
Loading

More GTK2 difficulties:

  • r23509: rencode build fix
  • r23511: prevents import errors causing the py2app step to fail
  • had to patch gtk-mac-bundler to avoid pango errors - why!?
  • r23512 new library dependencies that aren't picked up automatically..

@totaam
Copy link
Collaborator Author

totaam commented Aug 15, 2019 •
edited
Loading

GTK3: on the same system, with the same version of xcode.
Running the gtk-osx setup script and trying to bootstrap fails to link xz properly, like it is using the targeting the wrong SDK version. WTH!?

@totaam
Copy link
Collaborator Author

totaam commented Aug 15, 2019 •
edited
Loading

More updates:

@totaam
Copy link
Collaborator Author

totaam commented Aug 15, 2019 •
edited
Loading

So, the build errors are now also occurring with the first user - at least this part makes sense.
They are caused by xcode 8.x: homebrew: python3 failed to build on 10.11.6 with Xcode 8: There's an "issue" (Apple considers it a feature) with the new Xcode, which happens as a consequence of Apple retaining the single SDK structure rather than having one SDK in Xcode for 10.11 and another for 10.12.
More info here.
Patching dozens of projects for some Apple-only SDK version breakage would be crazy, so I'm now downgrading to xcode 7 instead.

@totaam
Copy link
Collaborator Author

totaam commented Aug 15, 2019 •
edited
Loading

GTK3:

  • bootstrap errors on itstool, something about python needing libxml2 - similar to configure: error: Python module libxml2 is needed to run this package but now for python3 - using itstool 2.0.2 fixes the problem (uses python2)
  • pango: need to skip tests - create patch?
  • gtk-mac-bundler path is wrong during build - untar from source archive and it proceeds
  • numpy build error
  • python-lzo needs lzo

Still TODO:

  • itstool must be built with PYTHON=/path/to/python2 ./configure ...
  • fix theming?
  • charset.alias?
  • gtk-mac-bundler pango patch?

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

GTK builds fail with:

$ python3 -c "from gi.repository import Gtk"

-* (process:786): WARNING **: 18:33:15.061: Failed to load shared library 'libgdk_pixbuf-2.0.0.dylib' referenced by the typelib: dlopen(libgdk_pixbuf-2.0.0.dylib, 9): image not found
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 145, in load_module
    importlib.import_module('gi.repository.' + dep.split("-")[0])
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 145, in load_module
    importlib.import_module('gi.repository.' + dep.split("-")[0])
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/importer.py", line 146, in load_module
    dynamic_module = load_overrides(introspection_module)
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/__init__.py", line 118, in load_overrides
    override_mod = importlib.import_module(override_package_name)
  File "/Users/gtk3/gtk/inst/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/GdkPixbuf.py", line 32, in <module>
    class Pixbuf(GdkPixbuf.Pixbuf):
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/overrides/__init__.py", line 195, in override
    assert g_type != TYPE_NONE
AssertionError

The typelib paths are strange:

strings $PREFIX/lib/girepository-1.0/*.typelib | grep dylib
libatk-1.0.0.dylib
libgirepository-1.0.1.dylib
/Users/gtk3/gtk/inst/lib/libgobject-2.0.0.dylib,/Users/gtk3/gtk/inst/lib/libglib-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgmodule-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgobject-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgdk-3.0.dylib
libgdk_pixbuf-2.0.0.dylib
libgdk_pixbuf-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgio-2.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstreamer-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstallocators-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstapp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstaudio-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstbase-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstcheck-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstcontroller-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstgl-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstinsertbin-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstmpegts-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstnet-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstpbutils-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstplayer-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstrtp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstrtsp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstsdp-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgsttag-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstvideo-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgstwebrtc-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libgtk-3.0.dylib,/Users/gtk3/gtk/inst/lib/libgdk-3.0.dylib
/Users/gtk3/gtk/inst/lib/libgtkmacintegration-gtk3.2.dylib
/Users/gtk3/gtk/inst/lib/libpango-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/libpango-1.0.0.dylib,/Users/gtk3/gtk/inst/lib/libpangocairo-1.0.0.dylib
/Users/gtk3/gtk/inst/lib/librsvg-2.2.dylib
libcairo-gobject.2.dylib

Adding the library path works around this issue - and hits another one.. (libjpeg):

DYLD_LIBRARY_PATH=$PREFIX/lib python3 -c "from gi.repository import Gtk"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/__init__.py", line 42, in <module>
    from . import _gi
ImportError: dlopen(/Users/gtk3/gtk/inst/lib/python3.7/site-packages/gi/_gi.cpython-37m-darwin.so, 2): Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Users/gtk3/gtk/inst/lib/libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO

Reproducible with just:

DYLD_LIBRARY_PATH=/Users/gtk3/gtk/inst/lib python3 -c "import gi"

Rebuilding gobject-introspection and pygobject3 does not help.
That's because the ImageIO framework needs the old system libjpeg, not the one we force with DYLD_LIBRARY_PATH..
So we need a way to get gdk-pixbuf to load the correct dylib without using DYLD_LIBRARY_PATH.
Is it a coincidence that gdk-pixbuf is one of the modules that has switched to the meson build system?

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

Some pointers:

This does allow us to import gtk:

DYLD_FALLBACK_LIBRARY_PATH=$PREFIX/lib python3 -c "from gi.repository import Gtk"

But using the same env var does not fix running the tests!

Options:

  • build without running the unit tests with python3 - no go, still fails during py2app..
  • revert the merge that switched to meson?
  • don't use turbo-jpeg? (may not help)

Then a new problem during packaging:

ValueError: '/Users/gtk3/gtk/inst/lib/libpython3.7.dylib' does not exist

And indeed, it does not.
We have a libpython3.7m.dylib instead because python was built with --with-pymalloc. Why doesn't py2app figure it out?
Solved by:

cd $PREFIX/lib/                  
ln -sf libpython3.7m.dylib libpython3.7.dylib

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

Reverting gdk-pixbuf to autotools in r23523 (+r23524 fixup) fixes that problem.
But then it's atk's turn:

$ python3 -c "from gi.repository import Gtk"

-* (-c:48674): WARNING **: 19:58:30.103: Failed to load shared library 'libatk-1.0.0.dylib' referenced by the typelib: dlopen(libatk-1.0.0.dylib, 9): image not found

So r23525 reverts that one to autotools.

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

New problem after that: gtk-mac-integration needs rebuilding, it depends on gtk which did get rebuilt. jhbuild should have figured it out...

Moving on, then the unit tests fail again, but later:

FAIL: test_all_codecs_found (__main__.TestDecoders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gtk3/Xpra/trunk/src/unittests/unit/codecs/codecs_selftest_test.py", line 45, in test_all_codecs_found
    selftest(True)
  File "xpra/codecs/vpx/encoder.pyx", line 739, in xpra.codecs.vpx.encoder.selftest
AssertionError: <module 'xpra.codecs.vpx.encoder' from '/Users/gtk3/gtk/inst/lib/python3.7/site-packages/xpra/codecs/vpx/encoder.cpython-37m-darwin.so'> is limited to 512x512 for vp8 and not 8192x4096

Then running the test by hand produces another strange error:

AssertionError: <module 'xpra.codecs.enc_x264.encoder' from \
    '/Users/gtk3/gtk/inst/lib/python3.7/site-packages/xpra/codecs/enc_x264/encoder.cpython-37m-darwin.so'> is limited to 512x512 and not 8192x4096

That's because of the missing numpy, which is used to generate the test pixel buffers.

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

Fixing numpy: revert to v1.16.4 in r23526.

@totaam
Copy link
Collaborator Author

totaam commented Aug 16, 2019 •
edited
Loading

More GTK3 fixes:

  • r23527 gdk-pixbuf packaging error (missing matching pkg-config entry in a GTK3 environment, so hardcoded..)
  • r23528 + r23529: don't include anything related to "pygtk"
  • r23530: add missing Tango theme

And with these changes, I can finally make both GTK2 and GTK3 builds again!

@totaam
Copy link
Collaborator Author

totaam commented Aug 17, 2019 •
edited
Loading

And... that's obviously not the end of it.
GTK2 builds have a new problem: the gi bindings are missing, and trying to force enable them in pygobject with --enable-introspection does not work:

  CC     _gi_la-pygi-foreign-gvariant.lo
pygi-info.c:165:14: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
        case GI_INFO_TYPE_ERROR_DOMAIN:
             ^
  CC     _gi_la-pygi-struct.lo
pygi-info.c:135:13: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
    switch (info_type)
            ^
  CC     _gi_la-pygi-argument.lo
pygi-info.c:484:22: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
                case GI_INFO_TYPE_ERROR_DOMAIN:
                     ^
pygi-info.c:448:21: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
            switch (info_type) {
                    ^
  CC     _gi_la-pygi-type.lo
pygi-info.c:863:26: error: use of undeclared identifier 'GI_INFO_TYPE_ERROR_DOMAIN'
                    case GI_INFO_TYPE_ERROR_DOMAIN:
                         ^
pygi-info.c:835:25: warning: enumeration value 'GI_INFO_TYPE_INVALID_0' not handled in switch [-Wswitch]
                switch (info_type) {
                        ^
  CC     _gi_la-pygi-boxed.lo
3 warnings and 3 errors generated.

Apparently, that's because the versions are incompatible: pygobject make error.

Looks like the one we want is actually pygobject3.
It builds OK, even against python2, but does not run:

 python2 -c "import gi"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "gi/__init__.py", line 23, in <module>
    from ._gi import _API, Repository
ImportError: No module named _gi

@totaam
Copy link
Collaborator Author

totaam commented Aug 17, 2019 •
edited
Loading

Ignore the second half of comment:17, which was run from a directory containing a "gi" folder, causing this spurious error.

The new builds do include the gi bindings, but pygtk needed rebuilding (a different version of atk got built at some point?)
Same for gtk-mac-integration-python

@totaam
Copy link
Collaborator Author

totaam commented Aug 20, 2019 •
edited
Loading

@smo: when you get a chance, you should be able to build clean GTK2-Python2 and GTK3-Python3 environments with trunk.
The only issues you will likely encounter:

  • we still have to symlink python to python3 by hand early as jhbuild doesn't handle that
  • scons needs python2 to build (so temporarily switch back)
  • pango tests fail (needs patch to Makefile to just skip them)
  • one or two modules don't build, when you drop to a shell their directory is empty... untar it by hand and continue
  • for building: ln -sf libpython3.7m.dylib libpython3.7.dylib (see comment:12)
  • itstool issue, see comment:10
  • gtk-mac-bundler needs a patch to ignore the magic pango macros it is trying to substitute

@totaam
Copy link
Collaborator Author

totaam commented Sep 3, 2019 •
edited
Loading

Latest .config/jhbuildrc-custom I'm using for gtk3 builds:

_gtk_osx_use_jhbuild_python = True

setup_sdk(target="10.11", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"

modules = ["meta-osx-xpra-deps"]

moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk3.modules"

os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"

@totaam
Copy link
Collaborator Author

totaam commented Sep 3, 2019 •
edited
Loading

New error I just hit with flac and the libogg 1.3.4 update from r23655:

error: unknown type name 'uint16_t'
...

Updating flac to 1.3.3 (r23708) did not help.
Fixed by adding #include <stdint.h> near the top of $PREFIX/inst/include/ogg/ogg.h.

@totaam
Copy link
Collaborator Author

totaam commented Sep 20, 2019 •
edited
Loading

Updates:

@totaam totaam closed this as completed Sep 20, 2019
@totaam
Copy link
Collaborator Author

totaam commented Sep 29, 2019 •
edited
Loading

From comment:11 :

Is it a coincidence that gdk-pixbuf is one of the modules that has switched to the meson build system?

No, it's not: Need to explicitly export DYLD_FALLBACK_LIBRARY_PATH : It mostly has to do with meson and rpaths, ... I just got GitHub up to date, so if that's where you got it pull and bundle again

Will try again for 4.0 : Xpra-org/xpra#2385

@totaam totaam transferred this issue from Xpra-org/xpra Jul 13, 2021
This was referenced Jul 13, 2021
Closed
Closed
@totaam totaam mentioned this issue Aug 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@totaam