macOS dylib version changed despite fontconfig version not changing #54
Comments
|
Minimal code to reproduce the issue: Then try the following: from rdkit.Chem import DrawThis gives: |
|
Thanks for filing — this is due to a switchover of the fontconfig build system, and we have some discussion ongoing in #53. There are some efforts to rebuild some of the depending packages which should make the issue go away, |
|
Thanks for the quick reply and clarification. Aplogies for not spotting the PR thread. I knew that the issue must have been introduced in the most recent PR, but hadn't thought of reading the thread first 🤦♂️ Using a modified version of the uggested code from this thread, i.e. reverse searching the conda-forge repodata json for the dependers of a package: # reverse_search.py
import json
import re
def reverse_dependency(package_index, upstream='fontconfig'):
"""Print packages from package_index who depend on any version of upstream.
Parameters
----------
package_index : string
The location of a package index JSON.
upstream : string
The name of a package (no version or build number) for which you want
to find other packages that depend on it.
"""
with open(package_index, 'r') as f:
dependency_tree = json.load(f)
packages = []
for package, metadata in dependency_tree['packages'].items():
for dependency in metadata['depends']:
if upstream in dependency:
m = re.search("^[a-z0-9]+(\-*[a-z])*-\\d", package)
packages.append(m.group(0)[:-2])
break
for package in sorted(set(packages)):
print(package)
if __name__ == '__main__':
reverse_dependency('conda-forge.json')Then running: gives the following dependers for (Note that this is any version of Assuming the new dylib version is correct, and that it won't take too long to get the dependers rebuilt, then that probably is the correct approach. In our case the issue would be resolved once |
|
I should have looked at the |
|
CC @ryanvolz. After looking into the migrator infrastructure a bit, it might be a bit of a hassle to try to set up a migrator to work through this issue, although I'm sure that it would be doable with sufficient effort. |
|
Here's my analysis/checklist of whether the packages listed above are OK with the known fontconfig:
Confirmed that no rebuild needed:
|
|
Yeah, I tried investigating the same. The only thought I had was whether putting in a migrator that sets the pin to the same version as it is currently would work. The biggest problem I currently see could be solved by a rebuild of As for what I'm familiar with in the rest of the list, |
|
I'm checking out a Pango rebuild now. Good point about |
|
Phew — looking at the |
|
Well that's a relief! A manual migration seems totally feasible then. |
|
Updates for the list:
|
|
Thanks for reopening and for your hard work on resolving this. I had only closed as, seeing the cairo rebuild, I had assumed that all others were in progress. Cheers. |
|
OK I've finished my analysis of what would need rebuilding. The outcome is pretty good. I'm about to disappear for the weekend but I'll file a few more PRs on what I think are the deepest dependencies. |
|
Apologies, I realised that I used the incorrect repodata.json link, i.e. one for the |
|
@lohedges No worries, the list was still quite helpful. The only significant new item I see with the updated list is |
|
Status update: the necessary rebuilds should be essentially done. |
|
According to an earlier note, libgd was rebuilt, but it's still failing for us: downgrading fontconfig fixes it: (there may be a build of 2.13.96 that works -- haven't tested that -- but what conda-forge provides now is broken :-( NOTE: I've realized that this is libgd 2.3.0 -- which is apparently pinned by py_gd. I'll see if we can update py_gd to use the latest libgd. But maybe we want an updated build of libgd 2.3.0 ? |
|
I don't know the details of |
|
@ryanvolz: thanks -- I'm working on rebuilding py_gd -- not sure about the tight pin -- py_gd works with the internals of libgd, but 2.3.* would probably work. I didn't expect an existing build for libgd 2.3.0 to actually break in the future though -- so I thought the tight pin was safer. Oh well, dependency heck is inevitable :-( And now py_gd won't build thanks to changes in setuptool what fun! But not relevant to this issue. |
|
OK -- looks like rebuilding py_gd is working -- so that's one more down. |
|
@ChrisBarker-NOAA Thanks for your patience and assistance working through this! |
|
I think that it might be about time to close this issue — I see that references to it are still coming in, but it looks like we've reached the point where problems are second-order ones related to not being able to pull in fully up-to-date dependencies. I'm still more than happy to help people work through rebuilds, but that's probably better done on issues/PRs associated with the specific feedstocks, rather than here. What do other people think? |
|
I agree -- it seems as much has been done with fontconfig as will be done. And this will be in the archive if more people come up with issues. |
|
Thanks for the effort to rebuild all the dependencies. I think this is working now when pulling in the latest versions. However, in CI runs we sometimes want to test things with older versions of packages and now such environments are sometimes not functional anymore, see e.g., https://github.com/flatsurf/e-antic/runs/5806878598. Here, libgd 2.2.5 is pulled in by sagelib=9.1 which does not understand the soname change in the latest version of fontconfig. @isuruf It would seem to me that metadata of previous builds that depend on fontconfig should still be patched with https://github.com/conda-forge/conda-forge-repodata-patches-feedstock so that old builds that depend on fontconfig remain functional. What do you think? |
|
You can add a constraint for fontconfig for all packages depending on fontconfig built before the latest build number. |
|
For future reference, if you are changing build system of a package, please go through https://conda-forge.org/docs/maintainer/knowledge_base.html#moving-from-an-autotools-build-to-a-cmake-build |
Solution to issue cannot be found in the documentation.
Issue
Hello there,
I am trying to debug a CI error where a dependency of ours (RDKit) is failing to import since the
fontconfigbuild number being updated. We only see this issue on macOS and it appears to be the result of a dylib version change:Comparing against the last macOS build that worked (literally hours before) I see the following difference in
fontconfigversion installed during the build process:works
fails
(All other packages are the same.)
Surely an update to the build number (no change to the version) shouldn't change the dylib version? Note that I see the updated build number for our Linux build too, but this doesn't result in an
ImportError.Cheers.
Installed packages
appnope: 0.1.2-py37hf985489_2 conda-forge argh: 0.26.2-pyh9f0ad1d_1002 conda-forge argon2-cffi: 21.3.0-pyhd8ed1ab_0 conda-forge argon2-cffi-bindings: 21.2.0-py37h271585c_1 conda-forge astunparse: 1.6.3-pyhd8ed1ab_0 conda-forge atk-1.0: 2.36.0-he69c4ee_4 conda-forge attrs: 21.4.0-pyhd8ed1ab_0 conda-forge backcall: 0.2.0-pyh9f0ad1d_0 conda-forge backports: 1.0-py_2 conda-forge backports.functools_lru_cache: 1.6.4-pyhd8ed1ab_0 conda-forge biopython: 1.79-py37h271585c_1 conda-forge biosimspace: 2022.1.0-py37_7 local bleach: 4.1.0-pyhd8ed1ab_0 conda-forge blosc: 1.21.0-he49afe7_0 conda-forge boost: 1.74.0-py37h326c9ff_5 conda-forge boost-cpp: 1.74.0-hdbf7018_7 conda-forge brotli: 1.0.9-h0d85af4_6 conda-forge brotli-bin: 1.0.9-h0d85af4_6 conda-forge brotlipy: 0.7.0-py37h271585c_1003 conda-forge bzip2: 1.0.8-h0d85af4_4 conda-forge c-ares: 1.18.1-h0d85af4_0 conda-forge ca-certificates: 2021.10.8-h033912b_0 conda-forge cairo: 1.16.0-he01c77b_1009 conda-forge certifi: 2021.10.8-py37hf985489_1 conda-forge cffi: 1.15.0-py37h446072c_0 conda-forge cftime: 1.6.0-py37h4105427_0 conda-forge charset-normalizer: 2.0.12-pyhd8ed1ab_0 conda-forge colorama: 0.4.4-pyh9f0ad1d_0 conda-forge configargparse: 1.5.3-pyhd8ed1ab_0 conda-forge cryptography: 36.0.1-py37h5e77fcc_0 conda-forge curl: 7.82.0-h9f20792_0 conda-forge cycler: 0.11.0-pyhd8ed1ab_0 conda-forge dbus: 1.13.6-h811a1a6_3 conda-forge debugpy: 1.5.1-py37hd8d24ac_0 conda-forge decorator: 5.1.1-pyhd8ed1ab_0 conda-forge defusedxml: 0.7.1-pyhd8ed1ab_0 conda-forge entrypoints: 0.4-pyhd8ed1ab_0 conda-forge expat: 2.4.7-h96cf925_0 conda-forge fftw: 3.3.10-nompi_hf082fe4_102 conda-forge fkcombu: 2020.04.14-h254f028_0 michellab/label/dev flit-core: 3.7.1-pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono: 2.37-hab24e00_0 conda-forge font-ttf-inconsolata: 3.000-h77eed37_0 conda-forge font-ttf-source-code-pro: 2.038-h77eed37_0 conda-forge font-ttf-ubuntu: 0.83-hab24e00_0 conda-forge fontconfig: 2.13.96-h676cef8_1 conda-forge fonts-conda-ecosystem: 1-0 conda-forge fonts-conda-forge: 1-0 conda-forge fonttools: 4.29.1-py37h271585c_0 conda-forge freetype: 2.10.4-h4cff582_1 conda-forge fribidi: 1.0.10-hbcb3906_0 conda-forge gdk-pixbuf: 2.42.6-h2e6141f_0 conda-forge gettext: 0.19.8.1-hd1a6beb_1008 conda-forge giflib: 5.2.1-hbcb3906_2 conda-forge git: 2.35.1-pl5321h33a4a8a_0 conda-forge graphite2: 1.3.13-h2e338ed_1001 conda-forge graphviz: 3.0.0-h6f567cf_0 conda-forge greenlet: 1.1.2-py37hd8d24ac_1 conda-forge griddataformats: 0.7.0-pyhd8ed1ab_0 conda-forge gsd: 2.5.1-py37h032687b_0 conda-forge gsl: 2.7-h93259b0_0 conda-forge gtk2: 2.24.33-h7c1209e_2 conda-forge gts: 0.7.6-hccb3bdf_2 conda-forge harfbuzz: 3.4.0-h447b35c_0 conda-forge hdf4: 4.2.15-hefd3b78_3 conda-forge hdf5: 1.12.1-nompi_ha60fbc9_104 conda-forge icu: 69.1-he49afe7_0 conda-forge idna: 3.3-pyhd8ed1ab_0 conda-forge importlib-metadata: 4.11.2-py37hf985489_0 conda-forge importlib_metadata: 4.11.2-hd8ed1ab_0 conda-forge importlib_resources: 5.4.0-pyhd8ed1ab_0 conda-forge iniconfig: 1.1.1-pyh9f0ad1d_0 conda-forge ipykernel: 6.9.1-py37h4c52d7d_0 conda-forge ipython: 7.32.0-py37hf985489_0 conda-forge ipython_genutils: 0.2.0-py_1 conda-forge ipywidgets: 7.6.5-pyhd8ed1ab_0 conda-forge jbig: 2.1-h0d85af4_2003 conda-forge jedi: 0.18.1-py37hf985489_0 conda-forge jinja2: 3.0.3-pyhd8ed1ab_0 conda-forge joblib: 1.1.0-pyhd8ed1ab_0 conda-forge jpeg: 9e-h0d85af4_0 conda-forge jsonschema: 4.4.0-pyhd8ed1ab_0 conda-forge jupyter_client: 7.1.2-pyhd8ed1ab_0 conda-forge jupyter_core: 4.9.2-py37hf985489_0 conda-forge jupyterlab_pygments: 0.1.2-pyh9f0ad1d_0 conda-forge jupyterlab_widgets: 1.0.2-pyhd8ed1ab_0 conda-forge khronos-opencl-icd-loader: 2022.01.04-h0d85af4_0 conda-forge kiwisolver: 1.3.2-py37h737db71_1 conda-forge krb5: 1.19.2-hb49756b_4 conda-forge lcms2: 2.12-h577c468_0 conda-forge lerc: 3.0-he49afe7_0 conda-forge libblas: 3.9.0-13_osx64_openblas conda-forge libbrotlicommon: 1.0.9-h0d85af4_6 conda-forge libbrotlidec: 1.0.9-h0d85af4_6 conda-forge libbrotlienc: 1.0.9-h0d85af4_6 conda-forge libcblas: 3.9.0-13_osx64_openblas conda-forge libclang: 13.0.1-default_he082bbe_0 conda-forge libcpuid: 0.5.1-h01edc0c_0 michellab/label/dev libcurl: 7.82.0-h9f20792_0 conda-forge libcxx: 12.0.1-habf9029_1 conda-forge libdeflate: 1.10-h0d85af4_0 conda-forge libedit: 3.1.20191231-h0678c8f_2 conda-forge libev: 4.33-haf1e3a3_1 conda-forge libffi: 3.4.2-h0d85af4_5 conda-forge libgd: 2.3.3-hb8742e1_1 conda-forge libgfortran: 5.0.0-9_3_0_h6c81a4c_23 conda-forge libgfortran5: 9.3.0-h6c81a4c_23 conda-forge libglib: 2.70.2-hf1fb8c0_4 conda-forge libiconv: 1.16-haf1e3a3_0 conda-forge liblapack: 3.9.0-13_osx64_openblas conda-forge libllvm13: 13.0.1-h64f94b2_2 conda-forge libnetcdf: 4.8.1-nompi_h6609ca0_101 conda-forge libnghttp2: 1.47.0-h942079c_0 conda-forge libopenblas: 0.3.18-openmp_h3351f45_0 conda-forge libpng: 1.6.37-h7cec526_2 conda-forge libpq: 14.2-hea3049e_0 conda-forge librsvg: 2.52.5-h5e5d7d9_2 conda-forge libsodium: 1.0.18-hbcb3906_1 conda-forge libssh2: 1.10.0-h52ee1ee_2 conda-forge libtiff: 4.3.0-h17f2ce3_3 conda-forge libtool: 2.4.6-he49afe7_1008 conda-forge libwebp: 1.2.2-h28dabe5_0 conda-forge libwebp-base: 1.2.2-h0d85af4_1 conda-forge libxcb: 1.13-h0d85af4_1004 conda-forge libxml2: 2.9.12-h7e28ab6_1 conda-forge libzip: 1.8.0-h8b0c345_1 conda-forge libzlib: 1.2.11-h9173be1_1013 conda-forge llvm-openmp: 13.0.1-hcb1a161_1 conda-forge lz4-c: 1.9.3-he49afe7_1 conda-forge markupsafe: 2.1.0-py37h69ee0a8_1 conda-forge matplotlib-base: 3.5.1-py37h3147e9e_0 conda-forge matplotlib-inline: 0.1.3-pyhd8ed1ab_0 conda-forge mdanalysis: 2.1.0-py37h0582d14_0 conda-forge mdtraj: 1.9.7-py37h7b928b5_1 conda-forge mistune: 0.8.4-py37h271585c_1005 conda-forge mmtf-python: 1.1.2-py_0 conda-forge mrcfile: 1.3.0-pyh44b312d_0 conda-forge msgpack-python: 1.0.3-py37h737db71_0 conda-forge munkres: 1.1.4-pyh9f0ad1d_0 conda-forge mysql-common: 8.0.28-h694c41f_0 conda-forge mysql-libs: 8.0.28-h115446f_0 conda-forge nbclient: 0.5.12-pyhd8ed1ab_0 conda-forge nbconvert: 6.4.2-py37hf985489_0 conda-forge nbformat: 5.1.3-pyhd8ed1ab_0 conda-forge ncurses: 6.3-he49afe7_0 conda-forge nest-asyncio: 1.5.4-pyhd8ed1ab_0 conda-forge netcdf4: 1.5.8-nompi_py37h5a78667_101 conda-forge networkx: 2.7.1-pyhd8ed1ab_0 conda-forge nglview: 3.0.3-pyh8a188c0_0 conda-forge notebook: 6.4.8-pyha770c72_0 conda-forge nspr: 4.32-hcd9eead_1 conda-forge nss: 3.74-h31e2bf1_0 conda-forge numexpr: 2.8.0-py37h5b83a90_1 conda-forge numpy: 1.21.5-py37h3c8089f_0 conda-forge ocl_icd_wrapper_apple: 1.0.0-hbcb3906_0 conda-forge openff-forcefields: 2.0.0-pyh6c4a22f_0 conda-forge openff-toolkit-base: 0.10.3-pyhd8ed1ab_0 conda-forge openjpeg: 2.4.0-h6e7aa92_1 conda-forge openmm: 7.7.0-py37hf0bdc88_0_khronos conda-forge openssl: 1.1.1l-h0d85af4_0 conda-forge packaging: 21.3-pyhd8ed1ab_0 conda-forge pandas: 1.3.5-py37h5b83a90_0 conda-forge pandoc: 2.17.1.1-h694c41f_0 conda-forge pandocfilters: 1.5.0-pyhd8ed1ab_0 conda-forge pango: 1.50.5-h0e59876_0 conda-forge parmed: 3.4.3-py37hd8d24ac_1 conda-forge parso: 0.8.3-pyhd8ed1ab_0 conda-forge patsy: 0.5.2-pyhd8ed1ab_0 conda-forge pcre: 8.45-he49afe7_0 conda-forge pcre2: 10.37-ha16e1b2_0 conda-forge perl: 5.32.1-2_h0d85af4_perl5 conda-forge pexpect: 4.8.0-pyh9f0ad1d_2 conda-forge pickleshare: 0.7.5-py_1003 conda-forge pillow: 9.0.1-py37h2540ef4_2 conda-forge pip: 22.0.4-pyhd8ed1ab_0 conda-forge pixman: 0.40.0-hbcb3906_0 conda-forge pluggy: 1.0.0-py37hf985489_2 conda-forge prometheus_client: 0.13.1-pyhd8ed1ab_0 conda-forge prompt-toolkit: 3.0.27-pyha770c72_0 conda-forge pthread-stubs: 0.4-hc929b4f_1001 conda-forge ptyprocess: 0.7.0-pyhd3deb0d_0 conda-forge py: 1.11.0-pyh6c4a22f_0 conda-forge pycairo: 1.21.0-py37h626a405_0 conda-forge pycparser: 2.21-pyhd8ed1ab_0 conda-forge pydot: 1.4.2-py37hf985489_1 conda-forge pygments: 2.11.2-pyhd8ed1ab_0 conda-forge pygtail: 0.11.1-py_0 conda-forge pymbar: 3.0.6-py37h032687b_0 conda-forge pyopenssl: 22.0.0-pyhd8ed1ab_0 conda-forge pyparsing: 3.0.7-pyhd8ed1ab_0 conda-forge pypdb: 2.0-pyhd8ed1ab_0 conda-forge pyqt: 5.12.3-py37hf985489_8 conda-forge pyqt-impl: 5.12.3-py37hab5ec1f_8 conda-forge pyqt5-sip: 4.19.18-py37h070e122_8 conda-forge pyqtchart: 5.12-py37hab5ec1f_8 conda-forge pyqtwebengine: 5.12.1-py37hab5ec1f_8 conda-forge pyrsistent: 0.18.1-py37h271585c_0 conda-forge pysocks: 1.7.1-py37hf985489_4 conda-forge pytables: 3.7.0-py37hc0663ee_0 conda-forge pytest: 7.0.1-py37hf985489_0 conda-forge python: 3.7.12-haf480d7_100_cpython conda-forge python-dateutil: 2.8.2-pyhd8ed1ab_0 conda-forge python_abi: 3.7-2_cp37m conda-forge pytz: 2021.3-pyhd8ed1ab_0 conda-forge pyyaml: 6.0-py37h271585c_3 conda-forge pyzmq: 22.3.0-py37h8f778e5_1 conda-forge qt: 5.12.9-h2a607e2_5 conda-forge rdkit: 2021.09.5-py37h72bc793_0 conda-forge readline: 8.1-h05e3726_0 conda-forge reportlab: 3.5.68-py37h08983f2_1 conda-forge requests: 2.27.1-pyhd8ed1ab_0 conda-forge scikit-learn: 1.0.2-py37h572704e_0 conda-forge scipy: 1.7.3-py37h4e3cf02_0 conda-forge seaborn: 0.11.2-hd8ed1ab_0 conda-forge seaborn-base: 0.11.2-pyhd8ed1ab_0 conda-forge send2trash: 1.8.0-pyhd8ed1ab_0 conda-forge setuptools: 59.8.0-py37hf985489_0 conda-forge sire: 2022.1.0-py37h01edc0c_14 michellab/label/dev six: 1.16.0-pyh6c4a22f_0 conda-forge smirnoff99frosst: 1.1.0-pyh44b312d_0 conda-forge snappy: 1.1.8-hb1e8313_3 conda-forge sqlalchemy: 1.4.32-py37h69ee0a8_0 conda-forge sqlite: 3.37.0-h23a322b_0 conda-forge statsmodels: 0.13.2-py37h032687b_0 conda-forge tbb: 2021.5.0-h940c156_0 conda-forge terminado: 0.13.3-py37hf985489_0 conda-forge testpath: 0.6.0-pyhd8ed1ab_0 conda-forge threadpoolctl: 3.1.0-pyh8a188c0_0 conda-forge tk: 8.6.12-h5dbffcc_0 conda-forge tomli: 2.0.1-pyhd8ed1ab_0 conda-forge tornado: 6.1-py37h271585c_2 conda-forge tqdm: 4.63.0-pyhd8ed1ab_0 conda-forge traitlets: 5.1.1-pyhd8ed1ab_0 conda-forge typing_extensions: 4.1.1-pyha770c72_0 conda-forge unicodedata2: 14.0.0-py37h271585c_0 conda-forge urllib3: 1.26.8-pyhd8ed1ab_1 conda-forge watchdog: 2.1.6-py37hb9fb9c6_1 conda-forge wcwidth: 0.2.5-pyh9f0ad1d_2 conda-forge webencodings: 0.5.1-py_1 conda-forge wheel: 0.37.1-pyhd8ed1ab_0 conda-forge widgetsnbextension: 3.5.2-py37hf985489_1 conda-forge xmltodict: 0.12.0-py_0 conda-forge xorg-libxau: 1.0.9-h35c211d_0 conda-forge xorg-libxdmcp: 1.1.3-h35c211d_0 conda-forge xz: 5.2.5-haf1e3a3_1 conda-forge yaml: 0.2.5-h0d85af4_2 conda-forge zeromq: 4.3.4-he49afe7_1 conda-forge zipp: 3.7.0-pyhd8ed1ab_1 conda-forge zlib: 1.2.11-h9173be1_1013 conda-forge zstd: 1.5.2-h582d3a0_0 conda-forgeEnvironment info
Can't access, since failure was during our CI. Can reproduce locally if needed.The text was updated successfully, but these errors were encountered: