Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3Packages.cffi: fix missing ffi_prep_closure_loc() on macos < 10.15 #280546

Merged
merged 1 commit into from
Feb 17, 2024

Conversation

abathur
Copy link
Member

@abathur abathur commented Jan 12, 2024

In the course of testing out the fix that ultimately landed in #278945, I ran into the following missing-function error in cffi's test suite:
Executing pytestCheckPhase
============================= test session starts ==============================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
rootdir: /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0
collecting ... ^Mcollecting 229 items                                                           ^Mcollecting 641 items                                                           ^Mcollecting 872 items 

==================================== ERRORS ====================================
_____________ ERROR collecting testing/cffi1/test_parse_c_type.py ______________
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/nix/store/riwkg7zfhj0fzn918s8yiblnvyp0vcmd-python3-3.11.7/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
testing/cffi1/test_parse_c_type.py:75: in <module>
    @ffi.callback("int(unsigned long long *)")
src/cffi/api.py:396: in callback_decorator_wrap
    return self._backend.callback(cdecl, python_callable,
E   SystemError: ffi_prep_closure_loc() is missing
------------------------------- Captured stderr --------------------------------
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
=============================== warnings summary ===============================
src/cffi/_imp_emulation.py:4
  /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0/src/c/../cffi/_imp_emulation.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in P
    from imp import *

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR testing/cffi1/test_parse_c_type.py - SystemError: ffi_prep_closure_loc() is missing
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 1 error in 7.96s ==========================

The missing function is in turn caused by these defines: https://github.com/python-cffi/cffi/blame/feaff378873d8a3f52bbb63f7d5e24f9ab18f1e9/src/c/_cffi_backend.c#L106-L113

If I understand correctly (I may not), cffi is assuming the system libffi here, which doesn't apply in our case. (cc @reckenrode)

(The underlying defines (which affect macOS before 10.15) have been in cffi for 4+ years now, so I'm not quite sure how we haven't picked up on it already.)

Description of changes

I edited the patch for this behavior (pkgs/development/python-modules/cffi/darwin-use-libffi-closures.diff) to remove the version check.

I went ahead and did this for all 3 checks, even though I suspect we didn't technically need all of them to pass the test failure above.

Things done

I was able to successfully build and use nix with this patch present.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

If I understand correctly (I may not), cffi is assuming the system
libffi here, which doesn't apply in our case.

The underlying defines (which affect macOS before 10.15) have been in
cffi for 4+ years now, so I'm not quite sure how we haven't picked up
on it already.
@reckenrode
Copy link
Contributor

lgtm, but I’d like to get some more eyes on it to make sure.

@abathur abathur requested review from tjni, mweinelt and toonn February 6, 2024 15:56
Copy link
Contributor

@toonn toonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vcunat vcunat merged commit 579b24c into NixOS:staging Feb 17, 2024
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants