MemoryError: Cannot allocate write+execute memory for ffi.callback() #873
Comments
|
FWIW, I get the same message, on Ubuntu 18.04.3. It actually results in the same stack trace even when I go to execute other things (like install unrelated packages via pip). Uninstalling pyOpenSSL (after the scrapy install) allowed other installations to proceed, showing that it was pyOpenSSL related, but of course scrapy and other dependants can't fully function without it. Andrii's first link above suggests some workarounds, but not for Ubuntu. Root of the issue might be a grsecurity override, in (say) a shared webhosting environment. Use of ffi.callback() is apparently frowned-upon now as insecure, and that link above has suggested workarounds. The end of my stack trace, just to note slight variations in line numbers from the above: |
|
pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL's callback system. |
|
Yeah, seems like there's a ready way to fix the code by changing to new-style "Extern Python" callbacks in CFFI. Here's another project that ran into the same issue and fixed it that way, with a real fast turnaround: Issue: FSX/misaka#72 I'm not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI. |
|
I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure's CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux). Reading the answers above, does it mean every library using openssl has to made that kind of changes ? When @tiran mentions "use another TLS library", is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ? |
|
tl;dr it's a problem in pyOpenSSL. OpenSSL is fine. I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python's standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL. It's an implementation artifact of pyOpenSSL's glue code that wraps OpenSSL's C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks. There are ways to work around the problem, but it's complicated and lots of work. Nobody has contributed a solution yet. |
|
This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs. Is there anyone who could take a look at this? |
|
It would take a serious effort and redesign to address the issue. Somebody would have to replace all existing callbacks in PyCA cryptography and pyOpenSSL with a different design. It's a non-trivial effort, because the implementation must be thread-safe and multi subinterpreter-safe. The effort could easily take two weeks or more even for an expert. At work we had similar issues with SELinux enforcement of read-only executable mmap pages. We followed a different approach and replaced pyOpenSSL with Python's ssl module. |
|
@tiran Thanks for the comment! I'll suggest dbt/snowflake-connector do the same thing you folks did. |
|
encounter the same problems on m1 mac |
|
Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞 |
|
Same issue with the new macbook pros M1 (no rosetta) |
|
Same problem on M1 MacBook Pro. It's going to be year 2022 and Apple Silicon is still not supported. |
|
That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful. |
|
And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1. |
|
FWIW, I've been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I've come back to it every few months to see if it was still crashing (and it has been). After seeing @alex's comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.
My test case has always been: $ scrapy shell 'http://httpforever.com/' # <- succeeds
$ scrapy shell 'https://n8henrie.com' # <- crashes with `Cannot allocate write+execute memory for ffi.callback()`For me, both seem to now work without error on |
|
I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips. It works on both Python 3.10.0 and 3.9.9 (the scrapy shell). I need to install PyTorch, but PyTorch isn't working with Python 3.10 right now... And when I was tring to build pytorch from source with M1, pytorch dragged me back to install conda before compiling ,which only supports upto python 3.9.7 right now. It's like a dead loop... |
|
Same thing happening on Python 3.8.9 |
|
|
|
Resolved under newest python 3.10.2 , using conda virturl env |
Can you give a specific way to reproduce the problem? e.g. a short, standalone python script which reproduces it for you? |
|
The root cause of this issue has been found and fixed (for |
|
@sfc-gh-mkeller Thanks for the pointer! It appears the changes have not been merged yet though. Hopefully, the change will be merged and build 3 will be available in the condo-forge channel soon. |
|
Still getting the issue in my Mac Pro 2021 M1 running MacOS Monterey. Tried all suggestions from above:
Still not working. Here's my full enviroment: Anything else I should try? The package I'm trying to use is called pynubank. Here's the full traceback: |
|
@WittmannF You are running build 2 of cffi (py310h2399d43_2) and that does not contain the patch @sfc-gh-mkeller shared. The patch is not yet merged and a new build published yet,. |
|
Thanks @jonbuffington ! I was able to solve it here with |
|
@WittmannF Yep. Until build 3 of (my_env) $ pip install --force-reinstall 'cffi>=1.15.1' |
Confirming that this worked for me on a failing conda environment with only snowflake connector, pandas, and jupyter installed. |
|
The patch @sfc-gh-mkeller found is merged and build 3 is available in the condo-forge channel. The above workaround I mentioned above is no longer necessary. |
It worked here. thanks a lot! |
This is useful! It works on python 3.9.Thank you a lot! |
As an M1 user, it worked for me also without using conda virtual env. Thank you so much! |
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
Trusts the libffi library inside of nixpkgs on Apple devices. When Apple's fork of libffi is not detected, cffi assumes that libffi uses a strategy for creating closures (i.e. callbacks) that is in certain cases susceptible to a security exploit. Based on some analysis I did: https://groups.google.com/g/python-cffi/c/xU0Usa8dvhk I believe that libffi already contains the code from Apple's fork that is deemed safe to trust in cffi. It uses a more sophisticated strategy for creating trampolines to support closures that works on Apple Silicon, while the simple approach that cffi falls back on does not, so this patch enables code that uses closures on M1 Macs again. Notably, pyOpenSSL is impacted and will be fixed by this, reported in pyca/pyopenssl#873 Note that libffi closures still will not work on signed apps without the com.apple.security.cs.allow-unsigned-executable-memory entitlement while libffi/libffi#621 is still open (which I haven't tested but is my best guess from reading). I am hopeful that all of these changes will be upstreamed back into cffi and libffi, and that this comment provides enough breadcrumbs for future maintainers to track and clean this up.
and others
For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Found originally here: scrapy/scrapy#4117
Seems like pyOpenSSL issue, that's why creating it here.
Any advice or workaround is appreciated.
Traceback:
OS:
The text was updated successfully, but these errors were encountered: