ffi.callback() issues in code-signed, notarized MacOS app

[mattgwwalker]

Hi,

Do you have any experience with this error message?
builtins.MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

I've spent all week dealing with the joys of code signing my work for macOS, only for this error to show its head on Friday evening.

One option looks to be just to accept that this code is dangerous and to apply the work around of giving my application the entitlement com.apple.security.cs.allow-unsigned-executable-memory. Maybe for a demo that sounds like a sufficient approach, but Apple's not exactly all roses over this: "Including this entitlement exposes your app to common vulnerabilities in memory-unsafe code languages. Carefully consider whether your app needs this exception." CFFI is even less positive:

[...] it is dangerous to allow write+execute memory in your program; that’s why the various “hardening” options above exist. But at the same time, these options open wide the door to another attack: if the program forks and then attempts to call any of the ffi.callback(), then this immediately results in a crash—or, with a minimal amount of work from an attacker, arbitrary code execution. To me it sounds even more dangerous than the original problem, and that’s why cffi is not playing along.

Another option is to look into converting sounddevice to the new style callbacks.

Regarding these new style callbacks, do you have any advice/feedback/experience/bald patches from already pulling your hair out over this?

Cheers,

Matthew

[mattgwwalker]

Oh, this looks like it's going to be a gnarly wee problem.

I added the required entitlement. And sure enough the error went away... only to be replaced with:

sounddevice.PortAudioError: Error opening Stream: Internal PortAudio error [PaErrorCode -9986]

I looked up that error number (in portaudio.h) and paInternalError is indeed what I'm staring at. It appears exactly 100 times in the PortAudio source code. Should be just marvelous working out even which of these 100 errors it might be... let alone fixing it.

Any, even vague, ideas?

I suspect I'm heading toward these new style callbacks rather than going down this current path.

[mgeier]

That's unfortunate ...

I've not experienced those error messages because I'm using Linux.

Any, even vague, ideas?

No, sorry.

Regarding these new style callbacks, do you have any advice/feedback/experience/bald patches from already pulling your hair out over this?

I don't have any experience with the new style callbacks per se, but I guess the main obstacle will be that they only work in API mode, right?

Currently, we are using out-of-line ABI mode.

Switching to API mode is a possibility, but I guess it would be a major undertaking, see #91.

[mattgwwalker]

Thanks for your comments. Given the contents of thread you linked to, I'll be very hesitant to even try to make that conversion to the new callback style. Thank you for the heads up.

Interestingly, I was unable to replicate the error sounddevice.PortAudioError: Error opening Stream: Internal PortAudio error [PaErrorCode -9986] on other systems (running the same version of OSX 10.15.6). I cannot explain what caused it on that one system, but the error seemed to magically disappear after I added the com.apple.security.device.audio-input entitlement too (see #267)... although it may have been fixed by other changes that were applied at the same time.

Thanks again.

[mgeier]

OK, so let's postpone the API-mode discussion ...

I guess it would be good to add instructions for the macOS settings, though?

Would you like to make a PR for this?

[mattgwwalker]

Where would you see such documentation being added? It's not really appropriate for the "installation" section, as that's more for development. Perhaps there needs to be a "packaging" section? But I'm not sure how much there would be to write.

I believe the advice applies only to "frozen" packages and my experience is currently limited only to PyInstaller on the Mac. However, I will soon be looking into the process under Windows.

If you wish your application to run on Catalina (OSX 10.15), you will need to specify the appropriate entitlements, code-sign, and then notarize the application.

To allow sounddevice's callbacks, your application will require the com.apple.security.cs.allow-unsigned-executable-memory entitlement. Callbacks are the recommended approach when using any Stream.

If your application uses the microphone, you will need to specify the entitlement com.apple.security.device.audio-input.

I do not know if the first entitlement is incompatible with Apple's sandboxing requirements for the App Store, however the combination of the above entitlements at least allows you to distribute the application yourself.

Would you like more detail than that? It's quite an elaborate process actually, but most of it is not sounddevice-specific.

[mgeier]

Where would you see such documentation being added?

I don't know.

I was hoping you had an idea.

It's not really appropriate for the "installation" section, as that's more for development. Perhaps there needs to be a "packaging" section?

Ah, now this is interesting!

So the problems you were describing all that time only happen when packaging?

They don't happen when installing the module with pip?

If that's the case, I think we can either make a completely new documentation page (if there is a lot of information), or just add a little section to "Contributing".

If you wish your application to run on Catalina (OSX 10.15), you will need to specify the appropriate entitlements, code-sign, and then notarize the application.

Is this also the case when simply pip-installing the module?

Would you like more detail than that? It's quite an elaborate process actually, but most of it is not sounddevice-specific.

I personally am not a Mac user, so I don't need that information, but I think it would be nice to provide help to other macOS users. Or users of any OS, for that matter. I have no clue whether anybody actually need this, though ...

[mattgwwalker]

So the problems you were describing all that time only happen when packaging?
They don't happen when installing the module with pip?

Correct. The problems listed in this thread are exclusively to do with frozen, code-signed, and notarized apps for MacOS. There are no issues when sounddevice is used with a pip-install.

If you wish your application to run on Catalina (OSX 10.15), you will need to specify the appropriate entitlements, code-sign, and then notarize the application.

Is this also the case when simply pip-installing the module?

No. Sounddevice runs out-of-the-box without issue with a pip-install.

I personally am not a Mac user, so I don't need that information, but I think it would be nice to provide help to other macOS users.

Are you aware that there's a specific sounddevice hook in PyInstaller? I had assumed you'd provided it. I've just found the original pull request; looks like my assumptions were wrong :o) I think it's certainly safe to say that other people use sounddevice with PyInstaller and if they use it on a Mac with OSX10.15 on it then they'll have the issues I did.

[mgeier]

Thanks for the clarification!

Can you please update the issue title to say that only applies to frozen macOS apps (or whatever is the proper term)?

Are you aware that there's a specific sounddevice hook in PyInstaller?

Yes.

I had assumed you'd provided it.

No, it was provided by someone else here (as you have found out yourself): pyinstaller/pyinstaller#4498

See also: #130 (comment)

However, in the meantime the "hooks" seem to have been removed from the main PyInstaller repo and moved to https://github.com/pyinstaller/pyinstaller-hooks-contrib.

The sounddevice hook seems to live at https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-sounddevice.py now.

[mixuala]

Correct. The problems listed in this thread are exclusively to do with frozen, code-signed, and notarized apps for MacOS. There are no issues when sounddevice is used with a pip-install.

I'm trying to install sounddevice but I still the this error. I even used pip-install to no effect. am I doing something wrong?

---

From a vscode juypter notebook and still having problems, I've done the following:

conda install -c conda-forge python-sounddevice

conda install -c conda-forge python-sounddevice
The following NEW packages will be INSTALLED:
portaudio conda-forge/osx-arm64::portaudio-19.6.0-hbdafb3b_4
python-sounddevice conda-forge/noarch::python-sounddevice-0.4.1-pyh9f0ad1d_0

I get:

import sounddevice as sd
print( sd.get_portaudio_version())
sd.play(clips[0], samplerate)

(1246720, 'PortAudio V19.6.0-devel, revision 396fe4b6699ae929d3a685b3ef8a7e97396139a4')
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

also tried python3 -m pip install sounddevice with the same result

% python3 -m pip uninstall sounddevice  
Found existing installation: sounddevice 0.4.1
Uninstalling sounddevice-0.4.1:
  Would remove:
    xxx/miniforge3/lib/python3.9/site-packages/_sounddevice.py
    xxx/miniforge3/lib/python3.9/site-packages/sounddevice-0.4.1.dist-info/*
    xxx/miniforge3/lib/python3.9/site-packages/sounddevice.py
Proceed (y/n)? y
  Successfully uninstalled sounddevice-0.4.1
(base) % python3 -m pip install sounddevice    
Collecting sounddevice
  Using cached sounddevice-0.4.2-py3-none-any.whl (31 kB)
Requirement already satisfied: CFFI>=1.0 in ./xxx/miniforge3/lib/python3.9/site-packages (from sounddevice) (1.14.6)
Requirement already satisfied: pycparser in ./xxx/miniforge3/lib/python3.9/site-packages (from CFFI>=1.0->sounddevice) (2.20)
Installing collected packages: sounddevice
Successfully installed sounddevice-0.4.2

conda info:

     active environment : base
    active env location : xxx/miniforge3
            shell level : 1
       user config file : xxx/.condarc
 populated config files : xxx/miniforge3/.condarc
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.1.final.0
       virtual packages : __osx=11.6=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : xxx/miniforge3  (writable)
      conda av data dir : xxx/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : xxx/miniforge3/pkgs
                          xxx/.conda/pkgs
       envs directories : xxx/miniforge3/envs
                          xxx/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.1 Darwin/20.6.0 OSX/11.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[mgeier]

Sorry, I don't really know anything about this, but I recently released a new version which includes a .dylib for arm64 architectures which might or might not make a difference regarding your problem.