PermissionsError when accessing some directories with os module #38

mrclary · 2022-09-18T03:20:40Z

🐛 Bug

A PermissionError is raised when using os.scandir with the conda-based application bundle.
I've noticed the same issue with Spyder as well with conda-based application bundle.

This issue only manifests with the three user directories ~/Desktop, ~/Documents, and ~/Downloads (and anything therein). No other directories on the system produce the error.

Interestingly, this issue does not manifest when launching the application executable directly.

$ ~/Applications/napari\ \(0.4.16rc7\).app/Contents/MacOS/napari_\(0.4.16rc7\)

I've observed identical behavior on two separate machines:

iMac with arm64 running 12.5.1
Macbook Pro with x86_64 running 12.5.1

To Reproduce

Steps to reproduce the behavior:

Launch condo-based napari application bundle napari (0.4.16rc7).app from Finder
View the iPython console
Attempt os.scandir for any of the three user directories ~/Desktop, ~/Documents, or ~/Downloads
Observe the error

In [1]: import os

In [2]: os.scandir('/Users/ryan/')
Out[2]: <posix.ScandirIterator at 0x179e18960>

In [3]: os.scandir('/Users/ryan/Documents/')
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 os.scandir('/Users/ryan/Documents/')

PermissionError: [Errno 1] Operation not permitted: '/Users/ryan/Documents/'
/Users/ryan/Library/napari-0.4.16rc7/envs/napari-0.4.16rc7/lib/python3.9/site-packages/ipykernel/kernelbase.py:751: RuntimeWarning: coroutine 'InProcessKernel._abort_queues' was never awaited!

Environment

napari: 0.4.16rc7
Platform: macOS-12.5.1-x86_64-i386-64bit
System: MacOS 12.5.1
Python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:52) [Clang 13.0.1 ]
Qt: 5.15.3
PySide2: 5.15.4
NumPy: 1.22.4
SciPy: 1.8.1
Dask: 2022.05.2
VisPy: 0.9.6

OpenGL:

GL version: 2.1 Metal - 76.3
MAX_TEXTURE_SIZE: 16384

Screens:

screen 1: resolution 2240x1260, scale 2.0

Plugins:

console: 0.0.4
napari-svg: 0.1.6
scikit-image: 0.4.16rc7

The text was updated successfully, but these errors were encountered:

mrclary · 2022-09-19T19:57:06Z

FYI @jaimergp

jaimergp · 2022-09-19T20:37:17Z

I suspect this has to do with the XML configuration shipped in the shortcut. I'll look into this, thanks for the report!

mrclary · 2022-09-19T21:20:51Z

I suspect this has to do with the XML configuration shipped in the shortcut. I'll look into this, thanks for the report!

That's what I'm hoping; that should be relatively easy to fix. However, I've been unable to see any affect by adding the following values to Spyder's info.plist (from our py2app generated info.plist, which does not exhibit the issue):

additional info.plist

	<key>CFBundleVersion</key>
	<string>0.0.0</string>


	<key>CFBundleInfoDictionaryVersion</key>
	<string>6.0</string>
	<key>CFBundleSignature</key>
	<string>????</string>
	<key>CFBundleDevelopmentRegion</key>
	<string>English</string>
	<key>CFBundleDocumentTypes</key>
	<array>
		<dict>
			<key>CFBundleTypeExtensions</key>
			<array>
				<string>py</string>
				<string>pyw</string>
				<string>ipy</string>
				<string>pyx</string>
				<string>pxd</string>
				<string>pxi</string>
				<string>c</string>
				<string>h</string>
				<string>cc</string>
				<string>cpp</string>
				<string>cxx</string>
				<string>h</string>
				<string>hh</string>
				<string>hpp</string>
				<string>hxx</string>
				<string>cl</string>
				<string>f</string>
				<string>for</string>
				<string>f77</string>
				<string>f90</string>
				<string>f95</string>
				<string>f2k</string>
				<string>f03</string>
				<string>f08</string>
				<string>pro</string>
				<string>m</string>
				<string>jl</string>
				<string>yaml</string>
				<string>yml</string>
				<string>patch</string>
				<string>diff</string>
				<string>rej</string>
				<string>bat</string>
				<string>cmd</string>
				<string>txt</string>
				<string>txt</string>
				<string>rst</string>
				<string>po</string>
				<string>pot</string>
				<string>nsi</string>
				<string>nsh</string>
				<string>scss</string>
				<string>css</string>
				<string>htm</string>
				<string>html</string>
				<string>xml</string>
				<string>js</string>
				<string>json</string>
				<string>ipynb</string>
				<string>enaml</string>
				<string>properties</string>
				<string>session</string>
				<string>ini</string>
				<string>inf</string>
				<string>reg</string>
				<string>cfg</string>
				<string>desktop</string>
				<string>md</string>
			</array>
			<key>CFBundleTypeName</key>
			<string>Text File</string>
			<key>CFBundleTypeRole</key>
			<string>Editor</string>
		</dict>
	</array>
	<key>LSEnvironment</key>
	<dict>
		<key>SPY_BRANCH</key>
		<string>(HEAD detached at pull/19485/merge)</string>
		<key>SPY_COMMIT</key>
		<string>f40da3195</string>
	</dict>
	<key>LSHasLocalizedDisplayName</key>
	<false/>
	<key>NSAppleScriptEnabled</key>
	<false/>
	<key>NSHumanReadableCopyright</key>
	<string>Copyright not specified</string>
	<key>NSMainNibFile</key>
	<string>MainMenu</string>
	<key>NSPrincipalClass</key>
	<string>NSApplication</string>
	<key>NSRequiresAquaSystemAppearance</key>
	<false/>
	<key>PyMainFileNames</key>
	<array>
		<string>__boot__</string>
	</array>
	<key>PyOptions</key>
	<dict>
		<key>alias</key>
		<false/>
		<key>argv_emulation</key>
		<false/>
		<key>emulate_shell_environment</key>
		<true/>
		<key>no_chdir</key>
		<false/>
		<key>prefer_ppc</key>
		<false/>
		<key>site_packages</key>
		<false/>
		<key>use_faulthandler</key>
		<false/>
		<key>use_pythonpath</key>
		<false/>
		<key>verbose</key>
		<false/>
	</dict>
	<key>PyResourcePackages</key>
	<array/>
	<key>PyRuntimeLocations</key>
	<array>
		<string>@executable_path/../Frameworks/libpython3.9.dylib</string>
	</array>
	<key>PythonInfoDict</key>
	<dict>
		<key>PythonExecutable</key>
		<string>/Users/runner/hostedtoolcache/Python/3.9.14/x64/bin/python</string>
		<key>PythonLongVersion</key>
		<string>3.9.14 (main, Sep  7 2022, 14:27:29)
[Clang 12.0.0 (clang-1200.0.32.29)]</string>
		<key>PythonShortVersion</key>
		<string>3.9</string>
		<key>py2app</key>
		<dict>
			<key>alias</key>
			<false/>
			<key>template</key>
			<string>app</string>
			<key>version</key>
			<string>0.28.2</string>
		</dict>
	</dict>

I was hoping to affect some behavior, but it did not even break the application. 🤷🏼

mrclary · 2022-09-24T04:26:47Z

So I've got some new information. I've been able to resolve this issue for Spyder in two independent ways:

Compiling the executable script into a binary. I had to install shc.

$ brew install sch
$ mv ~/Applications/Spyder.app/Contents/MacOS/Spyder ~/Applications/Spyder.app/Contents/MacOS/Spyder.bak
$ shc -o ~/Applications/Spyder.app/Contents/MacOS/Spyder -f ~/Applications/Spyder.app/Contents/MacOS/Spyder.bak

Changing the shabang line in the executable from /bin/bash to /bin/sh. Don't ask me why this makes a difference 🤷🏼‍♂️

I'm assuming, at this point, that issue has something to do with security policies in macOS. I only have macOS 12, so I don't know if this issue manifests on older systems. Perhaps something in the security policy distinguishes between scripts and executables? Perhaps there's an exception for sh (the default system shell?), as opposed to bash and zsh?

So, the former solution seems impractical, as shc may not be available on a user's system and, therefore, would have to be bundled with the installer package. The latter seems to be a simple fix by updating constructor (is there an option for the shabang?) or fixing with a post-install script.

However, because I don't fully understand what is going on, I'm concerned that there may be a more fundamental security issue at play here that needs to be addressed properly.

jaimergp · 2022-09-26T10:29:39Z

Intriguing.

I found this thread, which points out that /bin/sh is just a re-execing wrapper for whatever default shell is configured. From man sh:

NAME
     sh -- POSIX-compliant command interpreter

SYNOPSIS
     sh [options]

DESCRIPTION
     sh is a POSIX-compliant command interpreter (shell).  It is imple-
     mented by re-execing as either bash(1), dash(1), or zsh(1) as deter-
     mined by the symbolic link located at /private/var/select/sh.  If
     /private/var/select/sh does not exist or does not point to a valid
     shell, sh will use one of the supported shells.

FILES
     /private/var/select/sh

     $HOME/.profile

     /etc/profile

Also from this guide:

Scripting

MAC presents some serious challenges for scripting because scripts are run by interpreters and the system can’t distinguish file system operations done by the interpreter from those done by the script. For example, if you have a script that needs to manipulate files on your desktop, you wouldn’t want to give the interpreter that privilege because then any script could do that.

The easiest solution to this problem is to package your script as a standalone program that MAC can use for its tracking. This may be easy or hard depending on the specific scripting environment. For example, AppleScript makes it easy to export a script as a signed app, but that’s not true for shell scripts.

TCC and Main Executables

TCC expects its bundled clients — apps, app extensions, and so on — to use a native main executable. That is, it expects the CFBundleExecutable property to be the name of a Mach-O executable. If your product uses a script as its main executable, you are likely to encounter TCC problems. To resolve these, switch to using a Mach-O executable.

So there's a chance we need to do something similar to what the Python launcher system for Windows does:

Ship a small binary launcher.
The small binary launcher finds the script next to it and execs it.
For example, the binary launcher would be called Spyder, and the script would be named Spyder-script.

Possible launchers we can use:

Create a generic launcher with https://github.com/RJVB/DropScript
Create a small Platypus project - https://sveinbjorn.org/platypus
A subset of https://github.com/python/cpython/blob/main/PC/launcher2.c
Craft something with Obj-C / Swift / plain C, as in https://stackoverflow.com/questions/412562/execute-a-terminal-command-from-a-cocoa-app

Or borrow something from CPython itself. I found this bug report with related (but not too similar) problems.

mrclary · 2022-09-26T15:33:45Z

Some new development.
First a discovery: changing the shabang from /bin/bash to /bin/sh worked on the machine I was testing because I happen to have sh listed for Full Disk Access. Giving full disk access to /bin/bash also works when using that in the shabang.

So changing the shabang is not a general solution. I also think requiring users to manually add /bin/bash to the Full Disk Access list is not acceptable. Besides the inconvenience of an addition user-required step at installation, it will not narrowly apply to the installed application (e.g. napari or Spyder), but apply to any application that would similarly use /bin/bash.

macOS is supposed alert the user to applications requesting disk access via a dialog:

However, this is not invoked if the application executable is a script; it seems to only be invoked if the executable is a binary file. I've tried LSFileQuarantineEnabled in info.plist without affect. I've only been able to trigger this disk access dialog when a binary executable is used for the application.

So a binary executable seems to be the most desirable way to be consistent with macOS disk access protocols and maintain convenience for the end user.

The most elegant solution would be to have menuinst compile the application executable at install time by the mere addition of a subprocess call on the script ($ shc -o outfile -f script). This seems the most straight-forward, but would require building a standalone version of shc (https://github.com/neurobin/shc) to be included with menuinst (and subsequently in constructor).

However, to avoid requiring a standalone shc, the application executable would need to be compiled before or at build time, not install time. So how do we allow install-time parameters to be used with a precompiled binary? I submit two possible solutions.

Use a dummy binary executable

A generic shell script can be compiled and pre-signed by the developer and included in the application's menu package repo (e.g. napari-menu), or perhaps signed at the conda-build step. The only reference in the binary executable is the launch script name which must be known a-priori, but is provided to menuinst's yaml configuration
Myapp

#!/bin/bash
${dirname "$0"}/Myapp.sh

menuinst then creates the application executable script as it already does but names it with the .sh suffix, and moves the compiled "dummy" executable into place.
Myapp.sh

#!/bin/bash
eval "$("{{ CONDA_ROOT }}/_conda.exe" shell.bash activate "{{ CONDA_PREFIX }}")"
{{ COMMAND }}

Where CONDA_ROOT and CONDA_PREFIX are interpolated by menuinst at install time, as normal.

Myapp.app
└── Contents
    ├── Info.plist
    ├── MacOS
    │   ├── Myapp
    │   ├── Myapp.sh
    │   └── python -> {{ CONDA_PREFIX }}/bin/python

Use `LSEnvironment` variables

Rather than hard-coding install-time values into the application executable, menuinst can put these values in specifically named environment variables in the info.plist file at install time. Then the application executable can reference these variables and still be compiled and signed prior to packaging and distribution. The compiled binary would need to be provided by the developer in the menu package (e.g. napari-menu) and menuinst would need to made aware of it (e.g. specified in the configuration yaml file). Ranther than creating the application executable, menuinst merely moves the binary that exists in the installed (napari-menu) package to the appropriate location. For example:
Myapp

#!/bin/bash
eval "$("$_CONDA_ROOT/_conda.exe" shell.bash activate "$_CONDA_PREFIX")"
{{ COMMAND }}

This script is pre-compiled by the developer and references install-time variables _CONDA_ROOT AND _CONDA_PREFIX. menuinst places the values of these (and other desirable runtime/install-time variables) into info.plist's LSEnvironment at install-time.

Myapp.app
└── Contents
    ├── Info.plist
    ├── MacOS
    │   ├── Myapp
    │   └── python -> {{ CONDA_PREFIX }}/bin/python

With the current state of the tools on napari/label/bundle_tools I think I can achieve either of the above solutions using post-install scripts. I still need to test pre-signing the binary; I'm sure that constructor will not locate and sign it the way it does so for _conda.exe. Nevertheless, both of these solutions would make the process more convoluted for the developer.

What are your thoughts, @jaimergp?

mrclary · 2022-09-26T16:16:57Z

@jaimergp, looks I posted before I read your latest comment. It seems we are on the same page here.

For example, AppleScript makes it easy to export a script as a signed app, but that’s not true for shell scripts.

I wonder if osacompile can create the binary for us, then. If so, is that available on systems without xcode command line tools installed, i.e. can we depend on it being available?

mrclary · 2022-09-29T22:55:19Z

For Spyder, for now, I'm just doing the following:

brew install shc on CI.
Provide the application executable script Spyder.sh that is compiled in the patched conda.sh in the feedstock,
```
$ shc -o Spyder -f Spyder.sh
```
I'm not using a separate conda package for the menuinst configuration, so the feedstock patch also includes populating the Menu directory.
Copy the compiled executable to the application (replacing whatever menuinst created) with the post-install script.
Patch Info.plist to include install-time variables in LSEnvironment with the post-install script.

It seems to me that two good features for menuinst would be:

The ability to specify an application executable script in the configuration yaml as an alternative to the command option.
The ability to specify LSEnvironment variables in the yaml. Currently menuinst appears to only allow a limited set of keys to add to the Info.plist, which does not include LSEnvironment.
Automatically, or optionally, include install-time variables in LSEnvironment.

In fiddling around with this stuff I discovered that code signing the application executable does not appear to be necessary, and in fact does not work. I assumed that it would need to be code signed when included in the package installer in order for the package installer to be notarized. However, constructor somehow breaks the code signature, preventing the application from launching after installation. But the package installer seems to successfully notarize even when this is not signed. I don't know how notarizing succeeds in this case, but 🤷🏼‍♂️. Note that notarizing does fail if _conda.exe is not signed.

jaimergp · 2022-09-30T10:50:57Z

If the executable is coming from the feedstock (and hence the conda package), it might contain a hardcoded PREFIX somewhere, which gets replaced by conda at install time. That would break the signature. That's why I want a path agnostic executable (à la cli-64.exe in conda and conda-build).

I think we can provide LSEnvironment in the schema just fine, that shouldn't be a problem. I am not so sure about depending on shc stuff, though. Ideally we can just create a universal launcher instead.

Btw, I know I said I will work on this (and I am, but I had to rebase a few branches first). Right now, I am juggling a few tasks before tackling the whole thing in this issue, but I am reading your progress! It's super welcome and rest assured we will make this easier. Thanks a lot!

mrclary · 2022-09-30T15:57:14Z

If the executable is coming from the feedstock (and hence the conda package), it might contain a hardcoded PREFIX somewhere, which gets replaced by conda at install time. That would break the signature.

The feedstock conda.sh is modified to compile the script into the Menu directory of the conda package. It does not contain any hard-coded PREFIX, unless I'm misunderstanding. I'd have to double check at which stage the executable gets broken. I know it's not at conda package creation time. I suspect it is the constructor at installer package creation, not at install time.

That's why I want a path agnostic executable (à la cli-64.exe in conda and conda-build).

I'm not familiar with this or how the binary is compiled. But a path agnostic executable should work just as well; I think this is similar to the "dummy" executable in my previous comment.

I think we can provide LSEnvironment in the schema just fine, that shouldn't be a problem.

👍🏼

I am not so sure about depending on shc stuff, though. Ideally we can just create a universal launcher instead.

This was simply the best solution I found so far for compiling a shell script to binary. I would certainly welcome better solutions.

Btw, I know I said I will work on this (and I am, but I had to rebase a few branches first). Right now, I am juggling a few tasks before tackling the whole thing in this issue, but I am reading your progress! It's super welcome and rest assured we will make this easier. Thanks a lot!

Thank you for all your work! It has been indispensable for Spyder and I'm learning a lot.

jaimergp · 2022-10-30T17:24:12Z

osacompile does not create Mach-O binaries; the compiled files still need to be run with osascript so it won't work for our use case. shc is not available on conda-forge so if we can avoid that route, better; I am investigating the C-launcher now.

jaimergp · 2022-10-30T21:10:46Z

I have a working launcher written in C in this PR. I need to do some tests to see if this is enough to workaround the security limitations, but I am guessing it is? 🤞

jaimergp · 2022-11-18T17:14:11Z

The launcher alone did not work 🤷
However, if I sign the shortcut.app with no certificate (self signed) and some entitlements, it does prompt for permission! I now wonder if we need the C launcher or not, but I guess it won't hurt :D I added some support here, but I still need to do some more debugging.

jaimergp · 2022-11-18T17:29:59Z

Ok, confirmed both are needed 😅 That's nice to know; the C launcher wasn't in vain!

Look how cute the permissions request is now :)

And it appears here in the System Preferences as well:

goanpeca · 2022-11-18T17:35:02Z

Lovelly!

jaimergp · 2022-11-21T23:02:54Z

OMG, finally. The latest artifacts in #42 correctly implement permissions for macOS. This will be closed as soon as the work in that PR is cleaned up.

@mrclary, you can use the packages in napari/label/bundle_tools_2, which now rely on the latest constructor main + the menuinst changes on top. This doesn't need shc or anything. A minimal C launcher is bundled.

The JSON files support new keys to address your concerns with environment variables, pre-activation logic and entitlements for correct permissions.

Thank you so much for the report and constant feedback!

mrclary · 2022-11-21T23:06:39Z

@jaimergp, great! I'll take a look at implementing for Spyder.

mrclary added the bug Something isn't working label Sep 18, 2022

jaimergp self-assigned this Sep 26, 2022

mrclary mentioned this issue Oct 3, 2022

PR: Create conda-based application installer for macOS and Linux spyder-ide/spyder#19461

Merged

2 tasks

jaimergp transferred this issue from napari/napari Oct 30, 2022

jaimergp mentioned this issue Nov 3, 2022

Upstream constructor changes from fork to project #15

Closed

jaimergp mentioned this issue Nov 22, 2022

Unset PYTHON* vars and use entitlements in macOS conda menu shortcut napari/napari#5354

Merged

10 tasks

jaimergp linked a pull request Dec 8, 2022 that will close this issue

Update installer infra to catch up with upstream (part two) #49

Merged

liu-ziyang closed this as completed in #49 Dec 13, 2022

jaimergp mentioned this issue Feb 25, 2023

PermissionsError when accessing some directories with os module conda/menuinst#123

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PermissionsError when accessing some directories with os module #38

PermissionsError when accessing some directories with os module #38

mrclary commented Sep 18, 2022

mrclary commented Sep 19, 2022

jaimergp commented Sep 19, 2022

mrclary commented Sep 19, 2022 •

edited

Loading

mrclary commented Sep 24, 2022

jaimergp commented Sep 26, 2022

Scripting

TCC and Main Executables

mrclary commented Sep 26, 2022

mrclary commented Sep 26, 2022

mrclary commented Sep 29, 2022

jaimergp commented Sep 30, 2022

mrclary commented Sep 30, 2022

jaimergp commented Oct 30, 2022

jaimergp commented Oct 30, 2022

jaimergp commented Nov 18, 2022

jaimergp commented Nov 18, 2022

goanpeca commented Nov 18, 2022

jaimergp commented Nov 21, 2022

mrclary commented Nov 21, 2022

PermissionsError when accessing some directories with os module #38

PermissionsError when accessing some directories with os module #38

Comments

mrclary commented Sep 18, 2022

🐛 Bug

To Reproduce

Environment

mrclary commented Sep 19, 2022

jaimergp commented Sep 19, 2022

mrclary commented Sep 19, 2022 • edited Loading

mrclary commented Sep 24, 2022

jaimergp commented Sep 26, 2022

Scripting

TCC and Main Executables

mrclary commented Sep 26, 2022

Use a dummy binary executable

Use LSEnvironment variables

mrclary commented Sep 26, 2022

mrclary commented Sep 29, 2022

jaimergp commented Sep 30, 2022

mrclary commented Sep 30, 2022

jaimergp commented Oct 30, 2022

jaimergp commented Oct 30, 2022

jaimergp commented Nov 18, 2022

jaimergp commented Nov 18, 2022

goanpeca commented Nov 18, 2022

jaimergp commented Nov 21, 2022

mrclary commented Nov 21, 2022

mrclary commented Sep 19, 2022 •

edited

Loading

Use `LSEnvironment` variables