microsoft store python support #1362
Comments
|
@zooba this feels like bad packaging on Windows store side 🤔? |
|
@gaborbernat I think bug in virtualenv, because venv module work fine |
|
Not really man. venv provisions the created venv with whatever you have bundled. Here importing the network stack fails (which virtualenv does to install latest pip/setuptools). Pass in a --no-download and virtualenv should also work. |
|
@gaborbernat I don't want virtualenv, but pipenv use it and no have option to pass args |
|
It still boils down to a packaging issue of core, as _socket is not importable 🤔 Will need to validate this though. |
|
@pfmoore this seems pip related 🤔 but strangely manifests when called from virtualenv by putting the wheel on the python path 👍 Command C:\Users\traveler\gi...v\Scripts\python.exe -m pip config list had error code 1
Installing setuptools, pip, wheel...
Complete output from command C:\Users\traveler\gi...v\Scripts\python.exe - setuptools pip wheel:
Traceback (most recent call last):
File "<stdin>", line 10, in <module>
File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_internal\__init__.py", line 19, in <module>
File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\__init__.py", line 8, in <module>
File "C:\Users\traveler\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\virtualenv_support\pip-19.1.1-py2.py3-none-any.whl\pip\_vendor\urllib3\connectionpool.py", line 7, in <module>
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1008.0_x64__qbz5n2kfra8p0\Lib\socket.py", line 49, in <module>
import _socket
ImportError: DLL load failed: Access is denied.Edit: NM, seems our new virtualenv doesn't have access to _socket, my bad 🤦♂ |
|
So trying to import _socket on a freshly created virtualenv fails with: While doing the same from the installed python Now inspecting the access data for this file: Looking at this I'm not sure what's this missing 🤔 |
gaborbernat
changed the title
|
I've never used the Store distribution of Python. I'd suggest asking @zooba for help here, as he's the expert on that distribution. @JoDoX Can you do a simple test? Start up Python, and run That should work with no errors. If you get an error, then the issue is with something in Python, rather than with virtualenv (or pip). |
|
@pfmoore see my post above. Those commands work if starting it from the installed python. However, they fail from within the virtual environment python. Seems to be related somehow with the fact that the DLLs can be only accessed if run from the installed Python. Once you copy the executable it seems to lose the ability to access the upstream DLLs. Not sure given this the issue is virtualenv or Windows store packaging related. Edit: Creating a venv and looking at the startup log shows somehow venvs actually try loading the pyd from the original location. It actually loads everything from the original location. Both Include and Lib are empty. In virtualenv we copy these over for now.... Maybe there's a better way to create venvs than what we have now 🤔 Will need to investigate. vs |
|
@gaborbernat Ah, sorry - I skimmed your post and misunderstood what you were saying. My bad. I certainly wouldn't rule out that copying the Python executable might cause issues. We really need @zooba here, as I'm not aware of any developer-level docs on how the Store build works, and I'm not sufficiently familiar with the code to go diving into it. |
|
Yeah, you can't copy the executable. Once it's outside the app package it won't have its context properly activated by the operating system. You can either symlink it or copy the |
Somewhat offtopic, but can you point me at the code that makes this be the case (and/or the relevant MS docs on this)? I'd like to understand the store app better than I currently do (I basically understand nothing about MS Store apps at the moment), so I don't have to keep pinging you on this stuff ;-) |
|
I'm not sure that this part is even documented, and if it is, it'll probably be deep inside the MSIX docs or possibly in some AppV specifications if you can find that (which is the older version of the isolation technology used). The packages are meant to be used as apps, primarily, which means a single entry point (and prevent other apps from using our DLLs). So we can't Oh, and to be clear, these are all deliberate, by-design OS restrictions. Python did not choose to make any of this happen, so there's no code in CPython for it. Though the parts of launcher.c hidden behind the |
|
Thanks. I did some brief digging, and https://docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes looks like it might be relevant (although I haven't read it yet, I'm mostly posting this so I can find the link again later ;-)) @gaborbernat I suspect the "right" way for virtualenv to support store Python will be to copy the redirector from I'm happy to help out with coding, but I'll need your guidance on the approach you want to take. (Oh, and @zooba, while I have your attention, how well does the store Python co-exist with a "normal" Python install? If I want to test this stuff, will I be better with a clean VM, to avoid interfering with my main Python install? Or can I install and uninstall the store Python with no side-effects?) |
|
So I gave this a good thought. In the end, I think it makes sense to switch the core logic to:
A few places where stdlib provided might venv might not be present:
|
|
I'd be a strong +1 on this. With Python 2 reaching EOL very soon, we're going to be in a situation where basically all supported versions of Python have venv (and those that don't, still have the mechanisms - The hard part is deciding what constitutes the list of guarantees we offer. I'm not actually sure that there are many - the only difficult one is the |
|
I concur with @pfmoore on this. |
|
Just to be clear, Python 2 support is not planned to be dropped at least for the next 3 years. The new solution will just prefer to use stdlib venv where possible and fallback to builtin logic otherwise. |
|
Understood. The existing process needs to remain so that Python 2 is supported, but if we can avoid it where core venv is available that's great. |
|
Started a separate discussion in #1366 👍 |
Sorry I missed this, @pfmoore (Github notifications don't always get my attention...). You can install/uninstall the store Python with no side-effects - this is actually one of it's biggest features. (Though a downside of this is that it can't update the real user registry, and so PEP 514 isn't quite sufficient, since overwriting values would break the clean install/uninstall cycle.) |
|
I think we talked over Twitter that it would not be an issue of overwrite if per pep 514 we use a different company (e.g. Microsoft Store) instead of PythonCore. And then we just need to update py.exe to accept that too. |
|
using venv instead of virtualenv solves the issue for me |
gaborbernat
changed the title
|
The old virtualenv cannot and will not support this. virtualenv 20, the rewrite, should. We need to validate, sadly AFAIK it's not possible to add the store python to the Azure CI. @zooba can you confirm that this still is the case, and will be going ahead? |
|
In first iteration we can just mark the builtin/self-do method not support the store; to check if the python is store pyhton @zooba suggested:
|
|
This now has been achieved as part of #1502. Virtualenv will correctly mark the store python as not compatible with its builtin strategy and fallback to delegate the task to the venv module of the store python. The test suite of the project passes. Furthermore, now we will use the symlink instead of the copy method if the Windows OS supports it (we test this by trying to create a symlink inside a temp folder). Built-in support for Microsoft Store within the virtualenv may be done in a future subsequent ticket, probably by improving our cpython3windows class to use the new techniques needed for this. |
The text was updated successfully, but these errors were encountered: