Three dlls should be copied to the base directory of the package on windows #101

mattip · 2023-02-08T22:21:36Z

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

xref conda-forge/miniforge#385. Running the package on windows without adding the Library/bin directory to the path fails. That directory is added by activation, but the interpreter should run without requiring activation. The solution would be to copy libbz2.dll, libexpat.dll, ffi-8.dll into the base directory during the build.

Installed packages

not relevant

Environment info

not relevant

The text was updated successfully, but these errors were encountered:

isuruf · 2023-02-08T22:32:38Z

That would have been an option if symlinks worked, but they don't on windows without elevated permissions.
I'm not sure how to fix this.

isuruf · 2023-02-08T22:34:27Z

Maybe https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/3e82b658-e070-4b88-aaf2-0649ddb222d2/how-to-write-manifest-file-to-pick-up-dlls-via-dll-redirection?forum=vcgeneral

mattip · 2023-02-08T22:38:49Z

Can we add a copy command here in the script?

isuruf · 2023-02-08T22:39:35Z

No, then pypy will be using the DLLs that were used at build time and not the updated DLLs

mattip · 2023-02-08T22:42:30Z

OK. zlib.dll is already there, is that a bug?

BTW, I am working on getting os.symlink, os.readlink to work on windows + PyPy, but this will only be in the next release.

isuruf · 2023-02-08T22:45:58Z

zlib.dll is already there, is that a bug?

Not really. It's in the zlib package itself. We added it because it's a tiny DLL. expat, ffi on the hand are not.

mattip · 2023-02-09T06:14:44Z

Here is what the base directory looks like after copying the three dlls. I am not sure those three add all that much. There is the security problem that once copied, they become part of the release so updating any of them requires updating the feedstock. I think the last few times I saw a problem with libexpat or libbz2 the fixes required recompiling python anyway, so I am not sure how concerning security considerations are.

>dir d:\miniconda\envs\pypy\*.dll
06/05/2022  17:21            21,984 api-ms-win-core-console-l1-1-0.dll
06/05/2022  17:29            21,976 api-ms-win-core-console-l1-2-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-datetime-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-debug-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-errorhandling-l1-1-0.dll
06/05/2022  17:21            21,992 api-ms-win-core-fibers-l1-1-0.dll
06/05/2022  17:21            26,088 api-ms-win-core-file-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-file-l1-2-0.dll
06/05/2022  17:22            21,976 api-ms-win-core-file-l2-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-handle-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-heap-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-interlocked-l1-1-0.dll
06/05/2022  17:22            22,008 api-ms-win-core-libraryloader-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-localization-l1-2-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-memory-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-namedpipe-l1-1-0.dll
06/05/2022  17:21            22,008 api-ms-win-core-processenvironment-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-processthreads-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-processthreads-l1-1-1.dll
06/05/2022  17:21            21,992 api-ms-win-core-profile-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-rtlsupport-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-string-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-synch-l1-1-0.dll
06/05/2022  17:22            21,976 api-ms-win-core-synch-l1-2-0.dll
06/05/2022  17:21            21,984 api-ms-win-core-sysinfo-l1-1-0.dll
06/05/2022  17:21            21,976 api-ms-win-core-timezone-l1-1-0.dll
06/05/2022  17:21            21,992 api-ms-win-core-util-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-conio-l1-1-0.dll
06/05/2022  17:21            26,080 api-ms-win-crt-convert-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-environment-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-filesystem-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-heap-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-locale-l1-1-0.dll
06/05/2022  17:21            30,184 api-ms-win-crt-math-l1-1-0.dll
06/05/2022  17:21            30,176 api-ms-win-crt-multibyte-l1-1-0.dll
06/05/2022  17:21            75,232 api-ms-win-crt-private-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-process-l1-1-0.dll
06/05/2022  17:21            26,104 api-ms-win-crt-runtime-l1-1-0.dll
06/05/2022  17:21            26,080 api-ms-win-crt-stdio-l1-1-0.dll
06/05/2022  17:22            26,072 api-ms-win-crt-string-l1-1-0.dll
06/05/2022  17:21            21,984 api-ms-win-crt-time-l1-1-0.dll
06/05/2022  17:21            22,000 api-ms-win-crt-utility-l1-1-0.dll
12/01/2023  10:53           327,504 concrt140.dll
09/11/2021  22:16            25,600 ffi-8.dll
29/11/2020  01:13           187,904 libbz2.dll
25/10/2022  21:06           401,408 libexpat.dll
18/01/2023  19:13        43,297,280 libpypy3.9-c.dll
12/01/2023  10:53           579,920 msvcp140.dll
12/01/2023  10:53            35,664 msvcp140_1.dll
12/01/2023  10:53           197,456 msvcp140_2.dll
12/01/2023  10:53            50,000 msvcp140_atomic_wait.dll
12/01/2023  10:53            31,568 msvcp140_codecvt_ids.dll
06/05/2022  17:22         1,123,808 ucrtbase.dll
12/01/2023  10:53           414,544 vcamp140.dll
12/01/2023  10:53           345,936 vccorlib140.dll
12/01/2023  10:53           191,824 vcomp140.dll
12/01/2023  10:53           109,392 vcruntime140.dll
12/01/2023  10:53            49,488 vcruntime140_1.dll
14/10/2022  17:02            89,088 zlib.dll

mattip · 2023-02-09T06:17:01Z

Another option would be to load libpypy*.dll via dlopen in pypy.exe, and move the add_dll_directory code to run before that.

mattip · 2023-02-09T11:41:45Z

Here is the area in the code that would need to change to make main() load the libpypy dll dynamically.

mattip · 2023-02-09T12:06:02Z

I tried searching through CPython issues and history for why python.exe is statically linked to libpython.dll and if anyone had considered using dlopen() instead without much luck.

mattip · 2023-05-18T11:44:06Z

Looking at the link above about adding manifests to allow searching other directories, I think the steps are:

turn the dlls into an assembly by adding a python.assembly.manifest file to the Library/bin directory with the content

<?xml version="1.0" encoding="utf-8"?>
<asmv1:assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1" xmlns:asmv1="urn:schemas-microsoft-com:asm.v1" xmlns:asmv2="urn:schemas-microsoft-com:asm.v2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <assemblyIdentity type="Win32" name="python.assembly" version="1.0.0.0" processorArchitecture="AMD64"/>
 <file name="libexpat.dll" hashalg="SHA1"/>
 <file name="libbz2.dll" hashalg="SHA1"/>
 <file name="libffi-8.dll" hashalg="SHA1"/>
</asmv1:assembly>

Get pypy-c.exe to load the assembly by recompiling it with this link option (from the article)

#pragma comment(linker,"/manifestdependency:\"type='win32' "\
  "name='python.assembly' "\
  "version='1.0.0.0' "\
  "processorArchitecture='*' "\
  "language='*' "\
  "\"")

The source code for the exe (main.c) is one line:

int pypy_main_startup(int, char*[]); int main(int argc, char* argv[]) { return pypy_main_startup(argc, argv); }

and the commands to compile it is:

cl.exe /I<pypy-includes> main.c -o main.obj
link.exe /DEBUG /LARGEADDRESSAWARE /STACK:3145728 main.obj libpypy3.9-c.lib /out:pypy3.9-c.exe

~~I am not sure how that link option from the article would work here, would it be part of main.c?~~ So adding the content to force use of the assembly would make main.c into

#pragma comment(linker,"/manifestdependency:\"type='win32' "\
  "name='python.assembly' "\
  "version='1.0.0.0' "\
  "processorArchitecture='*' "\
  "language='*' "\
  "\"") 

int pypy_main_startup(int, char*[]); int main(int argc, char* argv[]) { return pypy_main_startup(argc, argv); }

Once the exe is compiled to use the assembly, an appropriate ~~pypy3.9-c.exe.manifest~~ pypy3.9-c.exe.config should be added to the directory with the exe, with the contents:

<configuration>
 <windows>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <probing privatePath="Library/bin"/>
 </assemblyBinding>
 </windows>
</configuration>

~~Did I miss something?~~ This seems to work locally!

mattip · 2023-05-19T14:21:32Z

@isuruf it seems a bit silly to implement this for these three DLLS when conda-forge copies about 50 runtime dlls into the base directory (as well as duplicating them for every environment into Library/bin

karthiknadig · 2023-07-12T18:51:15Z

This is affecting tools and extensions for VS Code, see : microsoft/vscode-isort#291

ndahn · 2023-08-08T14:15:46Z

As @mattip commented, there are already a few dozen libraries copied into the main folder, and the ffi-8, bz2 and expat are about half a megabyte in total. For 99.999% of all systems this won't make a difference, and the rest will heavily optimize any installation anyways.

I don't see why such a simple solution should be ignored. If there is ever an issue because of 3 duplicate files, you can still create some assembly or whatever 🙄

mattip added the bug Something isn't working label Feb 8, 2023

mattip mentioned this issue Jul 6, 2023

Use of PyPy crashed iSort server microsoft/vscode-isort#291

Closed

mattip mentioned this issue Jul 13, 2023

Mambaforge-PyPy windows installer is broken conda-forge/miniforge#385

Closed

This was referenced Aug 9, 2023

PyPy 7.3.12 was released with python 2.7, python 3.9, python 3.10 #104

Closed

V7.3.12 #105

Merged

mattip added a commit to mattip/pypy3.6-feedstock that referenced this issue Aug 28, 2023

remove fix for issue conda-forge#101, from review

8b70c87

isuruf mentioned this issue Sep 1, 2023

Fix running pypy.exe without activating #107

Merged

5 tasks

mattip closed this as completed in #107 Sep 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Three dlls should be copied to the base directory of the package on windows #101

Three dlls should be copied to the base directory of the package on windows #101

mattip commented Feb 8, 2023

isuruf commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented May 18, 2023 •

edited

Loading

mattip commented May 19, 2023

karthiknadig commented Jul 12, 2023

ndahn commented Aug 8, 2023

Three dlls should be copied to the base directory of the package on windows #101

Three dlls should be copied to the base directory of the package on windows #101

Comments

mattip commented Feb 8, 2023

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

isuruf commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 8, 2023

isuruf commented Feb 8, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented Feb 9, 2023

mattip commented May 18, 2023 • edited Loading

mattip commented May 19, 2023

karthiknadig commented Jul 12, 2023

ndahn commented Aug 8, 2023

mattip commented May 18, 2023 •

edited

Loading