On Windows, console script wrapper (exe) created by pip cannot process non-ANSI (Unicode) arguments

[fireattack]

Description

When install a module using pip that has a console script, the created XXX.exe wrapper cannot process non-ASNI arguments properly (characters become ???).

This issue has been reported before (pypa/setuptools#3268, #11040), but was closed due to "it's your console's issue".

Windows CMD does have its issues, but it can handle non-ANSI arguments fine. If I just feed them in directly into python, or using the companioned echotest-script.py, everything works.

See STR below for demonstration for detail.

Expected behavior

The wrapper should be able to handle non-ANSI characters.

pip version

23.0

Python version

3.11

OS

Win10 22H2

How to Reproduce

Here is a simple example.

Create a folder with two files:

setup.py:

from setuptools import setup

setup(
    name='echotest',
    version='1.0',
    description='echo test',
    author='test',
    author_email='[email protected]',
    install_requires=[],
    requires=[],
    packages=['echotest'],
    platforms=["Linux", "Mac OS-X", "Windows", "Unix"],
    entry_points={
        'console_scripts': [
           'echotest = echotest.__main__:main',
        ],
    },
)

echotest\__main__.py:

import sys

def main():
    print(sys.argv[1:])

if __name__ == "__main__":
    main()

Install this package, using pip install .

Now, change CWD to another folder (to avoid name confliction), and try the following (assuming /Scripts is already in %path%):

python -m echotest 中文
python "C:\Users\{yourusername}\AppData\Local\Programs\Python\Python311\Scripts\echotest-script.py" 中文
echotest 中文

Output

C:\files\echotest>pip install .           
Processing c:\files\echotest
  Preparing metadata (setup.py) ... done
Installing collected packages: echotest
  Attempting uninstall: echotest
    Found existing installation: echotest 1.0
    Can't uninstall 'echotest'. No files were found to uninstall.
  DEPRECATION: echotest is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for echotest ... done
Successfully installed echotest-1.0

D:\>python -m echotest 中文
['中文']

D:\>python C:\Users\****\AppData\Local\Programs\Python\Python311\Scripts\echotest-script.py 中文
['中文']

D:\>echotest 中文
['??']

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

The reproduction works for me without error. I suspect this is because I'm using Windows Terminal and Powershell, which have "better" UTF-8 handling than cmd. Having said that, it also works for me in cmd.

So this is somehow related to your console environment, which isn't to say it's not a problem, just that (1) it will probably improve over time as environments get better UTF-8 support and (2) it's hard for maintainers to fix as they can't reproduce the issue.

In this case, I suspect that the issue is with the script wrappers pip uses, which come from the distlib project. So it may be worth your while trying to reproduce this issue directly with the APIs in distlib.scripts (documented here), and then report the problem to that project.

[fireattack]

I can reproduce it in PS too. Which version do you use?

(I also have no idea why the Chinese glyphs don't show up properly in input only).

[pfmoore]

7.3.2

[fireattack]

Thanks. Unfortunately, neither works here, plain or in WT.

[pfmoore]

I sympathise, but this isn't a pip issue. As I said, you may be able to produce a test case using just distlib and report that to the distlib project, but that's about all I can suggest.

I'm going to close this, as I don't think it's something that needs fixing in pip (if it gets fixed in distlib, we'll pick up any fix when we vendor the new version of distlib).

[fireattack]

No worries, I will try to test it directly with distlib once I have time.

For what it's worth, I just removed setup.py and use a pyproject.toml instead:

[project]
name = "echotest"
version = "1.0"
dependencies = [
]

[project.scripts]
echotest = "echotest.__main__:main"

After re-installing, the new echotest.exe (now without echotest-script.py) works perfectly.

If you can share any insight about why they build the wrapper differently, it would be very helpful.

But either way, thanks for help.

[pfmoore]

Huh, that's weird.

Oh, wait. If you are getting echotest-script.py you are using the legacy code path in pip that invokes setup.py directly. You shouldn't be getting that. Let me test again.

Ah, you get the message "DEPRECATION: echotest is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at #8559" in your output. I don't, because I don't have the necessary condition to trigger it (setuptools installed but wheel not installed).

Having said that, I still don't get the issue - but setuptools does use a different wrapper. Maybe you have an old version of setuptools? What does pip show setuptools say?

[fireattack]

Name: setuptools
Version: 65.5.0

[pfmoore]

Hmm, I tried back to 54.2.0, and didn't reproduce the issue. I suspect it's a bug in the wrappers setuptools uses - but they may not be willing to fix it as we're moving away from that installation method. I'd say the best fix is simply to not use the legacy install method - either add a pyproject.toml, or use --use-pep517, or even just install wheel in your environment alongside setuptools.

[fireattack]

I was misleaded by pypa/setuptools#3268 (comment) & #11040 and thought it was pip who builds the launcher.

After you pointed it back to setuptools, I quickly find pypa/setuptools#595

So apparently it's a known issue (due to using outdated C API in Windows), just that it likely won't be fixed since distlib would be the future, as you also suggested.

I call this case closed. Again, thanks for your help!

[pfmoore]

Glad to help - it was an interesting question to investigate!