Pytest debug option "--debug" has problems to write non ASCII characters

[Tiarles]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Introduction

In my tests, I have some tests to check nonascii names for files. The test consists in import and compiles theses tests and everything works fine, but at the moment that I try to take a look in the execution of the hooks using the pytest --debug option the execution fail because pytest debug can't write the ascii character present in the Node ID of these tests, returning a UnicodeEncodeError.

pip list command

aenum 2.2.4
alabaster 0.7.12
allure-pytest 2.8.10
allure-python-commons 2.8.10
appy 0.9.6
Arpeggio 1.9.2
atomicwrites 1.4.0
attrs 20.2.0
Babel 2.8.0
backcall 0.2.0
backports.functools-lru-cache 1.6.1
bcrypt 3.2.0
bitstruct 8.11.0
cached-property 1.5.1
cantools 32.16.0
certifi 2020.6.20
cffi 1.14.2
chardet 3.0.4
Click 7.0
colorama 0.4.3
control 0.8.3
cparser 0.1.3
cryptography 3.1
cycler 0.10.0
Cython 0.29.21
decorator 4.4.2
diskcache 5.0.3
docutils 0.14
dss-python 0.10.6
FMPy 0.2.14
future 0.18.2
idna 2.10
imagesize 1.2.0
iniconfig 1.0.1
ipython 7.18.1
ipython-genutils 0.2.0
jedi 0.14.1
Jinja2 2.11.2
kiwisolver 1.2.0
lark-parser 0.9.0
lxml 4.5.2
MarkupSafe 1.1.1
matplotlib 3.1.2
model-converter 1.4.4
more-itertools 8.5.0
npTDMS 0.23.0
numexpr 2.7.1
numpy 1.19.0
OpenDSSDirect.py 0.4.0
openpyxl 1.7.0
packaging 20.4
pandas 1.0.5
paramiko 2.7.1
parglare 0.10.0
parso 0.5.1
pathlib 1.0.1
Paver 1.3.4
PeakUtils 1.3.3
pickleshare 0.7.5
Pillow 7.2.0
pip 20.2.3
pluggy 0.13.1
prompt-toolkit 3.0.7
psutil 5.7.2
py 1.9.0
pycparser 2.20
pycryptodomex 3.9.8
pyFFTW 0.12.0
pygit2 1.2.1
Pygments 2.7.0
pyModbusTCP 0.1.8
PyNaCl 1.4.0
pyparsing 2.4.0
PyQt5 5.15.0
PyQt5-sip 12.8.1
pyqtgraph 0.11.0+gabfac52
PyQtWebEngine 5.15.0
pyserial 3.4
pytest 5.4.1
pytest-assume 2.2.0
python-can 3.3.3
python-dateutil 2.8.0
pytz 2019.1
pyusb 1.0.2
pywin32 228
pyzmq 19.0.1
QScintilla 2.11.4
requests 2.24.0
scipy 1.5.1
setuptools 47.1.0
sip 5.4.0
six 1.12.0
snowballstemmer 2.0.0
Sphinx 1.7.0
sphinxcontrib-programoutput 0.16
sphinxcontrib-serializinghtml 1.1.4
sphinxcontrib-websupport 1.2.4
tables 3.6.1
tabulate 0.8.3
textparser 0.23.0
textX 2.1.0
toml 0.10.1
traitlets 5.0.4
urllib3 1.25.10
wcwidth 0.2.5
wheel 0.35.1
win-inet-pton 1.1.0
windows-curses 2.1.0
wrapt 1.12.1
yapf 0.28.0

Pytest and OS versions

pytest-6.0.2 - Windows 10 (Home Edition)

A brief example

# In test_sample.py
def test4_чћшђ_čćšđ():
    assert True

# Called using ``pytest test_sample.py`` -> OK
# Called using ``pytest --debug test_sample.py`` -> UnicodeEncodeError

Conclusion

Any doubt about my explanation about this issue just ping me, Thanks!

[The-Compiler]

Can you please show the full stacktrace?

[Tiarles]

I'm a little noob. What is exactly a "full stacktrace"?

[Tiarles]

Maybe this is what are you talking about:

===== ERRORS =====
____ ERROR collecting test session ____
venv\lib\site-packages\py\_path\common.py:383: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
venv\lib\site-packages\py\_path\common.py:435: in gen
    for p in self.gen(subdir):
venv\lib\site-packages\py\_path\common.py:435: in gen
    for p in self.gen(subdir):
venv\lib\site-packages\py\_path\common.py:424: in gen
    dirs = self.optsort([p for p in entries
venv\lib\site-packages\py\_path\common.py:425: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
venv\lib\site-packages\_pytest\nodes.py:569: in _recurse
    if ihook.pytest_ignore_collect(path=dirpath, config=self.config):
venv\lib\site-packages\pluggy\hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
venv\lib\site-packages\pluggy\manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
venv\lib\site-packages\pluggy\manager.py:334: in traced_hookexec
    before(hook.name, hook_impls, kwargs)
venv\lib\site-packages\pluggy\manager.py:352: in before
    hooktrace(hook_name, kwargs)
venv\lib\site-packages\pluggy\_tracing.py:59: in __call__
    self.root._processmessage(self.tags, args)
venv\lib\site-packages\pluggy\_tracing.py:34: in _processmessage
    self._writer(self._format_message(tags, args))
C:\python38\lib\encodings\cp1252.py:19: in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
E   UnicodeEncodeError: 'charmap' codec can't encode characters in position 144-146: character maps to <undefined>
==== short test summary info ====
ERROR tests - UnicodeEncodeError: 'charmap' codec can't encode characters in position 144-146: character maps to <undefined>
!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!
===== 1 error in 7.20s =====
wrote pytestdebug information to C:\Users\user\Documents\test_folder\pytestdebug.log

[The-Compiler]

Yep, that - sorry for the missing explanation! 🙂

I think what's going on there is that Windows is using some encoding like latin1 by default for files, so pytest can't write the test name to it. No idea how to best fix it though, I didn't even know --debug existed 😆

[Tiarles]

Yes, I discovered this option a couple of weeks ago, to develop some hooks to my testing. And I also don't know how to solve this encode issue.

[asottile]

found another bug while fixing this (I'm fixing both)

$ LANG=C pytest t.py 
============================= test session starts ==============================
platform linux -- Python 3.6.12, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /tmp/x/pytest, configfile: pyproject.toml
plugins: hypothesis-5.35.4
collected 1 item                                                               

t.py EE                                                                  [100%]

==================================== ERRORS ====================================
\x1b[31m\x1b[1m______________________ ERROR at setup of test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 _______________________\x1b[0m

self = environ({'LANG': 'C', 'SHELL': '/bin/bash', 'SESSION_MANAGER': 'local/babibox:@/tmp/.ICE-unix/1669,unix/babibox:/tmp/....DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'OLDPWD': '/tmp/x', '_': '/tmp/x/pytest/venv36/bin/pytest'})
key = b'PYTEST_CURRENT_TEST', value = 't.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 (setup)'

    def __setitem__(self, key, value):
        key = self.encodekey(key)
>       value = self.encodevalue(value)

/usr/lib/python3.6/os.py:674: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = 't.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 (setup)'

    def encode(value):
        if not isinstance(value, str):
            raise TypeError("str expected, not %s" % type(value).__name__)
>       return value.encode(encoding, 'surrogateescape')
E       UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-15: ordinal not in range(128)

/usr/lib/python3.6/os.py:745: UnicodeEncodeError
\x1b[31m\x1b[1m_____________________ ERROR at teardown of test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 _____________________\x1b[0m

self = environ({'LANG': 'C', 'SHELL': '/bin/bash', 'SESSION_MANAGER': 'local/babibox:@/tmp/.ICE-unix/1669,unix/babibox:/tmp/....DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'OLDPWD': '/tmp/x', '_': '/tmp/x/pytest/venv36/bin/pytest'})
key = b'PYTEST_CURRENT_TEST', value = 't.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 (teardown)'

    def __setitem__(self, key, value):
        key = self.encodekey(key)
>       value = self.encodevalue(value)

/usr/lib/python3.6/os.py:674: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = 't.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 (teardown)'

    def encode(value):
        if not isinstance(value, str):
            raise TypeError("str expected, not %s" % type(value).__name__)
>       return value.encode(encoding, 'surrogateescape')
E       UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-15: ordinal not in range(128)

/usr/lib/python3.6/os.py:745: UnicodeEncodeError
=========================== short test summary info ============================
ERROR t.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 - UnicodeEncodeError: 'ascii' codec can't encode ...
ERROR t.py::test4_\u0447\u045b\u0448\u0452_\u010d\u0107\u0161\u0111 - UnicodeEncodeError: 'ascii' codec can't encode ...
============================== 2 errors in 0.21s ===============================

[DanielNoord]

@asottile I ran into this while debugging #9652. I saw the associated #7787 PR. Seeing the date of the last comment on that PR I thought I would post a little reminder about it. For example for pylint this problem basically disallows us from using the --debug option in case we want to troubleshoot any pytest issues (with you). (Obviously we can change the name of the file for the debugging, but I guess you understand what I mean 😄)