UnicodeEncodeError running autopep8 on Windows using pyevn

[jpdom]

While running the msautopep8 extension in vscode I am getting this error whenever a file change happens (the autopep8 server starts with no problems the error only shows when actually formatting).

UnicodeEncodeError: 'charmap' codec can't encode character '\udc9d' in position 27664: character maps to <undefined>

Full log:

2023-05-04 12:48:13.956 [info] Server: Stop requested
2023-05-04 12:48:13.956 [info] [Trace - 12:48:13 PM] Sending request 'shutdown - (2)'.
2023-05-04 12:48:13.959 [info] [Trace - 12:48:13 PM] Received response 'shutdown - (2)' in 2ms.
2023-05-04 12:48:13.959 [info] [Trace - 12:48:13 PM] Sending notification 'exit'.
2023-05-04 12:48:13.963 [info] Server run command: c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python.exe c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\tool\lsp_server.py
2023-05-04 12:48:13.963 [info] Server: Start requested.
2023-05-04 12:48:14.393 [info] CWD Server: c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2
2023-05-04 12:48:14.394 [info] sys.path used to run Server:
c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs
c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\tool
c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python37.zip
c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\DLLs
c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib
c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9
c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib\site-packages
2023-05-04 12:48:14.394 [info] PYTHONPATH env variable used to run Server:
c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs;
2023-05-04 12:48:14.394 [info] Settings used to run Server:
[
{
"cwd": "c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2",
"workspace": "file:///c%3A/Users/joaop/OneDriveGW/OneDrive%20-%20WiseCompany/Documentos/Git/api_platform2",
"args": [
"--max-line-length=250"
],
"path": [],
"interpreter": [
"c:\Users\joaop\.pyenv\pyenv-win\versions\3.7.9\python.exe"
],
"importStrategy": "useBundled",
"showNotifications": "off"
}
]

2023-05-04 12:48:14.394 [info] Global settings:
{
"cwd": "C:\Users\joaop\AppData\Local\Programs\Microsoft VS Code",
"workspace": "C:\Users\joaop\AppData\Local\Programs\Microsoft VS Code",
"args": [],
"path": [],
"interpreter": [],
"importStrategy": "useBundled",
"showNotifications": "off"
}

2023-05-04 12:48:14.410 [info] c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python.exe -m autopep8 --version
2023-05-04 12:48:14.410 [info] CWD Server: c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2
2023-05-04 12:48:14.490 [info] Version info for formatter running for c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2:
autopep8 2.0.2 (pycodestyle: 2.10.0)

2023-05-04 12:48:14.491 [info] SUPPORTED autopep8>=1.7.0
FOUND autopep8==2.0.2

2023-05-04 12:48:14.491 [info] c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python.exe -m autopep8 --version
2023-05-04 12:48:14.491 [info] CWD Server: c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2
2023-05-04 12:48:14.560 [info] Version info for formatter running for c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2:
autopep8 2.0.2 (pycodestyle: 2.10.0)

2023-05-04 12:48:14.560 [info] SUPPORTED autopep8>=1.7.0
FOUND autopep8==2.0.2

2023-05-04 12:48:18.827 [info] [Trace - 12:48:18 PM] Sending notification 'textDocument/didChange'.
2023-05-04 12:48:34.872 [info] [Trace - 12:48:34 PM] Sending notification 'textDocument/didChange'.
2023-05-04 12:48:49.034 [info] [Trace - 12:48:49 PM] Sending request 'textDocument/formatting - (1)'.
2023-05-04 12:48:49.049 [info] [Trace - 12:48:49 PM] Received notification 'window/logMessage'.
2023-05-04 12:48:49.049 [info] c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python.exe -m autopep8 --max-line-length=250 -
2023-05-04 12:48:49.049 [info] [Trace - 12:48:49 PM] Received notification 'window/logMessage'.
2023-05-04 12:48:49.049 [info] CWD Server: c:\Users\joaop\OneDriveGW\OneDrive - WiseCompany\Documentos\Git\api_platform2
2023-05-04 12:48:50.158 [info] [Trace - 12:48:50 PM] Received notification 'window/logMessage'.
2023-05-04 12:48:50.158 [info] Traceback (most recent call last):
File "c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs\autopep8.py", line 4570, in
sys.exit(main())
File "c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs\autopep8.py", line 4516, in main
wrap_output(sys.stdout, encoding=encoding).write(fixed_stdin)
File "c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib\codecs.py", line 377, in write
data, consumed = self.encode(object, self.errors)
File "c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\lib\encodings\cp1252.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\udc9d' in position 27652: character maps to

2023-05-04 12:48:50.158 [info] [Trace - 12:48:50 PM] Received response 'textDocument/formatting - (1)' in 1124ms.

[karthiknadig]

@john1997x Can you run the command from the terminal? This looks like an issue with autopep8 itself or the encoding used to save the file (the file should be saved as UTF-8).

> set PYTHONPATH=c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs
> type <python-file-to-format> | c:\Users\joaop.pyenv\pyenv-win\versions\3.7.9\python.exe -m autopep8 --max-line-length=250 -

Make sure to include the - at the end in the second command.

[Howard-WeiKai]

similar problem when autopep8 formatting

log:

2023-05-25 11:16:33.900 [info] [Trace - 上午11:16:33] Received response 'textDocument/formatting - (2)' in 354ms.
2023-05-25 11:16:37.022 [info] [Trace - 上午11:16:37] Sending request 'textDocument/formatting - (3)'.
2023-05-25 11:16:37.027 [info] [Trace - 上午11:16:37] Received notification 'window/logMessage'.
2023-05-25 11:16:37.027 [info] c:\Users\jhele\AppData\Local\Programs\Python\Python310\python.exe -m autopep8 -
2023-05-25 11:16:37.027 [info] [Trace - 上午11:16:37] Received notification 'window/logMessage'.
2023-05-25 11:16:37.027 [info] CWD Server: c:\Users\jhele\Desktop\p2p_orm
2023-05-25 11:16:37.290 [info] [Trace - 上午11:16:37] Received notification 'window/logMessage'.
2023-05-25 11:16:37.290 [info] Traceback (most recent call last):
File "c:\Users\jhele\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\Users\jhele\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\Users\jhele.vscode\extensions\ms-python.autopep8-2023.4.0\bundled\libs\autopep8.py", line 4570, in
sys.exit(main())
File "c:\Users\jhele.vscode\extensions\ms-python.autopep8-2023.4.0\bundled\libs\autopep8.py", line 4516, in main
wrap_output(sys.stdout, encoding=encoding).write(fixed_stdin)
UnicodeEncodeError: 'cp950' codec can't encode character '\udc93' in position 1638: illegal multibyte sequence

[Usercyk]

Me too
I can use autopep8 without error in the terminal.

log:

2023-05-25 21:08:10.614 [info] [Trace - 21:08:10] Sending request 'textDocument/formatting - (6)'.
2023-05-25 21:08:10.620 [info] [Trace - 21:08:10] Received notification 'window/logMessage'.
2023-05-25 21:08:10.621 [info] d:\vscode\Pyside6\qt_env\Scripts\python.exe -m autopep8 -
2023-05-25 21:08:10.621 [info] [Trace - 21:08:10] Received notification 'window/logMessage'.
2023-05-25 21:08:10.622 [info] CWD Server: d:\vscode\Pyside6
2023-05-25 21:08:11.022 [info] [Trace - 21:08:11] Received notification 'window/logMessage'.
2023-05-25 21:08:11.022 [info] Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "d:\vscode\Pyside6\qt_env\Lib\site-packages\autopep8.py", line 4570, in
sys.exit(main())
^^^^^^
File "d:\vscode\Pyside6\qt_env\Lib\site-packages\autopep8.py", line 4516, in main
wrap_output(sys.stdout, encoding=encoding).write(fixed_stdin)
UnicodeEncodeError: 'gbk' codec can't encode character '\udcad' in position 303: illegal multibyte sequence
2023-05-25 21:08:11.022 [info] [Trace - 21:08:11] Received response 'textDocument/formatting - (6)' in 409ms.

[Usercyk]

If I remove all the code with Chinese, I won't get this error.

[karthiknadig]

@Usercyk Can you share your environment variables?

[karthiknadig]

@Usercyk Can you also confirm if the encoding for your python file is utf-8? You should be able to see the encoding on your status bar.

[TeoDar]

set PYTHONPATH=c:\Users\joaop.vscode\extensions\ms-python.autopep8-2023.2.0\bundled\libs

Has the same error. File encoding - UTF-8. Python 3.11.4 autopep8 upgraded.

[karthiknadig]

The issue is caused by a bug in autopep8 itself. hhatto/autopep8#685
Identified this issue here #104

[lanjia1028]

give a way to avoid the case:

change VS setting "files.encoding": "gbk" as same as the OS default encode .

or change OS default encode to utf-8.

the reason is
C:\Users\admin.vscode\extensions\ms-python.autopep8-2023.4.0\bundled\libs\autopep8.py
line 4511 : read_stdin = sys.stdin.read()
using OS default encode to read your code steam , it will be decoding wrong.