🚸 Improve support for Windows (#128)

* Resolve encoding crashes that occur on Windows

Windows doesn't consistently use UTF-8 as the default system encoding.

Enabling universal newlines causes Python to decode STDOUT
using the default system encoding, which may not be UTF-8 compatible.
In addition, when pip is executed via `subprocess`,
it may crash when attempting to write output.

To resolve these issues, Python's UTF-8 mode (available in 3.7+)
is enabled via an environment variable,
universal newlines are disabled,
and STDOUT is read and explicitly decoded as UTF-8.

* 👷 Add windows-smoketest job in CI

* 👷 Add verbosity to windows-smoketest

* 🐛 Disable unicode on Windows for --tree

* ✅ Add `pragma: no cover`

* ♻️ Use encoding check instead of platform check

Allows unicode `--tree` on utf-8 windows boxes:

"Changed in version 3.7: Return 'utf-8' if the Python UTF-8 Mode is enabled."

https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding

* 🚨 Run black

* 🐛 Disable unicode --tree for Windows again

* 🚨 Remove unused import

* 🚨 Run black

---------

Co-authored-by: ddelange <[email protected]>

.github/workflows/main.yml

@@ -68,8 +68,28 @@ jobs:
       run: |
         codecov
 
+  windows-smoketest:
+    runs-on: windows-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: 3.x
+
+    - name: Install
+      run: |
+        pip install -U pip setuptools wheel
+        pip install .
+
+    - name: Test
+      run: pipgrip -vvv --tree pipgrip
+
   CD:
-    needs: CI
+    needs: [CI, windows-smoketest]
     if: github.event_name == 'release'
     runs-on: ubuntu-latest
     steps:

src/pipgrip/cli.py

@@ -163,6 +163,14 @@ def build_tree(source, decision_packages):
 
 
 def render_tree(tree_root, max_depth, tree_ascii=False):
+    # Windows' cp1252 encoding does not supports anytree's unicode markers.
+    # Even in Github Actions where the windows runner has PYTHONUTF8 mode enabled,
+    # `if sys.getfilesystemencoding() == "cp1252":` check is insufficient and the
+    # error is still raised in CI.
+    # Hence disabling unicode trees altogether for Windows.
+    # ref https://github.com/pallets/click/issues/2121#issuecomment-1312773882
+    if click.utils.WIN:  # pragma: no cover
+        tree_ascii = True
     style = AsciiStyle() if tree_ascii else ContStyle()
     output = []
     for child in tree_root.children:

src/pipgrip/pipper.py

@@ -72,13 +72,20 @@ def stream_bash_command(args, echo=False):
     """Mimic subprocess.run, while processing the command output in real time."""
     # https://gist.github.com/ddelange/6517e3267fb74eeee804e3b1490b1c1d
     out = []
+    env = os.environ.copy()
+    # Enable Python's UTF-8 mode.
+    env["PYTHONUTF8"] = "1"
     process = subprocess.Popen(  # can use with-statement as of py 3.2
-        args, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
+        args,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        env=env,
     )
     for line in process.stdout:
-        out.append(line)
+        decoded_line = line.decode("utf-8", errors="replace")
+        out.append(decoded_line)
         if echo:
-            _echo(line[:-1])
+            _echo(decoded_line.rstrip())
     process.stdout.close()
     out = "".join(out)
     retcode = process.wait()