Analyzer Python compatibility check might fail, resulting in Python2 fallback

[nicorikken]

If there is an error with parsing the Python file into a Python3 ast, the python_compatibility.py script fails:

ort/analyzer/src/main/resources/scripts/python_compatibility.py

Line 46 in ee69cd7

file_content = open(file_path, encoding="UTF-8").read()

I wasn't able to debug this on first glance. Perhaps I should have enabled debug level logging to get a proper warning. I'm now trying to debug this issue some more to come up with ideas for usability improvements.

Thanks to @edulix for pinpointing the issue.

Possible improvements

• Adjust log levels to make it clearer what is causing the false behavior
• Just raise the error with print(e)
• Suggest running the python_compatibility.py script locally to find the troublesome files

[edulix]

Another important issue is that it is not clear which package manager is being run and with which python version. Without any previous knowledge of the code and as a noob user, I had to write some debugging code in the Pip.kt code to show the python version being used and found this python_compatibility.py script and more or less how it worked.

04:11:06.620 [DefaultDispatcher-worker-3] WARN  org.ossreviewtoolkit.analyzer.managers.PythonVersion - EDU: using python version '3'
04:11:06.628 [DefaultDispatcher-worker-3] WARN  org.ossreviewtoolkit.analyzer.managers.PythonVersion - EDU: using 3 pythonVersionCommand = 'Python 3.6.9`

Possible improvements / suggestions

• Always show a clear indication of which is the package manager detected (and perhaps even why, like package.json found => using NPM package manager).
• In the case of the Pip package manager, show the output of python --version to tell the user what version of python is being detected.

[nicorikken]

@edulix I like your suggested generic debug information.

This specific issue was solved for me with newer version of Python through #3902 Still I'm now sure why a Python file would not be read correctly as utf-8, which makes it difficult to debug or solve. I haven't been able to pinpoint the issue.

[sschuberth]

The python_compatibility.py script is not in use anymore as of c885541, so I believe this can be closed.