🏗️ Upgrades repository to python 3.8.10

[pcrespov]

What do these changes do?

• Upgrades all python code in repository, i.e. all packages, tests and services [except director whose development is frozen], to python 3.8.10. See what is new
• Repository's version in requirements/PYTHON_VERSION ( repo checked using tests/environment-setup/test_used_python.py)
• CI runs in 3.8.10 except for python-linting that is run in both 3.8 and 3.9

Main changes

• Dockerfile base images
• python_requires specs on setup.py and other metadata/checks
• requirement listings were re-compile using latest pip-tools
  • compilation run within controlled environment requirements/tools/Dockerfile
  • NO package was upgraded
  • Notice how all backport libraries (e.g. dataclasses) were automatically removed (SEE requirements/packages-notes.md for more details)
• config files in tooling (e.g. .pre-commit-config.yaml) etc
• Upgrade python in CI setup
• web-server's garbage_collector.py ignores cancellation and enters in an errors loop. I am still not sure what is the cause so I introduced a mechanism to enforce getting out of the infinite while True loop.

Some highlights in 3.8

• assignment expressions @mguidon, I know you were waiting for this!
• new features in asyncio @sanderegg @GitHK interesting:
  • asyncio.run(coro())
  • python -m asyncio creates an asyncio REPL 3.8.0
  • The exception asyncio.CancelledError now inherits from BaseException rather than Exception
  • tasks can now be named (see garbage-collector background task). Handy to avoid duplicates or to search ...
• time benchmark from 3.5 to 3.8

Related issue/s

• update osparc-simcore to python 3.7 or even 3.8 #1412 updates to py 3.8
• sync .python-version in all environs (ci, docker, devenv, ...) #1388

How to test

• RECOMMENDED: install pyenv in your system
• Check available versions: pyenv install --list | grep 3.8
• install 3.8: pyenv install 3.8.6
• see all installed versions: pyenv versions
• use 3.8 in osparc-simcore project. Will create a .python-version file:

cd osparc-simcore
pyenv local 3.8.10
python --version

• create devenv

 make devenv
 source .venv/bin/activate

Checklist

• when this is upgraded review New service-integration library in osparc-simcore #1884
• check speed difference between 3.8 and 3.6? and major changes
• CI matrix
• Upgrades python service's Dockerfile base
• Adapt code for new features
  • CancelledError is not in asyncio.future anymore
  • New AsyncMock. No need to use the trick of returning a Future to emulate a coroutine.
  • pylint (has some issues with 3.9)

[codecov]

Codecov Report

Merging #2079 (ab6c99a) into master (e19f6c1) will decrease coverage by 0.6%.
The diff coverage is 63.6%.

@@           Coverage Diff            @@
##           master   #2079     +/-   ##
========================================
- Coverage    74.8%   74.2%   -0.7%     
========================================
  Files         517     518      +1     
  Lines       20161   20320    +159     
  Branches     2000    2012     +12     
========================================
- Hits        15083   15080      -3     
- Misses       4544    4716    +172     
+ Partials      534     524     -10     

Flag	Coverage Δ
integrationtests	67.3% <55.0%> (-0.1%)	⬇️
unittests	67.7% <62.1%> (-0.5%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...s/service-library/src/servicelib/rest_responses.py	80.3% <ø> (ø)
...vice-library/src/servicelib/utils_debug_asyncio.py	0.0% <0.0%> (ø)
...r/src/simcore_service_sidecar/celery_task_utils.py	52.9% <0.0%> (ø)
.../sidecar/src/simcore_service_sidecar/log_parser.py	86.3% <0.0%> (+0.7%)	⬆️
.../simcore_service_webserver/diagnostics_handlers.py	54.5% <0.0%> (ø)
.../simcore_service_webserver/projects/projects_db.py	90.3% <ø> (+0.1%)	⬆️
.../web/server/src/simcore_service_webserver/utils.py	45.5% <0.0%> (+0.5%)	⬆️
...simcore_service_webserver/projects/projects_api.py	83.5% <30.0%> (-2.5%)	⬇️
...ore_service_api_server/utils/solver_job_outputs.py	51.7% <50.0%> (ø)
...ervices/storage/src/simcore_service_storage/dsm.py	71.9% <50.0%> (+0.1%)	⬆️
... and 121 more

[GitHK]

Very nice job. Thanks a lot for upgrading to 3.8! I have a few questions below.

[sanderegg]

The only thing maybe missing is:

• installation of pyenv as part of some makefile recipe (or maybe just a link to pyenv?)

[GitHK]

I've requested a few changes, they will avoid some pain when testing.

[sanderegg]

sweet!