✨ postgres readiness check includes migration completed

[pcrespov]

What do these changes do?

We noticed in the CI that under certain conditions, a service fails to start because while connecting to the database the migration (performed by the migration service) is still not complete (Notice that the swarm does not guarantee the order of deploy of services).

We could ensure that migration service always start first but this does not guarantee that the migration is completed. Therefore, this PR creates a check function to ensure the caller service that the db migration has successfully completed.

Hightlights of this PR:

• simcore_postgres_database
  • ✨ defines utils_migration.get_current_head to get head revision of current db (adds alembic dependency to base.txt)
  • ♻️ creates utils_cli.py and moves there all utilities used in cli.py
  • ♻️ creates utils_aiopg.py
  • ♻️ creates retry_policies.py for common tenacity-based retry policies
  • ♻️ cleanup errors.py to structure pg errors
• servicelib: steps towards fully depreciating aiopg utils
  • ♻️ moving most of the aiopg utils to simcore_postgres_database package
• ✨ All pg startup events now include a check to ensure that the pg migration is complete. Affects:
  • api-server
  • catalog
  • director-v2
  • storage
  • webserver
  • simcore-sdk
  • sidecar

Related issue/s

services start communicating with pg before or while migration is taking place, leading to startup failures

How to test

• all db tests involved

[codecov]

Codecov Report

Merging #2600 (addf0f3) into master (b014be7) will decrease coverage by 0.0%.
The diff coverage is 45.2%.

@@           Coverage Diff            @@
##           master   #2600     +/-   ##
========================================
- Coverage    77.1%   77.1%   -0.1%     
========================================
  Files         621     625      +4     
  Lines       24043   24094     +51     
  Branches     2367    2362      -5     
========================================
+ Hits        18561   18581     +20     
- Misses       4850    4880     +30     
- Partials      632     633      +1     

Flag	Coverage Δ
integrationtests	65.9% <78.6%> (+0.1%)	⬆️
unittests	71.6% <39.5%> (-0.2%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...gres-database/src/simcore_postgres_database/cli.py	0.0% <0.0%> (ø)
...abase/src/simcore_postgres_database/utils_aiopg.py	0.0% <0.0%> (ø)
...atabase/src/simcore_postgres_database/utils_cli.py	0.0% <0.0%> (ø)
...s/service-library/src/servicelib/retry_policies.py	0.0% <0.0%> (ø)
...s/catalog/src/simcore_service_catalog/db/events.py	39.2% <28.5%> (-5.6%)	⬇️
...server/src/simcore_service_api_server/db/events.py	86.2% <73.3%> (-13.8%)	⬇️
...c/simcore_service_director_v2/modules/db/events.py	87.5% <75.0%> (-12.5%)	⬇️
...sdk/src/simcore_sdk/node_ports_common/dbmanager.py	85.7% <78.5%> (-0.3%)	⬇️
services/sidecar/src/simcore_service_sidecar/db.py	88.2% <80.9%> (-3.8%)	⬇️
...ces/web/server/src/simcore_service_webserver/db.py	79.6% <80.9%> (+18.8%)	⬆️
... and 9 more

[sanderegg]

looking promising!

[GitHK]

Looks way better now. Thanks!

[sanderegg]

great, let's hope this brings stability