🚀 Pre-release master -> staging_Mithril5

[matusdrobuliak66]

What kind of pre-release?

master branch

Sprint Name

Mithril

Pre-release version

5

Commit SHA

0b60aee

Did the commit CI suceeded?

• The commit CI succeeded.

Motivation

• Weekly release to stagging

What Changed

• ♻️ Frontend: Notifications cleanup by @GitHK in ♻️ Frontend: Notifications cleanup #4004
• ✨ Gateway-server: add ENV variable to change how many CPUs/RAM is advertised by each sidecar by @sanderegg in ✨ Gateway-server: add ENV variable to change how many CPUs/RAM is advertised by each sidecar #4019
• ✨Add simcore_user_agent and product_name to user service labels by @GitHK in ✨Add simcore_user_agent and product_name to user service labels #3990
• ♻️Dask gateway: add missing variables + change deprecated calls by @sanderegg in ♻️Dask gateway: add missing variables + change deprecated calls #4022
• 🎨Frontend: Update services from Dashboard UX by @odeimaiz in 🎨Frontend: Update services from Dashboard UX #4011
• 🐛 Fix: Download Study Data from Firefox by @odeimaiz in 🐛 Fix: Download Study Data from Firefox #4026
• 🐛 Sometime docker timestamps are not parsed correctly by @sanderegg in 🐛 Sometime docker timestamps are not parsed correctly #4003
• ✨🗃️ Implements services listing for nih-portal (1/2) (⚠️ devops) by @pcrespov in ✨🗃️ Implements services listing for nih-portal (1/2) (⚠️ devops) #4024
• 🐛Dask: Handle when gateway is offline by @sanderegg in 🐛Dask: Handle when gateway is offline #4033
• ✨ Is683/view link can open a project with a single service (2/2) by @pcrespov in ✨ Is683/view link can open a project with a single service (2/2) #4038
• 🎨 Frontend: Update services UX II by @odeimaiz in 🎨 Frontend: Update services UX II and change Boot mode in the UI #4031
• 🐛 filetypes in webserver API are case insensitive by @pcrespov in 🐛 filetypes in webserver API are case insensitive #4041
• 🔨 Maintenance/upgrade python version to 3.10 by @matusdrobuliak66 in 🔨 Maintenance/upgrade python version to 3.10 #4010
• 🐛 Frontend: minors 30.03 RM by @odeimaiz in 🐛 Frontend: minors 30.03 RM #4045
• 🐛🗃️ Dask gateway: clusters cannot be deleted after usage by @sanderegg in 🐛🗃️ Dask gateway: clusters cannot be deleted after usage #4036
• 🐛 fixes broken node progress by @GitHK in 🐛 fixes broken node progress [📌 v1.51.2]  #4049
• ♻️ e2e: improve error reporting by @sanderegg in ♻️ e2e: improve error reporting #4048

Devops check ⚠️ devops

• ✨🗃️ Implements services listing for nih-portal (1/2) (⚠️ devops) (✨🗃️ Implements services listing for nih-portal (1/2) (⚠️ devops) #4024)

e2e testing check 🧪

#4054 (comment)
No response

Summary 📝

• make release-staging name=Mithril version=5 git_sha=0b60aee54ae17ef4b31156e833b44f1c7c47aee9
• Draft pre-release
• Announce

  {"start": "2023-04-04T12:30:00.000Z", "end": "2023-04-04T13:00:00.000Z", "reason": "Release Mithril5"}

Releasing

• Release (release draft)
• Check Release CI
• Check deployed
  • aws deploy
  • dalco deploy
• Delete announcement
• Check e2e runs
• Announce

https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_Mithril5

[matusdrobuliak66]

• unhealthy webserver

    await asyncio.wait_for(
simcore_service_webserver.rest_healthcheck.HealthCheckFailed: Last requests average latency is 27.843608220418293 secs and surpasses 10.0 secs
ERROR: [2023-04-03 15:12:35,021/MainProcess] [servicelib.aiohttp.monitoring:middleware_handler(220)]  -  Unexpected server error "<class 'aiohttp.web_exceptions.HTTPServiceUnavailable'>" from access: 127.0.0.1 "GET /v0/health" done in 0.00 secs. Responding with status 503

• webserver api call timeout: GET clusters/0/details ( @sanderegg is there a fix already in master for this?)

ERROR: [2023-04-03 15:59:29,759/MainProcess] [servicelib.aiohttp.monitoring:middleware_handler(220)]  -  Unexpected server error "<class 'aiohttp.web_exceptions.HTTPServiceUnavailable'>" from access: 10.87.9.250 "GET /v0/clusters/0/details" done in 41.72 secs. Responding with status 503
The above exception was the direct cause of the following exception:
asyncio.exceptions.TimeoutError
  File "/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 69, in __anext__
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/director_v2_core_base.py", line 76, in request_director_v2
    async for attempt in AsyncRetrying(**DEFAULT_RETRY_POLICY):
    do = self.iter(retry_state=self._retry_state)
  File "/home/scu/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 360, in iter
    raise retry_exc.reraise()
  File "/home/scu/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 193, in reraise
    raise self.last_attempt.result()
    raise asyncio.TimeoutError from None
  File "/home/scu/.venv/lib/python3.10/site-packages/aiohttp/helpers.py", line 720, in __exit__
  File "/home/scu/.venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 894, in start
    with self._timer:
    await resp.start(conn)
  File "/home/scu/.venv/lib/python3.10/site-packages/aiohttp/client.py", line 560, in _request
    self._resp = await self._coro
  File "/home/scu/.venv/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    raise self._exception
  File "/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/director_v2_core_base.py", line 79, in request_director_v2
    async with session.request(
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result

• director v2

raise DirectorServiceError(
simcore_service_webserver.director_v2_exceptions.DirectorServiceError: Unexpected error: director-v2 returned 503, reason 'request to director-v2 timed-out: ' after calling URL('http://staging_director-v2:8000/v2/clusters/0/details?user_id=41176')

• I have restarted director_v2
• Sylvain did some additional cleaning of his tests in aws stagging (?)
• Afterwards the errors were not showing anymore

[mrnicegyu11]

Looks good, green from my side! thanks :)

[matusdrobuliak66]

AWS stagging problem with pending services improved a lot after the release:

[matusdrobuliak66]

E2E tests: The same problems as before release (these are all tests that went after the release):

[matusdrobuliak66]

Seem e2e tests are back to normal: