Catalog KeyError: Caching issues? - portal links fail

[mrnicegyu11]

Long story short

The catalog on AWS causes all p2e tests (and the actual portal links) to fail.
The frontend gets the following error:

This comes from the webserver, who prints the following error:
WARNING: simcore_service_webserver.catalog_client:get_services_for_user_in_product(115) - Error while retrieving services for user 106520. Returning an empty list

The catalog is at the same time flooded with errors of the following pattern:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/scu/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 23, in __call__
    await responder(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 42, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/home/scu/.venv/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/home/scu/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/home/scu/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_catalog/utils/requests_decorators.py", line 39, in decorator
    return await handler_task
  File "/home/scu/.venv/lib/python3.9/site-packages/aiocache/decorators.py", line 93, in wrapper
    return await self.decorator(f, *args, **kwargs)
  File "/home/scu/.venv/lib/python3.9/site-packages/aiocache/decorators.py", line 108, in decorator
    result = await f(*args, **kwargs)
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_catalog/api/routes/services.py", line 172, in list_services
    *[
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_catalog/api/routes/services.py", line 177, in <listcomp>
    services_in_db[key, version],
KeyError: ('simcore/services/comp/isolve', '2.1.0')

This error does not come periodically, but can be triggered by visiting a portal link specifically, for example using the puppeteer in osparc-simcore via node CC_Rabbit.js https://osparc.io/study/ 47d0b766-af53-11e9-a62b-02420aff7e72 300000 --demo. Check graylog for more details with this query: container_name:/.*catalog.*/ AND "KeyError: ('simcore/services/"

The affected services in the KeyError vary, I saw at least iSolve 2.1.0, iSolve 2.0.81 and ascent-runner 1.1.0.

If the catalog is restarted, puppeteer links work for a while (some minutes), and then the mentioned issues with associated error patterns surface again. This behavior of "briefly working after the restart" is reproducible.

This error also happens on osparc-master for portal-links, but portal links work subjectively felt "more often" on master than on osparc.io

A brief investigation with @pcrespov hint at this being a caching related issue.

Currently, portal links on osparc.io are broken because of this issue.

Expected behavior

Portal links work

Actual behaviour

Portal links dont work

Steps to reproduce

Observer catalog logs on osparc.io
cd osparc-simcore/tests/e2e/portal && node CC_Rabbit.js https://osparc.io/study/ 47d0b766-af53-11e9-a62b-02420aff7e72 300000 --demo

Your environment

osparc.io

[mrnicegyu11]

Potential enhancement idea: Set caching time via env-var, zero seconds caching time means caching is disabled