feat: capture shard failures in the head runtime

[girishc13]

Goals:

• resolves feat: gather search results from available/on-line shards #5336
• capture (AioRpcError and InternalNetworkError) errors as results when sending requests to the shards
  • raise error only if all shards failed or the uses_before and uses_after requests fail.
• [ ] add the total number of shards and the number of failed shards to the metadata. Client side communication needs to be implemented as part of feat: return the number of shards failed (if any) in the Header proto #5349.

[codecov]

Codecov Report

Merging #5338 (dfc97d7) into master (6eb4e87) will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5338      +/-   ##
==========================================
+ Coverage   86.69%   86.83%   +0.14%     
==========================================
  Files          99       99              
  Lines        6470     6494      +24     
==========================================
+ Hits         5609     5639      +30     
+ Misses        861      855       -6     

Flag	Coverage Δ
jina	86.83% <100.00%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...untimes/request_handlers/worker_request_handler.py	97.50% <ø> (ø)
jina/serve/networking.py	88.73% <100.00%> (+1.69%)	⬆️
...ina/serve/runtimes/gateway/graph/topology_graph.py	99.47% <100.00%> (+0.01%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[JoanFM]

Let's print some debug information or put some information on the Request when not all shards have responded? So the client can see it?

[girishc13]

Let's print some debug information or put some information on the Request when not all shards have responded? So the client can see it?

Is there a reason why the gRPC metadata from the Gateway (for example) not returned to the Client as headers or as metadata?

[JohannesMessner]

Let's print some debug information or put some information on the Request when not all shards have responded? So the client can see it?

Is there a reason why the gRPC metadata from the Gateway (for example) not returned to the Client as headers or as metadata?

If there is one I don't know it

[girishc13]

This failing test is another example of incompatibility issues with macOS. This doesn't fail in my machine.

python -c 'from jina import Executor,requests'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/jina/jina/__init__.py", line 207, in <module>
    from jina.orchestrate.flow.asyncio import AsyncFlow
  File "/jina/jina/orchestrate/flow/asyncio.py", line 2, in <module>
    from jina.orchestrate.flow.base import Flow
  File "/jina/jina/orchestrate/flow/base.py", line 69, in <module>
    from jina.orchestrate.deployments import Deployment
  File "/jina/jina/orchestrate/deployments/__init__.py", line 26, in <module>
    from jina.orchestrate.pods.factory import PodFactory
  File "/jina/jina/orchestrate/pods/__init__.py", line 16, in <module>
    from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
  File "/jina/jina/serve/runtimes/asyncio.py", line 15, in <module>
    from jina.serve.networking import GrpcConnectionPool
  File "/jina/jina/serve/networking.py", line 287, in <module>
    class GrpcConnectionPool:
  File "/jina/jina/serve/networking.py", line 975, in GrpcConnectionPool
    ) -> asyncio.Task[Union[Tuple, AioRpcError, InternalNetworkError]]:
TypeError: 'type' object is not subscriptable

[JoanFM]

This failing test is another example of incompatibility issues with macOS. This doesn't fail in my machine.

python -c 'from jina import Executor,requests'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/jina/jina/__init__.py", line 207, in <module>
    from jina.orchestrate.flow.asyncio import AsyncFlow
  File "/jina/jina/orchestrate/flow/asyncio.py", line 2, in <module>
    from jina.orchestrate.flow.base import Flow
  File "/jina/jina/orchestrate/flow/base.py", line 69, in <module>
    from jina.orchestrate.deployments import Deployment
  File "/jina/jina/orchestrate/deployments/__init__.py", line 26, in <module>
    from jina.orchestrate.pods.factory import PodFactory
  File "/jina/jina/orchestrate/pods/__init__.py", line 16, in <module>
    from jina.serve.runtimes.asyncio import AsyncNewLoopRuntime
  File "/jina/jina/serve/runtimes/asyncio.py", line 15, in <module>
    from jina.serve.networking import GrpcConnectionPool
  File "/jina/jina/serve/networking.py", line 287, in <module>
    class GrpcConnectionPool:
  File "/jina/jina/serve/networking.py", line 975, in GrpcConnectionPool
    ) -> asyncio.Task[Union[Tuple, AioRpcError, InternalNetworkError]]:
TypeError: 'type' object is not subscriptable

I think this may be a problem of Python version, u can also add this with quotes 'asyncio.Task[Union[Tuple, AioRpcError, InternalNetworkError]]' so that noone will complain

[JoanFM]

Please make sure @girishc13 that we have a ticket for the tasks that need to follow up

[girishc13]

Please make sure @girishc13 that we have a ticket for the tasks that need to follow up

Follow up ticket has already been created and added to the current PR description.