Make the Translator to GoBGP Connection More Resilient

[samoehlert]

I received this error out of the blue today, where it seems like the connection between translator and gobgp died. Looking at the code it seems we have no means of trying to reconnect at all, the fix is to just restart the container(s). I think we want to look at a way of auto reconnecting if it notices a failure. The thing to keep in mind also is that docker still thinks both containers are happy. I think we could do either a docker healthcheck on these containers and or add some code to reconnect to the grpc client.

translator-1  | Traceback (most recent call last):
translator-1  |   File "/app/translator.py", line 118, in <module>
translator-1  |     loop.run_until_complete(main())
translator-1  |   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
translator-1  |     return future.result()
translator-1  |   File "/app/translator.py", line 109, in main
translator-1  |     await process(message, websocket, g)
translator-1  |   File "/app/translator.py", line 93, in process
translator-1  |     g.add_path(ip, event_data)
translator-1  |   File "/app/gobgp.py", line 135, in add_path
translator-1  |     self.stub.AddPath(
translator-1  |   File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1181, in __call__
translator-1  |     return _end_unary_response_blocking(state, call, False, None)
translator-1  |   File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
translator-1  |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
translator-1  | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
translator-1  | 	status = StatusCode.UNAVAILABLE
translator-1  | 	details = "Socket closed"
translator-1  | 	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-01-09T15:22:02.606614826+00:00", grpc_status:14, grpc_message:"Socket closed"}"
translator-1  | >