Error predicting on 512x512 image

[ngreenwald]

I tried uploading a 512x512 image to the current segmentation model on the website, and got the following time-out error:

Job Failed: Traceback (most recent call last): File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 194, in consume status = self._consume(redis_hash) File "/usr/src/app/redis_consumer/consumers/image_consumer.py", line 258, in _consume image = self.predict(image, model_name, model_version) File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 538, in predict model_dtype, untile=untile) File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 413, in _predict_big_image in_tensor_dtype=model_dtype) File "/usr/src/app/redis_consumer/consumers/base_consumer.py", line 282, in grpc_image prediction = client.predict(req_data, settings.GRPC_TIMEOUT) File "/usr/src/app/redis_consumer/grpc_clients.py", line 185, in predict response = self._retry_grpc(request, request_timeout) File "/usr/src/app/redis_consumer/grpc_clients.py", line 134, in _retry_grpc raise err File "/usr/src/app/redis_consumer/grpc_clients.py", line 123, in _retry_grpc response = api_call(request, timeout=request_timeout) File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.DEADLINE_EXCEEDED details = "Deadline Exceeded" debug_error_string = "{"created":"@1594928039.850059945","description":"Error received from peer ipv4:10.3.255.198:8500","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}" >

Here's the file: DNA.tiff.zip

[ngreenwald]

Looks like this was just a transient issue, will was able to run this without an issue. I'm going to leave this open for now as I run more test images to see if this comes up again, will close if it doesn't come back

[ngreenwald]

This popped up again. So far I've only run into this issue with the 'segmentation' model, and it hasn't happened with the mibi model.

[willgraf]

I believe this "Deadline Exceeded" error is a problem with TensorFlow serving, which seems to be documented. I'm not sure what the issue is exactly but we have a few paths:

• Upgrade TensorFlow Serving and hope the problem is fixed.
• Work on making models with cheaper inference.
• Monitor TensorFlow and alert when this problem happens / automatically refresh the deployment.

Regardless, the issue is not an issue with the kiosk-frontend, so I will close this issue.