Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DEADLINE_EXCEEDED code sometimes blocks all requests. #37

Open
willgraf opened this issue Oct 16, 2020 · 1 comment
Open

DEADLINE_EXCEEDED code sometimes blocks all requests. #37

willgraf opened this issue Oct 16, 2020 · 1 comment
Labels
bug Something isn't working

Comments

@willgraf
Copy link
Contributor

willgraf commented Oct 16, 2020 •
edited
Loading

While occasional DEADLINE_EXCEEDED codes are expected, it seems that sometimes TensorFlow Serving gets stuck in a deadline exceeded loop. I've seen some discussion indicating that the server does not give up on requests that have been canceled by the client, wasting work and getting bogged down in client-less requests. Currently the easiest way to resolve the issue is by deleting and refreshing the tensorflow serving instance.

As mentioned in vanvalenlab/kiosk-frontend#115, I think we have a few options to start resolving this issue:

  • Upgrade TensorFlow Serving and hope the problem is fixed.
  • Work on making models with cheaper inference.
  • Monitor TensorFlow and alert when this problem happens / automatically refresh the deployment.
@willgraf willgraf added the bug Something isn't working label Oct 16, 2020
@willgraf
Copy link
Contributor Author

I believe this is related to vanvalenlab/kiosk-console#338

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@willgraf