Use a longer driver timeout for drain #454
Conversation
| datastax-java-driver { | ||
| profiles { | ||
| slow { | ||
| basic.request.timeout = 180 seconds |
There was a problem hiding this comment.
Do we need to have this set as hardcoded value? At the moment, the way we call /drain might benefit from even no timeout at all, since the pod has a strict timeline until it gets killed in any case.
There was a problem hiding this comment.
I'd have to see if it can be set with a variable or something at runtime. The idea is to make this large enough so that none of the driver "calls" times out before getting a response. I don''t think there is a way to set no timeout for driver requests, assuming you mean wait forever. If Kubernetes things end up calling this and they have their own timeout, then as long is this is at least as long as K8s will wait, we should be fine. I'm still not sure the full context of when /drain is called and waiting longer than 3 minutes is expected.
We can raise this timeout to 5 or 10 minutes if you think that would be better. It won't hurt any drain calls that finish in less time. Also, the drain call is the only one using this new "slow" query thing for now. Although #292 might also benefit from using the slow query for taking a snapshot if that function is also taking a long time.
There was a problem hiding this comment.
Actually, it seems that we can configure the timeout to 0, which would effectively make it wait forever. Do we want to do this?
There was a problem hiding this comment.
Users are able to configure the /drain timeout using gracePeriod for pods.
There was a problem hiding this comment.
This change is only going to affect the internal driver query for the drain operation. Outside of a kubernetes environment, with this set to 0, a call to /drain won't get a driver query request timeout. Something else may happen that causes the operation to fail, but otherwise, the driver will wait forever for the agent to complete and return.
Fixes #453