Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cancel transport request handling on disconnection #56327

Closed
DaveCTurner opened this issue May 7, 2020 · 2 comments · Fixed by #56620
Closed

Cancel transport request handling on disconnection #56327

DaveCTurner opened this issue May 7, 2020 · 2 comments · Fixed by #56620
Labels
:Distributed Coordination/Network Http and internode communication implementations :Distributed Coordination/Task Management Issues for anything around the Tasks API - both persistent and node level. >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@DaveCTurner
Copy link
Contributor

Since #43332 we cancel the handling of some REST requests when the REST channel closes, on the grounds that we cannot respond on a closed channel so there's no point in carrying on with its computation.

Relatedly, when a node N leaves the cluster all the other nodes will close their connections to N, which means that the results of any ongoing requests that N is handling will also ultimately be discarded. Should we therefore cancel the corresponding tasks and save a bunch of effort?

Relates #55566 in which a node leaves the cluster due to overload but then carries on being overloaded for longer than the user expects since it carries on doing the work that caused the overload until completion.
@DaveCTurner DaveCTurner added >enhancement :Distributed Coordination/Network Http and internode communication implementations :Distributed Coordination/Task Management Issues for anything around the Tasks API - both persistent and node level. labels May 7, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Task Management)

@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label May 7, 2020
@DaveCTurner DaveCTurner mentioned this issue May 7, 2020
Closed
@dnhatn
Copy link
Member

dnhatn commented May 9, 2020

+1. Today we already cancel child tasks when the node with the parent task leaves the cluster. However, it's at the node-level. We would react more quickly with the connection level. I am working on a task heartbeat component that detects orphaned tasks (i.e., the parent task was gone) and cancels them. I will try to integrate this suggestion into it.

@dnhatn dnhatn mentioned this issue May 12, 2020
Merged
dnhatn added a commit that referenced this issue May 13, 2020
@dnhatn
Cancel task and descendants on channel disconnects (#56620)
595ce8b 
If a channel gets disconnected, then we should cancel the tasks 
associated with that channel as their results won't be retrieved.

Closes #56327
Relates #56619
@dnhatn dnhatn mentioned this issue May 14, 2020
Merged
dnhatn added a commit that referenced this issue May 14, 2020
@dnhatn
Cancel task and descendants on channel disconnects (#56620)
566b23c 
If a channel gets disconnected, then we should cancel the tasks
associated with that channel as their results won't be retrieved.

Closes #56327
Relates #56619

Backport of #56620
@dnhatn dnhatn mentioned this issue Dec 3, 2020
Merged
dnhatn added a commit that referenced this issue Dec 8, 2020
@dnhatn
Cancel proxy requests when the proxy channel closes (#65850)
95936eb 
Since #43332 and #56327 we cancel rest requests when the rest channel 
closes and transport requests when the transport channel closes. This
commit cancels proxy requests and its descendant requests when the proxy
channel closes. This change is also required to support cross-clusters
task cancellation.

Relates #43332
Relates #56327
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Dec 12, 2020
@dnhatn
Cancel proxy requests when the proxy channel closes (elastic#65850)
2641e9a 
Since elastic#43332 and elastic#56327 we cancel rest requests when the rest channel
closes and transport requests when the transport channel closes. This
commit cancels proxy requests and its descendant requests when the proxy
channel closes. This change is also required to support cross-clusters
task cancellation.

Relates elastic#43332
Relates elastic#56327
@dnhatn dnhatn mentioned this issue Dec 12, 2020
Merged
dnhatn added a commit that referenced this issue Dec 13, 2020
@dnhatn
Cancel proxy requests when the proxy channel closes (#65850)
6653ed0 
Since #43332 and #56327 we cancel rest requests when the rest channel
closes and transport requests when the transport channel closes. This
commit cancels proxy requests and its descendant requests when the proxy
channel closes. This change is also required to support cross-clusters
task cancellation.

Relates #43332
Relates #56327
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Network Http and internode communication implementations :Distributed Coordination/Task Management Issues for anything around the Tasks API - both persistent and node level. >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Cancel task and descendants on channel disconnects dnhatn/elasticsearch
3 participants
@DaveCTurner @dnhatn @elasticmachine