Set ZMQ receive timeouts in the interchange to prevent blocking threads #2625
Conversation
… HTEX.interchange Without the rcvtimeout the task_puller thread and _command_server block indefinitely and do not break when the kill_event is set. Since the recv_pyobj call is in c-land it has a tendency to ignore signals.
|
can you add some context/details for reproducing the problem? this is exiting fine with signals in other cases, I think, so I'd like to understand what is different in your test setup. |
|
I talked to Kevin who is working on funcX for more context about this. My understanding is this is to address an issue with the test suite hanging which appears in globus/globus-compute#1057 I have replicated what I think is the same hang on my laptop, and investigated it. The hang on my laptop comes from other parts of the funcX test suite installing a different handler for SIGTERM, and then the htex interchange inheriting that handler. This is bad behaviour by both the funcx test suite and the htex interchange. I opened an issue for the htex side of things - #2628 If this current PR #2625 does fix things for you, I'm interested in further investigating. If it turns out that it was because you also had a signal handler override hacked into your interchange, outside of this PR, then probably this PR can be closed without merge. |
|
changing this to draft because I think it can probably be closed but this can wait for yadu to look at later |
Description
Without the
RCVTIMEOthetask_pullerthread and_command_serveron the interchange block indefinitely and do not break when the kill_event is set. Since the recv_pyobj call is in c-land it has a tendency to ignore signals.This is rolling back changes from this PR: #2438
While we do not wait for the kill event, we still need the thread exit from the ZMQ context where it seems to ignore signals.
Currently, this is blocking/causing hangs on funcX tests wherever we use HTEX.
Type of change
Choose which options apply, and delete the ones which do not apply.