Graceful shutdown when a CTRL-C (or similar activity) is entered on the controller

[k-rister]

No description provided.

[k-rister]

I've been thinking about this and I think we need to resolve a few things to make this work the way we want. I believe the way to achieve what we want is for the controller to catch the CTRL-C signal (or similar activity) and then send an abort via roadblock so that all the participants in the run know to cleanup and shutdown. There are 2 immediate problems with this that immediately come to mind:

1. On the controller we need to ensure that rickshaw is properly handling the CTRL-C signal. If roadblock is currently running then this probably means that rickshaw-run needs to ignore the signal and roadblock itself needs to catch the signal and send the abort. If roadblock is not currently running then rickshaw-run needs to catch the signal and instruct the next roadblock to send an abort.
2. On the participant side (endpoints and engines) we are going to need to move as much of the execution of code into scripts that can be called using roadblock's wait-for functionality. This is because the distribution of the abort signal via roadblock is not going to actually cancel the run in a timely manner unless roadblock is running and in control of the execution flow. In many/most situations today roadblock is only executed when something has completed (such as a benchmark run) and then the next synchronization point is reached. In this mode it could be a very long time (ie. up to the entire length of a benchmark run) for the abort to be properly received by all participants.

I feel like the whole point of this feature is to be able to properly cleanup a canceled run as quickly as possible...if it's not being done quickly then what is the point of canceling?

Unfortunately, moving various code execution blocks to the roadblock wait-for functionality is probably not going to be a trivial exercise. I imagine that in some situations it will be pretty easy and in others it will be quite complicated. Right now there is a lot of global environment variables being used in some places that is not going to properly traverse the <program>-><roadblock>-><wait-for execution> exec+fork flow so we will need to clean that up and use config files, parameters, etc. to get that information to where it needs to be at runtime.

Another issue, albeit more minor, is that roadblock itself needs to have the way it handles a CTRL-C changed. Today it catches the signal and shutdowns down the program. It needs to instead catch the signal and then change the program flow from whatever it currently is to that of an abort path. This is simple in concept but it may cause some issues in the state machine that is current implemented. I don't think this is a huge problem, but it may take some time to figure out the modifications required in the state machine.

@atheurer thoughts?

[k-rister]

perftool-incubator/roadblock#50