Interruptible execution #433
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achirkin
added
feature request
jrhemstad
reviewed
Jan 4, 2022
cjnolet
requested changes
Jan 10, 2022
|
rerun tests |
cjnolet
previously approved these changes
Feb 3, 2022
cjnolet
approved these changes
Feb 3, 2022
|
@achirkin, this is ready to merge but RAFT's CPU builds were just enabled and aren't yet running successfully (hence the CI failure here). This will be merged very soon. |
cjnolet
approved these changes
Feb 3, 2022
|
rerun tests |
|
rerun tests |
tfeher
approved these changes
Feb 7, 2022
| * in code from outside of the thread. In particular, it provides an interruptible version of the | ||
| * blocking CUDA synchronization function, that allows dropping a long-running GPU work. | ||
| * | ||
| * |
There was a problem hiding this comment.
Consider copying parts of the PR description here:
Interruptible execution is facilitated using the following three functions:
static void synchronize(rmm::cuda_stream_view stream);
static void yield();
static void cancel(std::thread::id thread_id);
synchronize and yield serve as cancellation points for the executing CPU thread. cancel allows to throw an async exception in a target CPU thread, which is observed in the nearest cancellation point. Altogether, these allow to cancel a long-running job without killing the OS process.
The key to make this work is an obvious observation that the CPU spends most of the time waiting on cudaStreamSynchronize. By replacing that with interruptible::synchronize, we introduce cancellation points in all critical places in code. If that is not enough in some edge cases (the cancellation points are too far apart), a developer can use yield to ensure that a cancellation request is received sooner rather than later.
|
rerun tests |
|
@gpucibot merge |
rapids-bot bot
pushed a commit
to rapidsai/cuml
that referenced
this pull request
Feb 8, 2022
### Cooperative-style interruptible C++ threads.
This proposal attempts to make cuml experience more responsive by allowing easier way to interrupt/cancel long running cuml tasks. It replaces calls `cudaStreamSynchronize` with `raft::interruptible::synchronize`, which serve as a cancellation points in the algorithms. With a small extra hook on the python side, Ctrl+C requests now can interrupt the execution (almost) immediately. At this moment, I adapted just a few models as a proof-of-concept.
Example:
```python
import sklearn.datasets
import cuml.svm
X, y = sklearn.datasets.fetch_olivetti_faces(return_X_y=True)
model = cuml.svm.SVC()
print("Data loaded; fitting... (try Ctrl+C now)")
try:
model.fit(X, y)
print("Done! Score:", model.score(X, y))
except Exception as e:
print("Canceled!")
print(e)
```
#### Implementation details
rapidsai/raft#433
#### Adoption costs
From the changeset in this PR you can see that I introduce two types of changes:
1. Change `cudaStreamSynchronize` to either `handle.sync_thread` or `raft::interruptible::synchronize`
2. Wrap the cython calls with [`cuda_interruptible`](https://github.com/rapidsai/raft/blob/36e8de5f73e9ec7e604b38a4290ac82bc35be4b7/python/raft/common/interruptible.pyx#L28) and `nogil`
Change (1) is straightforward and can mostly be automated.
Change (2) is a bit more involved. You definitely have to wrap a C++ call with `interruptibleCpp` to make `Ctrl+C` work, but that is also rather simple. The tricky part is adding `nogil`, because you have to make sure there is no python objects within `with nogil` block. However, `nogil` does not seem to be strictly required for the signal handler to successfully interrupt the C++ thread. It worked in my tests without `nogil` as well. Yet, I chose to add `nogil` in the code where possible, because in theory it should reduce the interrupt latency and enable more multithreading.
#### Motivation
In general, this proposal makes executing threads (and thus algos/models) more controllable. The main use cases I see:
1. Being able to Ctrl+C the running model using signal handlers.
2. Stopping the thread programmatically, e.g. we can create the tests of sort "if running for more than n seconds, stop and fail".
Resolves #4384
Authors:
- Artem M. Chirkin (https://github.com/achirkin)
Approvers:
- Corey J. Nolet (https://github.com/cjnolet)
URL: #4463
vimarsh6739
pushed a commit
to vimarsh6739/cuml
that referenced
this pull request
Oct 9, 2023
### Cooperative-style interruptible C++ threads.
This proposal attempts to make cuml experience more responsive by allowing easier way to interrupt/cancel long running cuml tasks. It replaces calls `cudaStreamSynchronize` with `raft::interruptible::synchronize`, which serve as a cancellation points in the algorithms. With a small extra hook on the python side, Ctrl+C requests now can interrupt the execution (almost) immediately. At this moment, I adapted just a few models as a proof-of-concept.
Example:
```python
import sklearn.datasets
import cuml.svm
X, y = sklearn.datasets.fetch_olivetti_faces(return_X_y=True)
model = cuml.svm.SVC()
print("Data loaded; fitting... (try Ctrl+C now)")
try:
model.fit(X, y)
print("Done! Score:", model.score(X, y))
except Exception as e:
print("Canceled!")
print(e)
```
#### Implementation details
rapidsai/raft#433
#### Adoption costs
From the changeset in this PR you can see that I introduce two types of changes:
1. Change `cudaStreamSynchronize` to either `handle.sync_thread` or `raft::interruptible::synchronize`
2. Wrap the cython calls with [`cuda_interruptible`](https://github.com/rapidsai/raft/blob/36e8de5f73e9ec7e604b38a4290ac82bc35be4b7/python/raft/common/interruptible.pyx#L28) and `nogil`
Change (1) is straightforward and can mostly be automated.
Change (2) is a bit more involved. You definitely have to wrap a C++ call with `interruptibleCpp` to make `Ctrl+C` work, but that is also rather simple. The tricky part is adding `nogil`, because you have to make sure there is no python objects within `with nogil` block. However, `nogil` does not seem to be strictly required for the signal handler to successfully interrupt the C++ thread. It worked in my tests without `nogil` as well. Yet, I chose to add `nogil` in the code where possible, because in theory it should reduce the interrupt latency and enable more multithreading.
#### Motivation
In general, this proposal makes executing threads (and thus algos/models) more controllable. The main use cases I see:
1. Being able to Ctrl+C the running model using signal handlers.
2. Stopping the thread programmatically, e.g. we can create the tests of sort "if running for more than n seconds, stop and fail".
Resolves rapidsai#4384
Authors:
- Artem M. Chirkin (https://github.com/achirkin)
Approvers:
- Corey J. Nolet (https://github.com/cjnolet)
URL: rapidsai#4463
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cooperative-style interruptible C++ threads.
This proposal introduces
raft::interruptibleintroducing three functions:synchronizeandyieldserve as cancellation points for the executing CPU thread.cancelallows to throw an async exception in a target CPU thread, which is observed in the nearest cancellation point. Altogether, these allow to cancel a long-running job without killing the OS process.The key to make this work is an obvious observation that the CPU spends most of the time waiting on
cudaStreamSynchronize. By replacing that withinterruptible::synchronize, we introduce cancellation points in all critical places in code. If that is not enough in some edge cases (the cancellation points are too far apart), a developer can useyieldto ensure that a cancellation request is received sooner rather than later.Implementation
C++
raft::interruptiblekeeps anstd::atomic_flagin the thread-local storage in each thread, which tells whether the thread can continue executing (being in non-cancelled state).cancelclears this flag, andyieldchecks it and resets to the signalled state (throwing araft::interrupted_exceptionexception if necessary).synchronizeimplements a spinning lock querying the state of the stream andyielding on each iteration. I also add an overloadsync_streamto the raft handle type, to make it easier to modify the behavior of all synchronization calls in raft and cuml.python
This proposal adds a context manager
cuda_interruptibleto handle Ctrl+C requests during C++ calls (using posix signals).cuda_interruptiblesimply callsraft::interruptible::cancelon the target C++ thread.Motivation
See rapidsai/cuml#4463
Resolves rapidsai/cuml#4384