Add ability to cancel AsyncIO gRPC stream requests #417
Conversation
kthui
changed the title
| @@ -636,7 +636,7 @@ async def stream_infer( | |||
|
|
|||
| Parameters | |||
| ---------- | |||
| inputs_iterator : async_generator | |||
| inputs_iterator : asynchronous iterator | |||
There was a problem hiding this comment.
This is always "asynchronous iterator" not "asynchronous generator function", because inputs_iterator is used directly without inputs_iterator().
| @@ -624,7 +624,7 @@ async def infer( | |||
| except grpc.RpcError as rpc_error: | |||
| raise_error_grpc(rpc_error) | |||
|
|
|||
| async def stream_infer( | |||
There was a problem hiding this comment.
This used to be an "asynchronous generator function" which the user calls like:
async for response in stream_infer(...):
...Now, it is a normal function that returns an "asynchronous iterator", which the user can call it in the exact same way above.
README.md
Outdated
| ``` | ||
|
|
||
| See more details about these APIs in | ||
| [grpc/\aio/\__init__.py](src/python/library/tritonclient/grpc/aio/__init__.py). |
There was a problem hiding this comment.
this rendered weirdly, might not need backslashes, please double check it
README.md
Outdated
|
|
||
| ```python | ||
| infer_task = asyncio.create_task(aio_client.infer()) | ||
| await something_else |
There was a problem hiding this comment.
is this await something_else necessary/useful to the example?
There was a problem hiding this comment.
It is not necessary to demonstrate cancellation, so removed for simplicity.
A side note: we are assuming our users should know AsyncIO is single threaded. Without await between creating the task and cancelling the task, there is no context switching, so the task is cancelled without given a chance to run. Thus, the task should not have been created in the first place, in the real world.
Co-authored-by: Ryan McCormick <[email protected]>
No, our client example is not changed. For example, https://github.com/triton-inference-server/client/blob/main/src/python/examples/simple_grpc_aio_sequence_stream_infer_client.py#L133-L145 is not impacted. |
* Add ability to cancel AsyncIO gRPC stream or non-stream requests * Add docs on AsyncIO request cancellation * Improve AsyncIO docs * Improve example code styling Co-authored-by: Ryan McCormick <[email protected]> * Skip await on documentation --------- Co-authored-by: Ryan McCormick <[email protected]>
* Add ability to cancel AsyncIO gRPC stream or non-stream requests * Add docs on AsyncIO request cancellation * Improve AsyncIO docs * Improve example code styling * Skip await on documentation --------- Co-authored-by: Ryan McCormick <[email protected]>
Related PR: triton-inference-server/server#6408
This PR adds the ability to cancel AsyncIO gRPC stream requests. There are interface changes on
stream_infer(), but the "syntax" on the old and new interfaces are the same, so the amount of code breaking should be minimal if any.