Support for Dynamic Batch Size in CUDA Graph Inference with TensorRT

[OuyangChao]

I'm currently exploring TensorRT for inference tasks and aiming to optimize performance using CUDA graph. One of the requirements for my application is to support dynamic batch sizes during inference. While TensorRT provides dynamic shape support, I couldn't find sufficient information on how to incorporate this feature into CUDA graph inference.

I would appreciate any guidance, documentation, or examples demonstrating how to implement dynamic batch size support in CUDA graph inference with TensorRT.

Thank you for your assistance!

[zerollzeng]

See https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#cuda-graphs

Basically, when changing the input shapes, you have to re-capture the graph, because some internal state is changed.

[zerollzeng]

Therefore, the best practice is to use one execution context per captured graph, and to share memory across the contexts with createExecutionContextWithoutDeviceMemory(). will it help?

[lix19937]

share memory across the contexts need mutex /lock ? @zerollzeng

[ttyio]

share memory across the contexts need mutex /lock ? @zerollzeng

Yes, User need make sure there is concurrent execution when two context share the same memory. Because some reduction kernel might have race condition when they write to the same memory, the behavior is undefined.

[ttyio]

closing since this should already solved, thanks all!