Run GptSession without openmpi?

[haohuanw]

I have some relatively basic use cases (no parallelism) that I am able to run in environments that doesn't have MPI installed. However, i found that since 0130 release #1019 MPI is force initialized in this function call: https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/runtime/gptSession.cpp#L191-L192 which prevents me to do a trt-llm upgrade.

since this function is part of closed source, is it possible to update the function to not initialize MPI when it is not necessary?

[MartinMarciniszyn]

The code reference seems outdated. @haohuanw , can you please post the snippet that forces the MPI initialization? Generally, the code base is tightly integrated with MPI. If it is an option, the easy way forward would be installing MPI in your environment. Also, note that GptSession will not be actively maintained going forward. The official entry point in the API will be the Executor class.

[haohuanw]

basically KVCacheManager forces a MPI.SESSION initialize starting #1019 releases.

Previously, i just need to make sure WorldConfig is not constructed with mpi static function and it actually works fine.

[haohuanw]

@MartinMarciniszyn i had a chance to check new executor interface. to me it seems that if i can fake a Communicator class what i am looking for will be fulfilled.

i currently see that there are no method in communicator that i can override, do you know when would that happen?

[MartinMarciniszyn]

The code in KV Cache Manager has been modified. It should not require initialization of MPI for single device setups anymore. This will be released next Tuesday.