[CPU] Use allreduce_low_latency for AutoTP and implement low latency allreduce for CPU backend (single node)

[delock]

In optimize DeepSpeed AutoTP inference workload, allreduce latency is critical to inference scaling. This PR introduce a new deepspeed.comm interface allreduce_low_latency to allow communication backend to implement a low latency version of allreduce, which allows communication backend or library to optimize allreduce performance with different strategy.

In this PR we implemented a low latency allreduce on CCLBackend for CPU backend. Experiment shows for very small messages, low latency allreduce bring more than 2x performance boost, compared to calling oneCCL library which is optimized for large message size with concurrency.

The opportunity of low latency allreduce comes from the fact that allreduce in training is different from allreduce in inference tensor parallel. In training, allreduce usually has:

1. Large message size
2. Concurrency
   Optimize of large message size and concurrency improves communcation in training but also complicate the software infrastructure. In inference the message size is small and there is no need for concurrency. Thus these software design won't help allreduce latency but might hurt it. Seperate the path for latency and throughput gives an opportunity to optimize for latency in inference scenario.

[tjruwase]

@delock, thanks for the PR. I am concerned that allreduce_low_latency could be a misleading name since the low latency is only achieved for small message sizes and low concurrency. A user could mistakenly think that this function could help improve allreduce performance for training as well. Can we explore names that more descriptive of the target uses, such as:

1. small_message_allreduce
2. inference_allreduce

Thanks!

[delock]

@delock, thanks for the PR. I am concerned that allreduce_low_latency could be a misleading name since the low latency is only achieved for small message sizes and low concurrency. A user could mistakenly think that this function could help improve allreduce performance for training as well. Can we explore names that more descriptive of the target uses, such as:

1. small_message_allreduce
2. inference_allreduce

Thanks!

Thanks for the comments @tjruwase , I think inference_allreduce would be better since in training scenario concurrency could be assumed. Would make the change.

[delock]

@delock, thanks for the PR. I am concerned that allreduce_low_latency could be a misleading name since the low latency is only achieved for small message sizes and low concurrency. A user could mistakenly think that this function could help improve allreduce performance for training as well. Can we explore names that more descriptive of the target uses, such as:

1. small_message_allreduce
2. inference_allreduce

Thanks!

Thanks for the comments @tjruwase , I think inference_allreduce would be better since in training scenario concurrency could be assumed. Would make the change.

Will use inference_all_reduce to keep naming consistency with all_reduce.