Improve the performance of GMC router

[zhlsunshine]

Description

Improve the performance of GMC router by:

1. Define the custom transport
2. Re-use the http.Client with custom transport

Issues

n/a.

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)

Dependencies

n/a.

Tests

No change for use case and scenario.

[irisdingbj]

If we can have some performance data comparison, it will be better.

[zhlsunshine]

If we can have some performance data comparison, it will be better.

Hi @irisdingbj, how about to add this in the task list? Or we can pick cherry this as release 0.9?

[KfreeZ]

issue #338 #258 related
if this PR can fix the issues, please also revert the PR opea-project/GenAIExamples#646

[zhlsunshine]

issue #338 #258 related if this PR can fix the issues, please also revert the PR opea-project/GenAIExamples#646

Hi @KfreeZ, thanks for the information!

[zhlsunshine]

Hi @irisdingbj, I added the milestone 0.9 and 1.0 to this PR, then we may do the performance test based on milestone 0.9

[zhlsunshine]

issue #338 #258 related if this PR can fix the issues, please also revert the PR opea-project/GenAIExamples#646

Hi @KfreeZ, I have test manually based on your opea-project/GenAIExamples#646 in the Xeon node with model meta-llama/CodeLlama-7b-hf, below is the output:

[KfreeZ]

issue #338 #258 related if this PR can fix the issues, please also revert the PR opea-project/GenAIExamples#646

Hi @KfreeZ, I have test manually based on your opea-project/GenAIExamples#646 in the Xeon node with model meta-llama/CodeLlama-7b-hf, below is the output:

appreciate it, thank you @zhlsunshine !!!