Baichuan 2-13B Chat CPU utilization rate only 20%

[Storm0921]

When I conducted experiments using Baichuan 2-13B Chat, I found that the CPU utilization rate of each core was only 20%. I believe it is because BigDL LM did not perform relevant optimization.Can you give me an explain?

[Storm0921]

[Storm0921]

Baichuan1-13B-Chat not has same question, so I am very confused

[glorysdj]

I test Baichuan 13B Chat on an spr machine, it turned out CPU utilization per core could reach to almost 80%. The test script is sourced from here.

please also test Baichuan 2-13B Chat

[Zhengjin-Wang]

In my test environment, baichuan 2 13b chat has the same cpu utilization as baichuan 1.

[gc-fu]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

[Storm0921]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

Yes, I installed bigdl llm according to the instructions in README. However, I am not quite sure what you mean by bigdl llm init - t. Can you provide a detailed description？

[gc-fu]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

Yes, I installed bigdl llm according to the instructions in README. However, I am not quite sure what you mean by bigdl llm init - t. Can you provide a detailed description？

Just run 'source bigdl-llm-init -t'. Be careful to check if the OMP_NUM_THREADS is set to the correct number.

[Storm0921]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

Yes, I installed bigdl llm according to the instructions in README. However, I am not quite sure what you mean by bigdl llm init - t. Can you provide a detailed description？

Just run 'source bigdl-llm-init -t'. Be careful to check if the OMP_NUM_THREADS is set to the correct number.

What I understand is that bigdl llm init can fully mobilize the set number of cores to run the program, but the difficulty I am currently facing is that all cores are running, but the utilization rate is relatively low. I don't know why, but I feel that it has nothing to do with the screenshots

[gc-fu]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

Yes, I installed bigdl llm according to the instructions in README. However, I am not quite sure what you mean by bigdl llm init - t. Can you provide a detailed description？

Just run 'source bigdl-llm-init -t'. Be careful to check if the OMP_NUM_THREADS is set to the correct number.

What I understand is that bigdl llm init can fully mobilize the set number of cores to run the program, but the difficulty I am currently facing is that all cores are running, but the utilization rate is relatively low. I don't know why, but I feel that it has nothing to do with the screenshots

If you are using numactl correctly, then it should not run all of the cores. Assuming you want to use 0-23 cores to run your program, which should be in one socket, then try the following:

source bigdl-llm-init -t
export OMP_NUM_THREADS=24
numactl -C 0-23 -m 0 python3 xxx.py

This would restrict the program to run on core 0-23 so that it would not utilize all of the cores in your system.

Also, can you attach your hardware config through commands like lscpu?

[Storm0921]

Are you using the recommended environment variable settings? Try bigdl-llm-init -t

Yes, I installed bigdl llm according to the instructions in README. However, I am not quite sure what you mean by bigdl llm init - t. Can you provide a detailed description？

Just run 'source bigdl-llm-init -t'. Be careful to check if the OMP_NUM_THREADS is set to the correct number.

What I understand is that bigdl llm init can fully mobilize the set number of cores to run the program, but the difficulty I am currently facing is that all cores are running, but the utilization rate is relatively low. I don't know why, but I feel that it has nothing to do with the screenshots

If you are using numactl correctly, then it should not run all of the cores. Assuming you want to use 0-23 cores to run your program, which should be in one socket, then try the following:

source bigdl-llm-init -t
export OMP_NUM_THREADS=24
numactl -C 0-23 -m 0 python3 xxx.py

This would restrict the program to run on core 0-23 so that it would not utilize all of the cores in your system.

Also, can you attach your hardware config through commands like lscpu?

Thank you for your reply.
source bigdl-llm-init -t

Then export OMP_NUM_THREADS=32
Next I used the command：sudo numactl -C 0-31 -m 0 python3 mysql_qa_bench.py，But here's the hint
set_mempolicy: Operation not permitted
setting membind: Operation not permitted
What can I do to solve this problem?

[gc-fu]

Can you attach the hardware config using lscpu?

You can try sudo numactl -C 0-31 python3 mysql_qa_bench.py. The problem is possibly caused by memory socket bind.

[Storm0921]

Can you attach the hardware config using lscpu?

You can try sudo numactl -C 0-31 python3 mysql_qa_bench.py. The problem is possibly caused by memory socket bind.

[forcekeng]

When I conducted experiments using Baichuan 2-13B Chat, I found that the CPU utilization rate of each core was only 20%. I believe it is because BigDL LM did not perform relevant optimization.Can you give me an explain?

I don't see about 20% in your images, do you mean the first column? It seems not percentage of CPU but time.

[Storm0921]

When I conducted experiments using Baichuan 2-13B Chat, I found that the CPU utilization rate of each core was only 20%. I believe it is because BigDL LM did not perform relevant optimization.Can you give me an explain?

I don't see about 20% in your images, do you mean the first column? It seems not percentage of CPU but time.

Top command's first column.Isn'tit representing the CPU usage rate?

[gc-fu]

Can you attach the hardware config using lscpu?
You can try sudo numactl -C 0-31 python3 mysql_qa_bench.py. The problem is possibly caused by memory socket bind.

Hi, according to your lscpu result, please try this command:

sudo numactl -C 0-31 python3 mysql_qa_bench.py
# Or using taskset
taskset -c "0-31" python3 mysql_qa_bench.py

Also, consider using htop to get visual result of CPU utilization.