pretrain learning rate is le-8?

[hegang1-tal]

Required prerequisites

• I have read the documentation https://github.com/baichuan-inc/baichuan-7B/blob/HEAD/README.md.
• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Questions

hi, I found in deepspeed config file that the learning rate is le-8. I am wandering that is this too small for pretraining ?

Checklist

• I have provided all relevant and necessary information above.
• I have chosen a suitable title for this issue.