Optimize the megatron plugin

[zhangsheng377]

Optimize the megatron plugin, and make the megatron's version to r0.5.0.
#2501 (comment)
#2390 (comment)

[zhangsheng377]

@muellerzr @pacman100

[muellerzr]

Hey @zhangsheng377, let me give it a look now. FYI Sourab is no longer a part of HF, so I will be taking up Megatron/DS/FSDP

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zhangsheng377]

Hey @zhangsheng377, let me give it a look now. FYI Sourab is no longer a part of HF, so I will be taking up Megatron/DS/FSDP

It's really unfortunate to hear about Sourab's departure. I will try to modify the code format shortly. Thank you for letting me know.

[zhangsheng377]

@muellerzr Is there anything else that needs to be done?
And by the way, what are the requirements to become a committer?

[muellerzr]

Thanks we can go ahead and merge this. Why do you want permissions to be a committer out of curiosity?

[zhangsheng377]

Thanks we can go ahead and merge this. Why do you want permissions to be a committer out of curiosity?

Thank you. Because I have a strong interest in open source, I think the Accelerate project is very promising.
And my current working is training large models with Megatron, so I think I can contribute to Accelerate. (By the way, I also contributed distributed LoRa features to PEFT 😃)

[muellerzr]

For security reasons, you can only be at most a contributor, and we need to manually approve running the CI. However you have contributed to this project and it will show up in github as such :)