Allow DeepSpeed models to be initialized with optimizer=None #469
Conversation
gcooper-isi
requested review from
arashashari,
awan-10,
cli99,
conglongli,
eltonzheng,
jeffra,
minjiaz,
niumanar,
RezaYazdaniAminabadi,
samyam,
ShadenSmith and
tjruwase
as code owners
|
|
|
Hi @gcooper-isi , thanks for this contribution! I pushed a small commit that adds a unit test and also avoids an issue when no optimizer is specified in the JSON configuration. If you haven't yet, can you sign the CLA from the above bot? We will need it before merging. |
|
Thanks so much for adding the unit test! I have asked my employer for permission to sign the CLA, which I don't expect to be an issue, and I will sign it as soon as they say it's OK. |
|
@gcooper-isi just wanted to bump this back up. Okay to sign the CLA? |
I'm OK to sign it personally, but since this requires permission from my company it has had to work its way through the various levels of bureaucracy before they finish the approval. I keep getting periodic updates that it has made it through each next step and I'll continue to follow up. |
As far as my contracts team is concerned, though, I don't have a deadline for signing this contract. If you can provide any information that could help convince them that doing this in a timely way would be important, that may help speed things along. |
@gcooper-isi DeepSpeed is updating at a rapid pace. You might tell them that if they are too slow the risk is that your contributions might be too out of date to be successfully merged. We all see value in your contributions, so I hope they agree soon. The CLA as far as I understand shouldn’t be too controversial for this type of commit, but of course I’m not a lawyer :) |
|
I finally got permission from my employer to sign the CLA and have done so. Please let me know if there is anything else I can do. |
* Merge chatgpt v2 to v3 - finalized (#484) * [squash] staging chatgpt v1 (#463) Co-authored-by: Reza Yazdani <[email protected]> Co-authored-by: yaozhewei <[email protected]> Co-authored-by: Tunji Ruwase <[email protected]> * [partial] formatting fixes * quantizer fixes * fix for bert tests * formatting fixes * re-enable _param_slice_mappings in z2 * Enable the QKV requires_grad when in training mode (#466) Co-authored-by: Jeff Rasley <[email protected]> * fixes for attention enable_training flag * commit to trigger CI * fix for distil-bert param * fixes for training context errors * remove reza's qkv-optimization (#469) Co-authored-by: Jeff Rasley <[email protected]> * Chatgpt - Fuse lora params at HybridEngine (#472) Co-authored-by: Jeff Rasley <[email protected]> * add option to enable non-pin mode (#473) * Chatgpt - fuse lora non pinned case (#474) * Fix fuse/unfuse lora for Z3 and non-pinned parameter * unfuse_lora_weight for non-pinned case * fix the multiple issue for lora parameters * formatting * fuse lora only when available --------- Co-authored-by: Jeff Rasley <[email protected]> * Chatgpt/release inference cache (#475) * Fix fuse/unfuse lora for Z3 and non-pinned parameter * unfuse_lora_weight for non-pinned case * release/retake the inference cache after/before generate * remove duplicated _fuse_lora function * fix formatting * fix hybrid-engine config issue * update formatting * Chatgpt - fuse qkv v2 (#478) Co-authored-by: Jeff Rasley <[email protected]> * ChatGPT: Refactor Hybrid Engine Config (#477) Co-authored-by: Lok Chand Koppaka <[email protected]> * Inference Workspace Tweaks (#481) * Safety checks around inference workspace allocation, extra flushing * Formatting fixes * Merge fix * Chatgpt/inference tp (#480) * Update the merged-QKV weights only if there is difference with the model parameter * remove the hard-coded size * always reset qkv params to updated ones after running step * Add the infernce-tp group and tensor sharding to run inference in model-parallel mode * optimize the gather/mp-sharding part * Add hybrid_engine changes * fix config issue * Formatting fixes. Reset_qkv duplicate removal. * fix bloom container. * fix format. --------- Co-authored-by: Ammar Ahmad Awan <[email protected]> Co-authored-by: Lok Chand Koppaka <[email protected]> * fix formatting * more clean-up --------- Co-authored-by: Jeff Rasley <[email protected]> Co-authored-by: yaozhewei <[email protected]> Co-authored-by: Tunji Ruwase <[email protected]> Co-authored-by: Masahiro Tanaka <[email protected]> Co-authored-by: Michael Wyatt <[email protected]> Co-authored-by: Lok Chand Koppaka <[email protected]> Co-authored-by: Connor Holmes <[email protected]> Co-authored-by: Ammar Ahmad Awan <[email protected]> * fix a bug on lora-fusion (#487) * Cholmes/v3 workspace bugfixes (#488) * Miscellaneous workspace fixes, new config param * Fix typo --------- Co-authored-by: Reza Yazdani <[email protected]> Co-authored-by: Jeff Rasley <[email protected]> Co-authored-by: yaozhewei <[email protected]> Co-authored-by: Tunji Ruwase <[email protected]> Co-authored-by: Masahiro Tanaka <[email protected]> Co-authored-by: Michael Wyatt <[email protected]> Co-authored-by: Lok Chand Koppaka <[email protected]> Co-authored-by: Connor Holmes <[email protected]>
The optimizer uses quite a bit of GPU memory. For training, this is necessary, but not for simply running the model. Therefore, allowing
optimizerto be set toNonecan dramatically reduce GPU memory usage when running the model. However, in the current version of the code, doing so will cause an exception to be thrown.