Fix printing momentum for non-deepspeed optimizer

[Zixxy]

Fix printing momentum for non-deepspeed optimizer

[tjruwase]

@Zixxy Thanks for creating this PR. Please consider the following comments:

1. Which optimizer (or use case) motivated this PR?

2. 'betas' is not from DeepSpeed, but rather from the Adam-style optimizers of Torch: Adam, AdamW, Adamax, etc.

3. It would be great to support optimizers that name their momentum differently from 'betas', but the current fix will break optimizers that use 'betas' but are not DeepSpeed optimizers. How about changing get_mom() to take an optional string parameter representing the momentum name? def get_mom(self, name=None) or def get_mom(self, name='betas')

Will (3) work for your scenario?

[Zixxy]

1. torch.optim.SGD / torch.optim.RMSProp
2. Ah right, fixed
3. It is fix for your _report_progress function: https://github.com/microsoft/DeepSpeed/blob/2efea6944616c7be9e35874adc37dbaf150ea05e/deepspeed/runtime/engine.py#L1007 , I would have to determine optimizer type inside of this function to call get_mom(name="betas"). Let me know if that is preferable anyway.

[tjruwase]

Oh I see, you are fixing the call within the engine code. For some reason, I was thinking of the client calling engine.get_mom(). In that case, your fix works just fine. Thanks so much.

It is exciting to see other optimizers, besides Adam & Lamb, are working seemingly out-of-the-box with DeepSpeed.