[HybridParallel] Add ClipGradByGlobalNorm & check_finite_and_unscale in Dygraph

[ForFishes]

PR types

New features

PR changes

Others

Describe

[HybridParallel] Add ClipGradByGlobalNorm & check_finite_and_unscale in Dygraph

支持fleet下，使用amp/clip global grad。引入distributed_scaler来wrapper scaler。具体示例代码

    # 用户的单卡组网
    model = Bert(**config)
    optimizer = paddle.optimizer.AdamW(parameters=model.parameters())
    if args.use_amp:
        # 使用amp，则需要GradScaler
        scaler = paddle.amp.GradScaler(init_loss_scaling=args.scale_loss)

    # 添加fleet支持
    if paddle.distributed.get_world_size() > 1:
        model = fleet.distributed_model(model)
        optimizer = fleet.distributed_optimizer(optimizer)
        # 如果是mp/pp下的amp，则需要用distributed_scaler wrapper。原因：check_nan_inf 需要allreduce_max各个卡上信息。
        if args.use_amp:
            scaler = fleet.distributed_scaler(scaler)

   # 接下来代码，用单卡训练一致
    loss = model(input)
    if args.use_amp:
        scaler.scale(loss).backward()
        scaler.minimize(optimizer, loss)
    else:
        loss.backward()
        optimizer.step()

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[sandyhouse]

LGTM

[sandyhouse]

‘int8’ is enough.