[Kunlun] add gen_bkcl_id_op, support multi XPU cards training using multiprocess #30858
Conversation
* fix some uts' interfaces * add todos and fix err_messages
fix,test=notest fix,test=notest fix,test=notest fix,test=notest fix
|
Thanks for your contribution! |
There was a problem hiding this comment.
后续TODO建议 @vslyu
1、统一gen_nccl_id、c_gen_nccl_id、gen_bkcl_id、c_gen_bkcl_id，叫c_gen_comm_id，采用kernel op的形式。
2、c_comm_init_op，采用kernel op的形式。
| @@ -0,0 +1,194 @@ | |||
| /* Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. | |||
| @@ -566,12 +574,12 @@ def run_gpu_fleet_api_trainer(self, args): | |||
| args.trainer_id = paddle.distributed.get_rank() | |||
|
|
|||
| # 3. init parallel env | |||
| if args.update_method == "nccl2": | |||
| if args.update_method == "nccl2" or "bkcl": | |||
There was a problem hiding this comment.
this fleet is paddle.fluid.incubate.fleet, maybe need another test
There was a problem hiding this comment.
New paddle.distributed.fleet API interface is imported again In this run_use_fleet_api_trainer function:
import paddle.distributed.fleet as fleet
import paddle.distributed.fleet.base.role_maker as role_maker
so , the unittest finally runs with new paddle.distributed.fleet API.
| import os | ||
| import copy | ||
| import sys | ||
| sys.path.append("..") |
There was a problem hiding this comment.
from ..launch_function_helper
There was a problem hiding this comment.
保留sys.path.append("..")，保持和其他xpu单测一致吧
| platform::errors::PreconditionNotMet( | ||
| "CCommInitOp can run on gpu place only.")); | ||
| "CCommInitOp can run on gpu or xpu place only.")); | ||
|
|
||
| auto var = scope.FindVar(Input("X")); | ||
| PADDLE_ENFORCE_NOT_NULL( | ||
| var, platform::errors::InvalidArgument("Input con not be empty.")); |
| PADDLE_THROW(platform::errors::PreconditionNotMet( | ||
| "PaddlePaddle should compile with GPU.")); | ||
| PADDLE_THROW(platform::errors::PreconditionNotMet( | ||
| "PaddlePaddle should compile with GPU.")); |
There was a problem hiding this comment.
should compile -> should be compiled?
| bkcl_id, nranks, rank_id, device_id, rid); | ||
| #else | ||
| PADDLE_THROW(platform::errors::PreconditionNotMet( | ||
| "PaddlePaddle should compile with XPU.")); |
收到，待xpu动态图，静态图性能优化时再考虑精简代码。 |
PR types
New features
PR changes
Others
Describe