How to debug the ColossalAI script in PyCharm

[songyuc]

Hi, guys,
I am a learner of ColossalAI, and I find the ColossalAI script needs to run under the "colossalai run" program as:

# with engine
colossalai run --nproc_per_node 1 train.py

So, how can I debug it in PyCharm, which require a python interpreter to run a script.

Your answer and guide will be appreciated!

[YuliangLiu0306]

If you want to debug your model, I suggest you use single gpu mode, then you can use "python XXX.py" to run your program. If you have to debug program with multiple devices, you can use debugpy in vscode, or something have same effect in pycharm.

[songyuc]

Hi, @YuliangLiu0306 , I met a problem today when running with python train.py, as below,

/home/user/software/python/anaconda/anaconda3/envs/conda-general/bin/python /home/user/***/***
/ColossalAI-Examples/image/resnet/train.py
Traceback (most recent call last):
  File "/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/site-packages/colossalai/initialize.py", line 210, in launch_from_torch
    rank = int(os.environ['RANK'])
  File "/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'RANK'

During handling of the above exception, another exception occurred:

...

RuntimeError: Could not find 'RANK' in the torch environment, visit https://www.colossalai.org/ for more information on launching with torch

Is this error due to the absence of environment variable RANK in my Ubuntu?

[binmakeswell]

Reply in #1074

[tongping]

If you want to debug your model, I suggest you use single gpu mode, then you can use "python XXX.py" to run your program. If you have to debug program with multiple devices, you can use debugpy in vscode, or something have same effect in pycharm.

As suggested, I was able to debug the model when using single gpu mode. However, sometimes we cannot use single gpu mode, such as tensor parallelism. How to debug in such situations?

[tongping]

Hi, @YuliangLiu0306 , I met a problem today when running with python train.py, as below,

/home/user/software/python/anaconda/anaconda3/envs/conda-general/bin/python /home/user/***/***
/ColossalAI-Examples/image/resnet/train.py
Traceback (most recent call last):
  File "/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/site-packages/colossalai/initialize.py", line 210, in launch_from_torch
    rank = int(os.environ['RANK'])
  File "/home/user/software/python/anaconda/anaconda3/envs/conda-general/lib/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'RANK'

During handling of the above exception, another exception occurred:

...

RuntimeError: Could not find 'RANK' in the torch environment, visit https://www.colossalai.org/ for more information on launching with torch

Is this error due to the absence of environment variable RANK in my Ubuntu?
Yes, that is the reason.
This is easy to fix. You could define the environment variables in your code like this:
os.environ['RANK'] = '0'
os.environ['LOCAL_RANK'] = '0'
os.environ['WORLD_SIZE'] = '2'
os.environ['MASTER_ADDR'] = '1d2d8b0466e2'
os.environ['MASTER_PORT'] = '42043'

[kangxi1818]

what does '1d2d8b0466e2' and '42043' means and how to get that?