Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

which version of hydra do you recommend to work with CUDA? #31

Open
ahsanfarooqui opened this issue Sep 17, 2022 · 2 comments
Open

which version of hydra do you recommend to work with CUDA? #31

ahsanfarooqui opened this issue Sep 17, 2022 · 2 comments

Comments

@ahsanfarooqui
Copy link

Hello. I am getting this error everytime I run the code

/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'cls': Defaults list is missing _self_. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/default_element.py:126: UserWarning: In 'model/Menghao': Usage of deprecated keyword in package header '# @Package group'.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/changes_to_package_header for more information
See {url} for more information"""
{'model': {'name': 'Menghao'}, 'batch_size': 16, 'epoch': 200, 'learning_rate': 0.001, 'gpu': 1, 'num_point': 1024, 'optimizer': 'Adam', 'weight_decay': 0.0001, 'normal': True}
[2022-09-17 13:35:42,215][main][INFO] - Load dataset ...
The size of train data is 9843
The size of test data is 2468
Error executing job with overrides: []
Traceback (most recent call last):
File "train_cls.py", line 162, in
main()
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
config_name=config_name,
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
lambda: hydra.run(
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in
overrides=args.overrides,
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
_ = ret.return_value
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "train_cls.py", line 68, in main
classifier = getattr(importlib.import_module('models.{}.model'.format(args.model.name)), 'PointTransformerCls')(args).cuda()
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 689, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 579, in _apply
module._apply(fn)
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 602, in _apply
param_applied = fn(param)
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 689, in
return self._apply(lambda t: t.cuda(device))
File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/cuda/init.py", line 217, in _lazy_init
torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

If I dont run it through hydra, it compiles easily. Any specific versions of Hydra that you are using?

@tengfeixue-victor
Copy link

Hello. I am getting this error everytime I run the code

/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'cls': Defaults list is missing _self_. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information warnings.warn(msg, UserWarning) /home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/default_element.py:126: UserWarning: In 'model/Menghao': Usage of deprecated keyword in package header '# @Package group'. See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/changes_to_package_header for more information See {url} for more information""" {'model': {'name': 'Menghao'}, 'batch_size': 16, 'epoch': 200, 'learning_rate': 0.001, 'gpu': 1, 'num_point': 1024, 'optimizer': 'Adam', 'weight_decay': 0.0001, 'normal': True} [2022-09-17 13:35:42,215][main][INFO] - Load dataset ... The size of train data is 9843 The size of test data is 2468 Error executing job with overrides: [] Traceback (most recent call last): File "train_cls.py", line 162, in main() File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main config_name=config_name, File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra lambda: hydra.run( File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 214, in run_and_report raise ex File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report return func() File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in overrides=args.overrides, File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run _ = ret.return_value File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value raise self._return_value File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job ret.return_value = task_function(task_cfg) File "train_cls.py", line 68, in main classifier = getattr(importlib.import_module('models.{}.model'.format(args.model.name)), 'PointTransformerCls')(args).cuda() File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 689, in cuda return self._apply(lambda t: t.cuda(device)) File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 579, in _apply module._apply(fn) File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 602, in _apply param_applied = fn(param) File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 689, in return self._apply(lambda t: t.cuda(device)) File "/home/myuser/anaconda3/envs/myenv/lib/python3.7/site-packages/torch/cuda/init.py", line 217, in _lazy_init torch._C._cuda_init() RuntimeError: No CUDA GPUs are available

If I dont run it through hydra, it compiles easily. Any specific versions of Hydra that you are using?

Any solutions? I got the same issue. thanks!

@LudvigDillen
Copy link

LudvigDillen commented Jun 21, 2023

I got the same issue and solved it by removing the line
os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu)
from the train_cls.py file. Apparently, the script tried to access my second GPU, even though I only have one. Now everything runs for me.
Hope this helps :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants