Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort currently does not support bool dtype on CUDA #664

Closed
twsl opened this issue Dec 7, 2021 · 1 comment · Fixed by #665
Closed

Sort currently does not support bool dtype on CUDA #664

twsl opened this issue Dec 7, 2021 · 1 comment · Fixed by #665
Labels
bug / fix Something isn't working help wanted Extra attention is needed

Comments

@twsl
Copy link
Contributor

twsl commented Dec 7, 2021

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

  1. Run mAP on CUDA
RuntimeError: Sort currently does not support bool dtype on CUDA.
    self._call_and_handle_interrupt(
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
    self._dispatch()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1272, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in run_stage
    return self._run_train()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1312, in _run_train
    self.fit_loop.run()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 234, in advance
    self.epoch_loop.run(data_fetcher)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 146, in run
    self.on_advance_end()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 246, in on_advance_end
    self._run_validation()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 341, in _run_validation
    self.val_loop.run()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 151, in run
    output = self.on_run_end()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 133, in on_run_end
    self._on_evaluation_epoch_end()
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 240, in _on_evaluation_epoch_end
    self.trainer.call_hook(hook_name)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1483, in call_hook
    output = model_fx(*args, **kwargs)
  File "/home/swarm/tkupek/sync/minimal.py", line 101, in on_validation_epoch_end
    result = self.val_map.compute()  # GPUs get stuck here
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/torchmetrics/metric.py", line 380, in wrapped_func
    value = compute(*args, **kwargs)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/torchmetrics/detection/map.py", line 668, in compute
    overall, map, mar = self._calculate(self._get_classes())
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/torchmetrics/detection/map.py", line 513, in _calculate
    eval_imgs = [
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/torchmetrics/detection/map.py", line 514, in <listcomp>
    self._evaluate_image(img_id, class_id, area, max_detections, ious)
  File "/home/swarm/tkupek/venv/lib/python3.8/site-packages/torchmetrics/detection/map.py", line 374, in _evaluate_image
    ignore_area_sorted, gtind = torch.sort(ignore_area)
RuntimeError: Sort currently does not support bool dtype on CUDA.

Process finished with exit code 1

Code sample

See tests

Expected behavior

Calculate mAP on CUDA without issues

Environment

  • PyTorch Version (e.g., 1.0):
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, source):
  • Build command you used (if compiling from source):
  • Python version:
  • CUDA/cuDNN version:
  • GPU models and configuration:
  • Any other relevant information:

Additional context

Raised by @tkupek

@twsl twsl added bug / fix Something isn't working help wanted Extra attention is needed labels Dec 7, 2021
@twsl twsl mentioned this issue Dec 7, 2021
4 tasks
@tkupek
Copy link
Contributor

tkupek commented Dec 7, 2021

Thanks for opening this.

Environment

  • PyTorch Version (e.g., 1.0): 1.10.0
  • OS (e.g., Linux): Linux
  • How you installed PyTorch (conda, pip, source): pip
  • Build command you used (if compiling from source):
  • Python version: 3.8
  • CUDA/cuDNN version:
  • GPU models and configuration: GTX 2080
  • Any other relevant information:

@Borda Borda closed this as completed in #665 Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants