Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

subprocess.CalledProcessError: Command '' died with <Signals.SIGSEGV: 11>. #16

Open
SibylGao opened this issue Sep 19, 2022 · 2 comments
Open

subprocess.CalledProcessError: Command '' died with <Signals.SIGSEGV: 11>. #16

SibylGao opened this issue Sep 19, 2022 · 2 comments

Comments

@SibylGao
Copy link

SibylGao commented Sep 19, 2022 •
edited
Loading

Hi！Thanks for sharing your awesome code.
But I have some problem when i running this code...
My error massages:

data/kitti/training/image_2/001773.png
data/kitti/training/image_2/001816.png
data/kitti/training/image_2/002829.png
data/kitti/training/image_3/001773.png
data/kitti/training/image_3/001816.png
data/kitti/training/image_3/002829.png
{'NAME': 'filter_truncated', 'AREA_RATIO_THRESH': None, 'AREA_2D_RATIO_THRESH': None, 'GT_TRUNCATED_THRESH': 0.98}
filter truncated ratio: null 3d boxes [[ 2.99       -3.87       -0.66499996  4.43        1.84        1.75
  -0.2907964 ]] flipped False image idx 890 frame_id 001773 

/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:143: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
data/kitti/training/image_2/004052.png
data/kitti/training/image_3/004052.png
Traceback (most recent call last):
  File "/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/home/users/gaoshiyu01/anaconda3/envs/liga5/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/users/gaoshiyu01/anaconda3/envs/liga5/bin/python', '-u', 'tools/train.py', '--local_rank=1', '--launcher', 'pytorch', '--fix_random_seed', '--sync_bn', '--save_to_file', '--cfg_file', './configs/stereo/kitti_models/liga.3d-and-bev.yaml', '--exp_name', 'test1']' died with <Signals.SIGSEGV: 11>.

Seems like a common bug caused by mmdet, so i followed the instruction from: mmdet bug report and checked my running/compiling libraries with nvcc, but everything seems alright, i still have no idea how to fix it, could you please provide more info, thanks a lot :)

My environment:

  • nvcc --version: 10.1
  • nvidia-smi: 10.2
  • Cudatoolkit: 10.1
  • python: 3.7.13
  • pytorch: 1.6.0
  • spconv : 1.2.1
  • mmcv-full: 1.2.1
  • mmdet: 2.6.0
  • mmpycocotools : 12.0.3
My conda list: ```

Name Version Build Channel
libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
addict 2.4.0 pypi_0 pypi
blas 1.0 mkl defaults
ca-certificates 2022.07.19 h06a4308_0 defaults
certifi 2022.6.15 py37h06a4308_0 defaults
cudatoolkit 10.1.243 h6bb024c_0 defaults
cycler 0.11.0 pypi_0 pypi
cython 0.29.32 pypi_0 pypi
easydict 1.9 pypi_0 pypi
fire 0.4.0 pypi_0 pypi
fonttools 4.37.2 pypi_0 pypi
freetype 2.11.0 h70c0345_0 defaults
future 0.18.2 pypi_0 pypi
giflib 5.2.1 h7b6447c_0 defaults
imageio 2.21.3 pypi_0 pypi
importlib-metadata 4.12.0 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561 defaults
jpeg 9e h7f8727e_0 defaults
kiwisolver 1.4.4 pypi_0 pypi
lcms2 2.12 h3be6417_0 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
lerc 3.0 h295c915_0 defaults
libdeflate 1.8 h7f8727e_5 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libpng 1.6.37 hbc83047_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
libtiff 4.4.0 hecacb30_0 defaults
libwebp 1.2.2 h55f646e_0 defaults
libwebp-base 1.2.2 h7f8727e_0 defaults
liga 0.1.0+0 dev_0
llvmlite 0.39.1 pypi_0 pypi
lz4-c 1.9.3 h295c915_1 defaults
matplotlib 3.5.3 pypi_0 pypi
mkl 2021.4.0 h06a4308_640 defaults
mkl-service 2.4.0 py37h7f8727e_0 defaults
mkl_fft 1.3.1 py37hd3c417c_0 defaults
mkl_random 1.2.2 py37h51133e4_0 defaults
mmcv-full 1.2.1 pypi_0 pypi
mmdet 2.6.0 dev_0
mmpycocotools 12.0.3 pypi_0 pypi
ncurses 6.3 h5eee18b_3 defaults
networkx 2.6.3 pypi_0 pypi
ninja 1.10.2 h06a4308_5 defaults
ninja-base 1.10.2 hd09550d_5 defaults
numba 0.56.2 pypi_0 pypi
numpy 1.21.5 py37h6c91a56_3 defaults
numpy-base 1.21.5 py37ha15fc14_3 defaults
opencv-python 4.6.0.66 pypi_0 pypi
openssl 1.1.1q h7f8727e_0 defaults
packaging 21.3 pypi_0 pypi
pillow 9.2.0 py37hace64e9_1 defaults
pip 22.1.2 py37h06a4308_0 defaults
protobuf 3.20.1 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
python 3.7.13 h12debd9_0 defaults
python-dateutil 2.8.2 pypi_0 pypi
pytorch 1.6.0 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch
pywavelets 1.3.0 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1 defaults
scikit-image 0.19.3 pypi_0 pypi
scipy 1.7.3 pypi_0 pypi
setuptools 59.8.0 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_1 defaults
spconv 1.2.1 pypi_0 pypi
sqlite 3.39.2 h5082296_0 defaults
tensorboardx 2.5.1 pypi_0 pypi
termcolor 2.0.1 pypi_0 pypi
terminaltables 3.1.10 pypi_0 pypi
tifffile 2021.11.2 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
torchvision 0.7.0 py37_cu101 pytorch
tqdm 4.64.1 pypi_0 pypi
typing-extensions 4.3.0 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0 defaults
xz 5.2.5 h7f8727e_1 defaults
yapf 0.32.0 pypi_0 pypi
zipp 3.8.1 pypi_0 pypi
zlib 1.2.12 h5eee18b_3 defaults
zstd 1.5.2 ha4553b6_0 defaults

</details>
@SibylGao SibylGao changed the title subprocess.CalledProcessError: Command '['/anaconda3/envs/liga5/bin/python', '-u', 'tools/train.py', '--local_rank=1', '--launcher', 'pytorch', '--fix_random_seed', '--sync_bn', '--save_to_file', '--cfg_file', './configs/stereo/kitti_models/liga.3d-and-bev.yaml', '--exp_name', 'test1']' died with <Signals.SIGSEGV: 11>. subprocess.CalledProcessError: Command '' died with <Signals.SIGSEGV: 11>. Sep 19, 2022
@thu-yn
Copy link

thu-yn commented May 1, 2024

Have you solved it? I also encountered the same problem

@SibylGao
Copy link
Author

SibylGao commented May 1, 2024

Have you solved it? I also encountered the same problem

sorry, it's been a long time, i can't remember.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SibylGao @thu-yn