Segmentation fault after 2h of training

[raphaelsulzer]

Hi,
thank you for publishing this great work here.

I am trying to retrain NKSR on ShapeNet (with a differently sampled point cloud).

Training starts and runs for ~2h, but then stops with a segmentation fault. I would appreciate any help to debug this.
Below my conda environment and the output of train.py:

Conda environment:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
absl-py                   1.4.0              pyhd8ed1ab_0    conda-forge
addict                    2.4.0                    pypi_0    pypi
aiohttp                   3.8.4           py310h2372a71_1    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
ansi2html                 1.8.0                    pypi_0    pypi
antlr-python-runtime      4.9.3              pyhd8ed1ab_1    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
binutils_linux-64         2.40                 hbdbef99_0    conda-forge
blas                      1.0                         mkl  
blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
calmsize                  0.1.3                    pypi_0    pypi
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cmake                     3.26.4               hcfe8598_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.3                    pypi_0    pypi
configargparse            1.5.3                    pypi_0    pypi
contourpy                 1.1.0           py310hd41b1e2_0    conda-forge
cryptography              41.0.1          py310h75e40e8_0    conda-forge
cuda-cccl                 11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-command-line-tools   11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-compiler             11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-cudart               11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-cudart-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-cuobjdump            11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-cupti                11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-cuxxfilt             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-documentation        11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-driver-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-gdb                  11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-libraries            11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-libraries-dev        11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-memcheck             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nsight               11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nsight-compute       11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-nvcc                 11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvdisasm             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvml-dev             11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvprof               11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-nvprune              11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvrtc                11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvrtc-dev            11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvtx                 11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-nvvp                 11.8.87                       0    nvidia/label/cuda-11.8.0
cuda-profiler-api         11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-runtime              11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-sanitizer-api        11.8.86                       0    nvidia/label/cuda-11.8.0
cuda-toolkit              11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-tools                11.8.0                        0    nvidia/label/cuda-11.8.0
cuda-visual-tools         11.8.0                        0    nvidia/label/cuda-11.8.0
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dash                      2.11.0                   pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.7                    pypi_0    pypi
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
docker-pycreds            0.4.0                      py_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fastjsonschema            2.17.1                   pypi_0    pypi
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
fire                      0.5.0                    pypi_0    pypi
flask                     2.2.5                    pypi_0    pypi
flatten-dict              0.4.2              pyhd8ed1ab_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.40.0          py310h2372a71_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
frozenlist                1.3.3           py310h5764c6d_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gcc_impl_linux-64         11.4.0               h7aa1c59_0    conda-forge
gcc_linux-64              11.4.0               hfd045f2_0    conda-forge
gds-tools                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
gettext                   0.21.1               h27087fc_0    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
glib                      2.76.3               hfc55251_0    conda-forge
glib-tools                2.76.3               hfc55251_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
google-auth               2.21.0             pyh1a96a4e_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpcio                    1.46.3          py310hba10ccf_0    conda-forge
gst-plugins-base          1.22.3               h938bd60_1    conda-forge
gstreamer                 1.22.3               h977cf35_1    conda-forge
gxx_impl_linux-64         11.4.0               h7aa1c59_0    conda-forge
gxx_linux-64              11.4.0               hfc1ae95_0    conda-forge
harfbuzz                  7.3.0                hdb3a94d_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.7.0              pyha770c72_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561  
ipykernel                 6.23.3                   pypi_0    pypi
ipython                   8.14.0             pyh41d4057_0    conda-forge
ipywidgets                8.0.6                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jsonschema                4.17.3                   pypi_0    pypi
jupyter-client            8.3.0                    pypi_0    pypi
jupyter-core              5.3.1                    pypi_0    pypi
jupyterlab-widgets        3.0.7                    pypi_0    pypi
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcap                    2.67                 he9d0100_0    conda-forge
libclang                  16.0.6          default_h1cdf331_0    conda-forge
libclang13                16.0.6          default_h4d60ac6_0    conda-forge
libcublas                 11.11.3.6                     0    nvidia/label/cuda-11.8.0
libcublas-dev             11.11.3.6                     0    nvidia/label/cuda-11.8.0
libcufft                  10.9.0.58                     0    nvidia/label/cuda-11.8.0
libcufft-dev              10.9.0.58                     0    nvidia/label/cuda-11.8.0
libcufile                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
libcufile-dev             1.4.0.31                      0    nvidia/label/cuda-11.8.0
libcups                   2.3.3                h36d4200_3    conda-forge
libcurand                 10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcurand-dev             10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcurl                   8.1.2                h409715c_0    conda-forge
libcusolver               11.4.1.48                     0    nvidia/label/cuda-11.8.0
libcusolver-dev           11.4.1.48                     0    nvidia/label/cuda-11.8.0
libcusparse               11.7.5.86                     0    nvidia/label/cuda-11.8.0
libcusparse-dev           11.7.5.86                     0    nvidia/label/cuda-11.8.0
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-devel_linux-64     11.4.0               h922705a_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libhwloc                  2.9.1           nocuda_h7313eea_6    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
libllvm16                 16.0.6               h5cf9203_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnpp                    11.8.0.86                     0    nvidia/label/cuda-11.8.0
libnpp-dev                11.8.0.86                     0    nvidia/label/cuda-11.8.0
libnsl                    2.0.0                h7f98852_0    conda-forge
libnvjpeg                 11.9.0.86                     0    nvidia/label/cuda-11.8.0
libnvjpeg-dev             11.9.0.86                     0    nvidia/label/cuda-11.8.0
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hbcd7760_1    conda-forge
libprotobuf               3.19.6               h3eb15da_0    conda-forge
libsanitizer              11.4.0               h4dcbe23_0    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-devel_linux-64  11.4.0               h922705a_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libsystemd0               253                  h8c4010b_1    conda-forge
libtiff                   4.5.1                h8b53f26_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.44.2               h166bdaf_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.0                h0b41bf4_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.5.0                h5d7e998_3    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lightning-utilities       0.8.0              pyhd8ed1ab_0    conda-forge
llvm-openmp               16.0.6               h4dfa4b3_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown                  3.4.3              pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py310h2372a71_0    conda-forge
matplotlib                3.7.1           py310hff52083_0    conda-forge
matplotlib-base           3.7.1           py310he60537e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
mkl                       2021.4.0           h8d4b97c_729    conda-forge
mkl-service               2.4.0           py310ha2c4b55_0    conda-forge
mkl_fft                   1.3.1           py310h2b4bcf5_1    conda-forge
mkl_random                1.2.2           py310h00e6091_0  
mpc                       1.3.1                hfe3b2da_0    conda-forge
mpfr                      4.2.0                hb012696_0    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4           py310h1fa729e_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_0    conda-forge
mysql-libs                8.0.33               hca2cd23_0    conda-forge
nbformat                  5.5.0                    pypi_0    pypi
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.6                    pypi_0    pypi
networkx                  3.1                pyhd8ed1ab_0    conda-forge
ninja                     1.11.1               h924138e_0    conda-forge
nksr                      1.0.3+pt20cu118          pypi_0    pypi
nsight-compute            2022.3.0.22                   0    nvidia/label/cuda-11.8.0
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.24.3          py310hd5efca6_0  
numpy-base                1.24.3          py310h8e6c178_0  
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
omegaconf                 2.3.0              pyhd8ed1ab_0    conda-forge
open3d                    0.16.1+c65c7ef           pypi_0    pypi
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.2           py310h7cbd5c2_0    conda-forge
parameterized             0.9.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pathlib2                  2.3.7.post1     py310hff52083_2    conda-forge
pathtools                 0.1.2                      py_1    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.5.0           py310h582fbeb_1    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              3.8.0              pyhd8ed1ab_0    conda-forge
plotly                    5.15.0                   pypi_0    pypi
ply                       3.11                       py_1    conda-forge
plyfile                   0.9                      pypi_0    pypi
pooch                     1.7.0              pyha770c72_3    conda-forge
prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
protobuf                  3.19.6          py310heca2aa9_0    conda-forge
psutil                    5.9.5           py310h1fa729e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio-client         16.1                 hb77b528_4    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pybind11                  2.10.4          py310hdf3cbec_0    conda-forge
pybind11-global           2.10.4          py310hdf3cbec_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyg                       2.3.0           py310_torch_2.0.0_cu118    pyg
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pyjwt                     2.7.0              pyhd8ed1ab_0    conda-forge
pykdtree                  1.3.7.post0              pypi_0    pypi
pyntcloud                 0.3.1              pyhd8ed1ab_0    conda-forge
pynvml                    11.5.0                   pypi_0    pypi
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.0              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7          py310hab646b1_3    conda-forge
pyqt5-sip                 12.11.0         py310heca2aa9_3    conda-forge
pyquaternion              0.9.9                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-pycg               0.5.2                    pypi_0    pypi
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytorch                   2.0.0           py3.10_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h7e8668a_5    pytorch
pytorch-lightning         1.9.4              pyhd8ed1ab_1    conda-forge
pytorch-mutex             1.0                        cuda    pytorch
pytorch-scatter           2.1.1           py310_torch_2.0.0_cu118    pyg
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
pyzmq                     25.1.0                   pypi_0    pypi
qt-main                   5.15.8              h01ceb2d_12    conda-forge
randomname                0.2.1                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
retrying                  1.3.4                    pypi_0    pypi
rhash                     1.4.3                h166bdaf_0    conda-forge
rich                      13.4.2             pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scikit-learn              1.2.2           py310hf7d194e_2    conda-forge
scipy                     1.10.1          py310hd5efca6_0  
screeninfo                0.8.1                    pypi_0    pypi
sentry-sdk                1.21.1             pyhd8ed1ab_0    conda-forge
setproctitle              1.3.2           py310h5764c6d_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.9           py310hc6cd4ac_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
sysroot_linux-64          2.12                he073ed8_15    conda-forge
tbb                       2021.9.0             hf52228f_0    conda-forge
tenacity                  8.2.2                    pypi_0    pypi
tensorboard               2.11.2             pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.6.1           py310h600f1e7_4    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
termcolor                 2.3.0                    pypi_0    pypi
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
torchmetrics              0.11.4             pyhd8ed1ab_0    conda-forge
torchtriton               2.0.0                     py310    pytorch
tornado                   6.3.2           py310h2372a71_0    conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
trimesh                   3.22.1             pyhd8ed1ab_0    conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unicodedata2              15.0.0          py310h5764c6d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
usd-core                  23.5                     pypi_0    pypi
wandb                     0.15.4             pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.3                    pypi_0    pypi
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.7                    pypi_0    pypi
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.39                 hd590300_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.2           py310h2372a71_0    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge

output of train.py

1 Global seed set to 0
   2 /user/rsulzer/home/.conda/envs/nksr/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:478: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
   3   rank_zero_deprecation(
   4 /user/rsulzer/home/.conda/envs/nksr/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
   5   rank_zero_deprecation(
   6 Auto select gpus: [0]
   7 GPU available: True (cuda), used: True
   8 TPU available: False, using: 0 TPU cores
   9 IPU available: False, using: 0 IPUs
  10 HPU available: False, using: 0 HPUs
  11 You are using a CUDA device ('NVIDIA RTX A6000') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
  12  >>>> ======= MODEL HYPER-PARAMETERS ======= <<<<
  13 exec: null
  14 include: null
  15 visualize: false
  16 test_set_shuffle: false
  17 no_mesh_vis: false
  18 solver_verbose: false
  19 runtime_density: false
  20 runtime_visualize: false
  21 test_print_metrics: false
  22 test_n_upsample: 2
  23 test_use_gt_structure: false
  24 test_transform: null
  25 url: ''
  26 name: shapenet/scan_3k
  27 model: nksr_net
  28 feature: none
  29 geometry: kernel
  30 voxel_size: 0.02
  31 kernel_dim: 16
  32 tree_depth: 4
  33 adaptive_depth: 1
  34 unet:
  35   f_maps: 32
  36 udf:
  37   enabled: false
  38 interpolator:
  39   n_hidden: 2
  40   hidden_dim: 32
  41 solver:
  42   pos_weight: 10000.0
  43   normal_weight: 10000.0
  44 batch_size: 1
  45 accumulate_grad_batches: 4
  46 optimizer: Adam
  47 learning_rate:
  48   init: 0.0001
  49   decay_mult: 0.7
  50   decay_step: 50000
  51   clip: 1.0e-06
  52 weight_decay: 0.0
  53 grad_clip: 0.5
  54 adaptive_policy:
  55   method: normal
  56   tau: 0.1
  57 supervision:
  58   structure_weight: 20.0
  59   gt_type: PointTSDFVolume
  60   gt_surface:
  61     value: 200.0
  62     normal: 100.0
  63     subsample: 50000
  64   spatial:
  65     weight: 300.0
  66     reg_sdf_weight: 0.0
  67     samplers:
  68     - type: uniform
  69       n_samples: 50000
  70       expand: 1
  71       expand_top: 3
  72     - type: band
  73       n_samples: 50000
  74       eps: 0.5
  75     gt_type: l1
  76     gt_soft: true
  77     gt_band: 1.0
  78     pd_transform: true
  79     vol_sup: true
  80   udf:
  81     weight: 150.0
  82     samplers:
  83     - type: uniform
  84       n_samples: 80000
  85       expand: 1
  86       expand_top: 5
  87     - type: band
  88       n_samples: 20000
  89       eps: 0.5
  90 structure_schedule:
  91   start_step: 2500
  92   end_step: 10000
  93 _shapenet_path: /data/rsulzer/ShapeNet
  94 _shapenet_categories:
  95 - '02691156'
  96 - '02828884'
  97 - '02933112'
  98 - '02958343'
  99 - '03211117'
 100 - '03001627'
 101 - '03636649'
 102 - '03691459'
 103 - '04090263'
 104 - '04256520'
 105 - '04379243'
 106 - '04401088'
 107 - '04530566'
 108 _shapenet_custom_name: snet-3k-scan
 109 train_dataset: ShapeNetDataset
 110 train_val_num_workers: 4
 111 train_kwargs:
 112   onet_base_path: /data/rsulzer/ShapeNet
 113   categories:
 114   - '02691156'
 115   - '02828884'
 116   - '02933112'
 117   - '02958343'
 118   - '03211117'
 119   - '03001627'
 120   - '03636649'
 121   - '03691459'
 122   - '04090263'
 123   - '04256520'
 124   - '04379243'
 125   - '04401088'
 126   - '04530566'
 127   transforms: null
 128   custom_name: snet-3k-scan
 129   split: train
 130   random_seed: 0
 131 val_dataset: ShapeNetDataset
 132 val_kwargs:
 133   onet_base_path: /data/rsulzer/ShapeNet
 134   categories:
 135   - '02691156'
 136   - '02828884'
 137   - '02933112'
 138   - '02958343'
 139   - '03211117'
 140   - '03001627'
 141   - '03636649'
 142   - '03691459'
 143   - '04090263'
 144   - '04256520'
 145   - '04379243'
 146   - '04401088'
 147   - '04530566'
 148   transforms: null
 149   custom_name: snet-3k-scan
 150   split: val
 151   random_seed: fixed
 152 test_dataset: ShapeNetDataset
 153 test_num_workers: 4
 154 test_kwargs:
 155   onet_base_path: /data/rsulzer/ShapeNet
 156   categories:
 157   - '02691156'
 158   - '02828884'
 159   - '02933112'
 160   - '02958343'
 161   - '03211117'
 162   - '03001627'
 163   - '03636649'
 164   - '03691459'
 165   - '04090263'
 166   - '04256520'
 167   - '04379243'
 168   - '04401088'
 169   - '04530566'
 170   transforms: null
 171   custom_name: snet-3k-scan
 172   split: test
 173   random_seed: fixed
 174 _shapenet_transforms: null
 175  >>>> ====================================== <<<<
 176 Sanity Checking DataLoader 0:   0%|                                                                              | 0/2 [00:00<?, ?it/s]
 177 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
 178   | Name    | Type        | Params
 179 ----------------------------------------
 180 0 | network | NKSRNetwork | 12.0 M
 181 ----------------------------------------
 182 12.0 M    Trainable params
 183 0         Non-trainable params
 184 12.0 M    Total params
 185 48.113    Total estimated model params size (MB)
 186 Epoch 0:   0%|                                                                                               | 0/35032 [00:00<?, ?it/s]
 ...
 Segmentation fault