Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault when running demo_ivfpq_indexing_gpu #8

Closed
Quasimondo opened this issue Mar 2, 2017 · 17 comments
Closed

Segmentation fault when running demo_ivfpq_indexing_gpu #8

Quasimondo opened this issue Mar 2, 2017 · 17 comments
Labels
bug

Comments

@Quasimondo
Copy link

After compiling both the CPU and the GPU version, the CPU test completed successfully, but the GPU test fails with a Segmentation fault when "Adding the vectors to the index". Unfortunately the error message is not very verbose:

[0.562 s] Generating 100000 vectors in 128D for training
[0.707 s] Training the index
Training IVF quantizer on 100000 vectors in 128D
Clustering 100000 points in 128D to 1788 clusters, redo 1 times, 10 iterations
Preprocessing in 0.00984204 s
Iteration 9 (0.57 s, search 0.38 s): objective=930934 imbalance=1.255 nsplit=0
computing residuals
training 4 x 256 product quantizer on 16384 vectors in 128D
Training PQ slice 0/4
Clustering 16384 points in 32D to 256 clusters, redo 1 times, 25 iterations
Preprocessing in 0.000524902 s
Iteration 24 (2.06 s, search 1.68 s): objective=27271.5 imbalance=1.018 nsplit=0
Training PQ slice 1/4
Clustering 16384 points in 32D to 256 clusters, redo 1 times, 25 iterations
Preprocessing in 0.000452148 s
Iteration 24 (1.76 s, search 1.41 s): objective=27193.4 imbalance=1.016 nsplit=0
Training PQ slice 2/4
Clustering 16384 points in 32D to 256 clusters, redo 1 times, 25 iterations
Preprocessing in 0.000414062 s
Iteration 24 (1.54 s, search 1.23 s): objective=27230.8 imbalance=1.021 nsplit=0
Training PQ slice 3/4
Clustering 16384 points in 32D to 256 clusters, redo 1 times, 25 iterations
Preprocessing in 0.000456055 s
Iteration 24 (1.73 s, search 1.39 s): objective=27174 imbalance=1.023 nsplit=0
[8.535 s] storing the pre-trained index to /tmp/index_trained.faissindex
[8.569 s] Building a dataset of 200000 vectors to index
[8.813 s] Adding the vectors to the index
Segmentation fault (core dumped)

I've got a Ubuntu 16.04 system with 2 GeForce GTX 970, both have 4Gb of memory. Any idea what I am doing wrong?
@mdouze
Copy link
Contributor

mdouze commented Mar 2, 2017

Hi,
Could you give the output of

ldd gpu/test/demo_ivfpq_indexing_gpu

Thanks

@Quasimondo
Copy link
Author

Here you are:

ldd gpu/test/demo_ivfpq_indexing_gpu
linux-vdso.so.1 => (0x00007fffadfe4000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007ff89b6cb000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ff898c32000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff898a2a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff89880d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff898608000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff898280000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff897f77000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ff897d48000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff897b31000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff897768000)
/lib64/ld-linux-x86-64.so.2 (0x000055d1e847a000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ff897436000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ff8971f6000)

@mdouze
Copy link
Contributor

mdouze commented Mar 2, 2017

Looks correct. Could you do a backtrace as well?

gdb gpu/test/demo_ivfpq_indexing_gpu

then
r
bt

thanks

@Quasimondo
Copy link
Author

unfortunately that command triggers a "ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted." - I might have to recompile gdb from source first.

@mdouze
Copy link
Contributor

mdouze commented Mar 2, 2017

You may want to move ~/.gdbinit aside and run /usr/bin/gdb directly to make sure it is not running some exotic startup scripts.

@Quasimondo
Copy link
Author

Clearing my PYTHONPATH variable fixed it - looks like Python 2 and 3 are using the same.

Okay, here is the debug trace (I left out the parts where no error occured):

[9.567 s] Adding the vectors to the index

Thread 1 "demo_ivfpq_inde" received signal SIGSEGV, Segmentation fault.
0x00007fffd5806c9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
(gdb) bt
#0 0x00007fffd5806c9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007fffd572d696 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007fffd5862992 in cuEventDestroy_v2 () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00000000004f63d4 in cudart::cudaApiEventDestroy(CUevent_st*) ()
#4 0x0000000000524c74 in cudaEventDestroy ()
#5 0x0000000000440bd8 in faiss::gpu::streamWaitBase<std::vector<CUstream_st*, std::allocator<CUstream_st*> >, std::initializer_list<CUstream_st*> > (
listWaiting=std::vector of length 2, capacity 2 = {...}, listWaitOn=...)
at impl/../utils/DeviceUtils.h:131
#6 0x0000000000479374 in faiss::gpu::streamWait<std::vector<CUstream_st*, std::allocator<CUstream_st*> > > (b=..., a=std::vector of length 2, capacity 2 = {...})
at impl/../utils/DeviceUtils.h:140
#7 faiss::gpu::runL2Distance (resources=0x7fffffffd380, centroids=...,
centroidNorms=centroidNorms@entry=0xbcfa4b0, queries=..., k=k@entry=1,
outDistances=..., outIndices=..., ignoreOutDistances=true, tileSize=256)
at impl/Distance.cu:110
#8 0x000000000047040e in faiss::gpu::runL2Distance (resources=,
vectors=..., vectorNorms=vectorNorms@entry=0xbcfa4b0, queries=..., k=k@entry=1,
outDistances=..., outIndices=..., ignoreOutDistances=, tileSize=-1)
at impl/Distance.cu:307
#9 0x000000000042e654 in faiss::gpu::FlatIndex::query (this=0xbcfa3f0, vecs=...,
k=k@entry=1, outDistances=..., outIndices=...,
exactDistance=exactDistance@entry=false, tileSize=-1) at impl/FlatIndex.cu:121
#10 0x00000000004433a3 in faiss::gpu::IVFPQ::classifyAndAddVectors (this=0xbd0c920,
vecs=..., indices=...) at impl/IVFPQ.cu:138
#11 0x00000000004250b0 in faiss::gpu::GpuIndexIVFPQ::add_with_ids (this=0x7fffffffd4d0,
n=200000, x=0x7fffb7578010, xids=0xbdd6250) at GpuIndexIVFPQ.cu:355
#12 0x0000000000420a0b in faiss::gpu::GpuIndexIVF::add (this=0x7fffffffd4d0, n=200000,
x=0x7fffb7578010) at GpuIndexIVF.cu:254
#13 0x000000000040e83f in main ()

@mdouze mdouze added the bug label Mar 2, 2017
@wickedfoo
Copy link
Contributor

Hi,

Can you run cuda-memcheck demo_ivfpq_indexing_gpu?

It will print out a lot of this, which is expected (basically it pops up when we test a CPU pointer to see what address space it lives in):

========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x126239]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x16e44]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
=========     Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib64/nvidia/libcuda.so.1 [0x2eea03]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x11de53]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x16e65]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d066]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1d1e2]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x1889f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x194e5]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xb504c]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x2332f]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0x260d0]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf8cb]
=========     Host Frame:/lib64/libc.so.6 (__libc_start_main + 0xf5) [0x21b35]
=========     Host Frame:test/demo_ivfpq_indexing_gpu [0xf415]

but I'm interested in anything that is not that.

I attempted and failed to repro locally by restricting memory on my GPU to 4 GB, but didn't get anywhere near the limit; this shouldn't be using that much memory anyways. Internally I've only run this on K40, Maxwell Titan X, Maxwell M40 and Pascal P100, so maybe there's something weird with GeForce GTX 970.

@mdouze mdouze mentioned this issue Mar 10, 2017
Closed
@danielhauagge
Copy link

danielhauagge commented Mar 14, 2017 •
edited
Loading

I'm running into a similar issue with a similar setup: Ubuntu 16.04, single GTX 970, nvidia driver version 375.26, cuda 8. I'm able to run the code with nb = 3590 but if I go to nb = 3600 it crashes.

Here are the last few lines of the output from cuda-memcheck

=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x120749]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x15de4]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2565c]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x15e1d]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2565c]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaPointerGetAttributes. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x120749]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x15de4]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x23a6a]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x15e1d]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x23a6a]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaMemcpyAsync. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x12cf33]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x25754]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaLaunch. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x11ac3e]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x54e63]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x77058]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaGetLastError. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x118363]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x54a3d]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x77058]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaEventCreateWithFlags. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x1274d9]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x43e71]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaEventRecord. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x129082]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x43e7e]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaStreamWaitEvent. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x128ea5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x43ece]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Program hit cudaErrorIllegalAddress (error 77) due to "an illegal memory access was encountered" on CUDA API call to cudaStreamWaitEvent. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2ef503]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x128ea5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x43ece]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x7718f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2f844]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x46ba3]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2585b]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0x2078f]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xde3b]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf5) [0x21ec5]
=========     Host Frame:./test/demo_ivfpq_indexing_gpu [0xd7b5]
=========
========= Error: process didn't terminate successfully
=========        The application may have hit an error when dereferencing Unified Memory from the host. Please rerun the application under cuda-gdb or Nsight Eclipse Edition to catch host side errors.
========= Internal error (20)
========= No CUDA-MEMCHECK results found

@danielhauagge
Copy link

danielhauagge commented Mar 14, 2017 •
edited
Loading

Continuing on the prev post. The output of some of the other commands @mdouze mentioned

  • ldd
	linux-vdso.so.1 =>  (0x00007fff905f7000)
	libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f63aa9c0000)
	libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f63a7f28000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f63a7d1f000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63a7b01000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63a78fd000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f63a75ed000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63a72e7000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f63a70d0000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f63a6eb8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63a6af4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f63ac78a000)
	libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f63a67cb000)
	libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f63a658c000)
  • gdb backtrace
(gdb) bt
#0  0x00007fffd5b9ec9a in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1  0x00007fffd5ac5696 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007fffd5bfa992 in cuEventDestroy_v2 () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00000000004f8614 in cudart::cudaApiEventDestroy(CUevent_st*) ()
#4  0x0000000000526eb4 in cudaEventDestroy ()
#5  0x0000000000443ef8 in faiss::gpu::streamWaitBase<std::vector<CUstream_st*, std::allocator<CUstream_st*> >, std::initializer_list<CUstream_st*> > (
    listWaiting=std::vector of length 2, capacity 2 = {...}, listWaitOn=...) at impl/../utils/DeviceUtils.h:131
#6  0x000000000047718f in streamWait<std::vector<CUstream_st*> > (b=..., a=std::vector of length 2, capacity 2 = {...}) at impl/../utils/DeviceUtils.h:140
#7  faiss::gpu::runL2Distance<float> (resources=0x7fffffffe040, centroids=..., centroidNorms=centroidNorms@entry=0xbe32530, queries=..., k=k@entry=1, outDistances=..., outIndices=..., 
    ignoreOutDistances=true, tileSize=256) at impl/Distance.cu:110
#8  0x0000000000473a4e in faiss::gpu::runL2Distance (resources=<optimized out>, vectors=..., vectorNorms=vectorNorms@entry=0xbe32530, queries=..., k=k@entry=1, outDistances=..., outIndices=..., 
    ignoreOutDistances=<optimized out>, tileSize=256) at impl/Distance.cu:307
#9  0x000000000042f844 in faiss::gpu::FlatIndex::query (this=0xbe32470, vecs=..., k=k@entry=1, outDistances=..., outIndices=..., exactDistance=exactDistance@entry=false, tileSize=-1)
    at impl/FlatIndex.cu:121
#10 0x0000000000446ba3 in faiss::gpu::IVFPQ::classifyAndAddVectors (this=0xbe34a70, vecs=..., indices=...) at impl/IVFPQ.cu:138
#11 0x000000000042585b in faiss::gpu::GpuIndexIVFPQ::add_with_ids (this=0x7fffffffdfb0, n=3600, x=<optimized out>, xids=<optimized out>) at GpuIndexIVFPQ.cu:355
#12 0x000000000042078f in faiss::gpu::GpuIndexIVF::add (this=0x7fffffffdfb0, n=3600, x=0xc422670) at GpuIndexIVF.cu:254
#13 0x000000000040de3b in main ()

@ck196
Copy link

ck196 commented Mar 18, 2017

I got same problem, one more problems when i install on AWS g2.x2large instance.

Faiss assertion prop.major > 3 || (prop.major == 3 && prop.minor >= 5) failed in virtual void faiss::gpu::StandardGpuResources::initializeForDevice(int) at StandardGpuResources.cpp:122Aborted (core dumped)

@mdouze
Copy link
Contributor

mdouze commented Mar 18, 2017

Hi
@ck196, It seems that the GPU just too old to run Faiss. It should have compute capbality >= 3.5

@ck196
Copy link

ck196 commented Mar 21, 2017

@mdouze thank you, I got it. With newer GPU, I got them same problem as @Quasimondo and @danielhauagge.

@wickedfoo
Copy link
Contributor

@ck196 Can you repro this on an AWS instance with a newer GPU? If so, what type of instance (P2?)

@ck196
Copy link

ck196 commented Mar 21, 2017 •
edited
Loading

@wickedfoo No, I gave up on AWS instances, I tried newer GPU on my personal computer.
I tried P2 instance, but it was an old GPU (<3.5)

@tumb1er
Copy link

tumb1er commented Mar 21, 2017

nvidia Quadro K1200 - same segfault.

@mdouze
Copy link
Contributor

mdouze commented Mar 29, 2017

Closing issue now. If similar problems arise, please open a new issue.

@mdouze mdouze closed this as completed Mar 29, 2017
@Geek0x0 Geek0x0 mentioned this issue Apr 5, 2017
Closed
@luisremis luisremis mentioned this issue Dec 12, 2017
Closed
This was referenced May 9, 2018
Closed
Closed
@q423462798 q423462798 mentioned this issue May 30, 2018
Closed
@ZhuoranLyu ZhuoranLyu mentioned this issue Jun 7, 2018
Closed
1 task
@GitHubProgress3 GitHubProgress3 mentioned this issue Sep 19, 2018
Closed
@fenglonz fenglonz mentioned this issue Dec 3, 2019
Closed
@zhangqiang4002 zhangqiang4002 mentioned this issue Feb 2, 2021
Closed
mqnfred pushed a commit to mqnfred/faiss that referenced this issue Oct 23, 2023
@Enet4
Merge pull request facebookresearch#8 from gensmusic/upgrade-faiss
30da2c4 
upgrade faiss facebookresearch#7
@cduk
Copy link

cduk commented Nov 25, 2024

@danielhauagge
I know this is old, but you should be aware that this GPU has a strange VRAM set-up. Only the first 3.5GB of VRAM are connected normally, the remaining 0.5GB is very slow. This might explain why it works only to 3.5GB.

See here:
https://www.reddit.com/r/buildapc/comments/3poy93/was_the_4gb_vram_issues_on_the_gtx_970_ever_fixed/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@Quasimondo @ck196 @danielhauagge @tumb1er @wickedfoo @cduk @mdouze