Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Unit tests failed: ai.rapids.cudf.CudaFatalException: cudaErrorLaunchFailure: unspecified launch failure #1745

Closed
NvTimLiu opened this issue Jan 28, 2024 · 2 comments · Fixed by #1746
Closed

[BUG] Unit tests failed: ai.rapids.cudf.CudaFatalException: cudaErrorLaunchFailure: unspecified launch failure #1745

NvTimLiu opened this issue Jan 28, 2024 · 2 comments · Fixed by #1746
Assignees
@thirtiseven
Labels
bug Something isn't working test

Comments

@NvTimLiu
Copy link
Collaborator

Describe the bug
On branch-24.02, Below unit tests failed on: ai.rapids.cudf.CudaFatalException: cudaErrorLaunchFailure: unspecified launch failure

parseURISparkTest : exclusive_scan failed to synchronize

parseURIUTF8Test: std::bad_alloc: CUDA error

parseURIIP4Test: std::bad_alloc: CUDA error

parseURIIP6Test : std::bad_alloc: CUDA error

Detailed log:

ai.rapids.cudf.CudaFatalException: exclusive_scan failed to synchronize: cudaErrorLaunchFailure: unspecified launch failure
	at com.nvidia.spark.rapids.jni.ParseURI.parseQueryWithColumn(Native Method)
	at com.nvidia.spark.rapids.jni.ParseURI.parseURIQueryWithColumn(ParseURI.java:85)
	at com.nvidia.spark.rapids.jni.ParseURITest.testQuery(ParseURITest.java:157)
	at com.nvidia.spark.rapids.jni.ParseURITest.parseURISparkTest(ParseURITest.java:286)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at e.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at e.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at e.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at e.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:4
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:4
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.junit.platform.surefire.provider.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:155)
	at org.junit.platform.surefire.provider.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:134)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)

[ERROR] parseURIUTF8Test  Time elapsed: 0.01 s  <<< ERROR!
ai.rapids.cudf.CudaFatalException: std::bad_alloc: CUDA error at: /home/jenkins/agent/workspace/jenkins-spark-rapids--232-cuda12/target/libcudf-install/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorLaunchFailure unspecified 
	at ai.rapids.cudf.Rmm.allocInternal(Native Method)
	at ai.rapids.cudf.Rmm.alloc(Rmm.java:506)
	at ai.rapids.cudf.DeviceMemoryBuffer.allocate(DeviceMemoryBuffer.java:147)
	at ai.rapids.cudf.DeviceMemoryBuffer.allocate(DeviceMemoryBuffer.java:137)
	at ai.rapids.cudf.HostColumnVector.copyToDevice(HostColumnVector.java:240)
	at ai.rapids.cudf.ColumnVector.fromStrings(ColumnVector.java:1460)
	at com.nvidia.spark.rapids.jni.ParseURITest.testProtocol(ParseURITest.java:42)
	at com.nvidia.spark.rapids.jni.ParseURITest.parseURIUTF8Test(ParseURITest.java:297)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at e.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at e.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at e.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at e.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:4
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:4
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at ne.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.junit.platform.surefire.provider.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:155)
	at org.junit.platform.surefire.provider.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:134)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
@NvTimLiu NvTimLiu added bug Something isn't working ? - Needs Triage test labels Jan 28, 2024
@thirtiseven
Copy link
Collaborator

Seems related to #1740
Tests passed before this commit. The tests will only failed with -DUSE_SANITIZER=ON.
Investigating...

@thirtiseven thirtiseven self-assigned this Jan 29, 2024
@thirtiseven
Copy link
Collaborator

sanitizer log:

========= COMPUTE-SANITIZER
========= Invalid __global__ read of size 1 bytes
=========     at 0x11240 in spark_rapids_jni::detail::<unnamed>::parse_uri_char_counter(cudf::column_device_view, spark_rapids_jni::detail::URI_chunks, const char *, int *, int *, unsigned int *, thrust::optional<const cudf::column_device_view>)
=========     by thread (53,0,0) in block (0,0,0)
=========     Address 0x7f1216584df1 is out of bounds
=========     and is 1 bytes after the nearest allocation at 0x7f1216584600 of size 2033 bytes
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame: [0x2fa8f0]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x3472edb]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x34b2b28]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:__device_stub__ZN16spark_rapids_jni6detail45_GLOBAL__N__af96a273_12_parse_uri_cu_cf92446222parse_uri_char_counterEN4cudf18column_device_viewENS0_10URI_chunksEPKcPiS7_PjN6thrust8optionalIKS3_EE(cudf::column_device_view const&, spark_rapids_jni::detail::URI_chunks, char const*, int*, int*, unsigned int*, thrust::optional<cudf::column_device_view const>&) [0xd25256]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::detail::(anonymous namespace)::parse_uri_char_counter(cudf::column_device_view, spark_rapids_jni::detail::URI_chunks, char const*, int*, int*, unsigned int*, thrust::optional<cudf::column_device_view const>) [0xd25285]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::detail::parse_uri(cudf::strings_column_view const&, spark_rapids_jni::detail::URI_chunks, std::optional<cudf::strings_column_view const>, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd259ca]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8adb5997]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaStreamSynchronize_ptsz.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34afcf8]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::detail::parse_uri(cudf::strings_column_view const&, spark_rapids_jni::detail::URI_chunks, std::optional<cudf::strings_column_view const>, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2634b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::detail::parse_uri(cudf::strings_column_view const&, spark_rapids_jni::detail::URI_chunks, std::optional<cudf::strings_column_view const>, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd26352]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5f7a]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::device_buffer::~device_buffer() [0xc5f1f0]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5eb6]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::device_buffer::~device_buffer() [0xc5f1f0]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5ec2]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:std::unique_ptr<cudf::column, std::default_delete<cudf::column> >::~unique_ptr() [0xc5ffc9]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5ece]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:std::_Function_handler<void (cudf::column_device_view*), cudf::(anonymous namespace)::create_device_view_from_view<cudf::column_view, cudf::column_device_view>(cudf::column_view const&, rmm::cuda_stream_view)::{lambda(cudf::column_device_view*)#2}>::_M_invoke(std::_Any_data const&, cudf::column_device_view*&&) [0xf6ac19]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:std::unique_ptr<cudf::column_device_view, std::function<void (cudf::column_device_view*)> >::~unique_ptr() [0xcc15b7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5eda]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:std::_Function_handler<void (cudf::column_device_view*), cudf::(anonymous namespace)::create_device_view_from_view<cudf::column_view, cudf::column_device_view>(cudf::column_view const&, rmm::cuda_stream_view)::{lambda(cudf::column_device_view*)#2}>::_M_invoke(std::_Any_data const&, cudf::column_device_view*&&) [0xf6ac19]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:std::unique_ptr<cudf::column_device_view, std::function<void (cudf::column_device_view*)> >::~unique_ptr() [0xcc15b7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8a5ee6]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:spark_rapids_jni::parse_uri_to_query(cudf::strings_column_view const&, cudf::strings_column_view const&, rmm::cuda_stream_view, rmm::mr::device_memory_resource*) [0xd2aa0d]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_com_nvidia_spark_rapids_jni_ParseURI_parseQueryWithColumn [0xc76659]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x882490]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x882497]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaDeviceSynchronize.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34aa3a7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x8824a6]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_free [0xda1ab7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe8721704]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaMalloc.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b4b7b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc5845f]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc58485]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94733c]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaDeviceSynchronize.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34aa3a7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94734b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaMalloc.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b4b7b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc5845f]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc58485]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94733c]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaDeviceSynchronize.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34aa3a7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94734b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaMalloc.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b4b7b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc5845f]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:rmm::mr::cuda_memory_resource::do_allocate(unsigned long, rmm::cuda_stream_view) [0xc58485]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_allocInternal [0xda1844]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94733c]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaDeviceSynchronize.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34aa3a7]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0x94734b]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe870383d]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaFree.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34b52ae]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_initDefaultCudaDevice [0xda0ac3]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaGetLastError.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x481946]
=========                in /usr/lib64/libcuda.so.1
=========     Host Frame: [0x34acb34]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:cudf::jni::cuda_exception(JNIEnv_*, cudaError, _jthrowable*) [0xd424e5]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame:Java_ai_rapids_cudf_Rmm_initDefaultCudaDevice [0xda0af8]
=========                in /tmp/cudf2019081457469959978.so
=========     Host Frame: [0xffffffffe7d85b26]
=========                in 
========= 
========= Error: process didn't terminate successfully
========= Target application returned an error
========= ERROR SUMMARY: 34 errors

@thirtiseven thirtiseven mentioned this issue Jan 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thirtiseven thirtiseven

Labels
bug Something isn't working test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix a out of bound bug in parse_url query thirtiseven/spark-rapids-jni
3 participants
@sameerz @thirtiseven @NvTimLiu