[BUG] cubic_spline throws cudaErrorInvalidAddressSpace error

[drabastomek]

Describe the bug
A clear and concise description of what the bug is.

Steps/Code to reproduce bug
I tried to replicate the behavior from the docs

t = cudf.Series(np.repeat(cp.arange(100), 1000)).astype('float32')
y = cudf.Series(np.random.random(100*1000)).astype('float32')

prefix_sum = cudf.Series(cp.arange(1000)*100).astype('int32')

new_samples = cudf.Series(
    np.repeat(np.linspace(0, 100, 1000), 1000)
).astype('float32')

curve = cuspatial.CubicSpline(t, y, prefixes=prefix_sum)
new_points = curve(new_samples, prefix_sum*10)

Full stack:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-5745a70b247c> in <module>
      7 
      8 curve = cuspatial.CubicSpline(t, y, prefixes=prefix_sum)
----> 9 new_points = curve(new_samples, prefix_sum*10)
     10 

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuspatial/core/interpolate.py in __call__(self, coordinates, groups)
    175                 ).astype("int32")
    176             result = _cubic_spline_fit(
--> 177                 coordinates, self.groups, self.prefix, self.t, self.c
    178             )
    179             return Series(result)

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuspatial/core/interpolate.py in _cubic_spline_fit(points, points_ids, prefixes, original_t, c)
     28     original_t_c = original_t._column
     29     result_column = cubicspline_interpolate(
---> 30         points_c, points_ids_c, prefixes_c, original_t_c, c
     31     )
     32     return result_column

cuspatial/_lib/interpolate.pyx in cuspatial._lib.interpolate.cubicspline_interpolate()

cuspatial/_lib/interpolate.pyx in cuspatial._lib.interpolate.cubicspline_interpolate()

RuntimeError: for_each: failed to synchronize: cudaErrorInvalidAddressSpace: operation not supported on global/shared address space

Expected behavior
new_points contains interpolated points.

Environment details (please complete the following information):

• Environment location: Docker
• Method of cuSpatial install: Docker
  • Stable git pull from rapidsai/rapidsai, v 0.16, CUDA 11, Python 3.7.8

[drabastomek]

@zhangjianting I just tested in 0.17 and this call still throws cudaErrorInvalidAddressSpace. If I encounter this once, looks like the whole RMM context gets corrupted as any subsequent run of any cell throws the same error

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-3-7dae3700009f> in <module>
----> 1 ring, poly_idx, polygons = cuspatial.io.shapefile.read_polygon_shapefile('../data/gadm36_USA_1.shp')
      2 polygons = polygons.rename(columns={'x': 'longitude', 'y': 'latitude'})
      3 polygons['line'] = cp.random.randint(10,240, len(polygons))
      4 polygons = polygons.query(
      5         'longitude > -130'

/opt/conda/envs/rapids/lib/python3.8/site-packages/cuspatial/io/shapefile.py in read_polygon_shapefile(filename)
     31                 y-components of each polygon's points
     32     """
---> 33     result = cpp_read_polygon_shapefile(filename)
     34     f_pos = Series(result[0], name="f_pos")
     35     r_pos = Series(result[1], name="r_pos")

cuspatial/_lib/shapefile_reader.pyx in cuspatial._lib.shapefile_reader.read_polygon_shapefile()

cuspatial/_lib/shapefile_reader.pyx in cuspatial._lib.shapefile_reader.read_polygon_shapefile()

MemoryError: std::bad_alloc: CUDA error at: /opt/conda/envs/rapids/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorInvalidAddressSpace operation not supported on global/shared address space

In this example I read a Shape-file that, if I run it before I run the cubic_spline, reads the data without problems.

cc: @kkraus14 @harrism

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.