Provide stream-aware containers

[upsj]

device_ptr and device_reference use the stream provided by the CUDA system (probably the default stream, though I didn't check). That means that there is no (to me obvious) way to use them safely with the par_nosync execution policy or in other non-default stream contexts. They are a really convenient tool for copying single values between host and device, so it would be really helpful if we could find a way to make them stream-aware, or provide a stream-aware alternative.

This issue came up in rapidsai/cudf#11919 (comment)

[jrhemstad]

it would be really helpful if we could find a way to make them stream-aware, or provide a stream-aware alternative.

I don't think it will be possible/wise to make them stream aware in a safe and easy way. These entities try and emulate a normal pointer and reference where there is no opportunity to inject a stream. For example device_vector::operator[] can't take a stream. Or device_ptr's deference operator can't take a stream.

While there is no arguing that they are convenient, their convenience can be a tremendous footgun for both stream ordering and hiding host/device copies. This is why RMM's device_uvector doesn't attempt to provide the device_ptr/device_reference functionality and instead opts for explicit element-wise functions that accept a stream.

There is certainly room for improvement with stream-ordered data structures in Thrust and beyond. This is something that @griwes will be thinking about and working on in the medium-term future.

[lilohuang]

No need to make device_vector::operator[] to take a stream. The device_vector should provide a constructor with stream-aware (or execution policy) like device_uvector, and then device_ptr or its base class should hold the associated stream to make data access to be stream-aware.

[jrhemstad]

I'll keep this issue open for now, but it's likely that we'll solve this problem in libcu++ and not Thrust.