cuda.parallel: Should stream be a required parameter?

[shwina]

In #3348 (comment), it came up that we may want to consider making stream a required argument wherever a CUDA stream can be passed.

At the moment, stream is not a required parameter and defaults to None (translating to nullptr or "default stream" at the C++ layer). This aligns with underlying C++ APIs which also use stream=0 as a default (example).

However, as pointed out in the discussion above, users can forget to pass a stream in situations like the following:

with stream as s:
    arr = cp.random.random(...)

reduce_into(..., d_in=arr, ...)  # forgot to pass s here for whatever reason

This is seemingly a broader problem in the ecosystem (any API from any library accepting a CUDA stream has the same problem). So it would be good to hear perspectives from e.g., @vyasr as I know cuDF is also considering exposing streams in their APIs.

[vyasr]

I would support making streams a required argument in more places. Practically speaking I think that we will need to differentiate between low- and high-level libraries. For a library like cudf, for example, I don't think a user should ever have to specify a stream if they don't want to. Aside from the fact that it's meant to look and feel like pandas, it's also meant to be a high-level interface that lets users write simple code. To me, requiring streams smells like a power-user API. I am not opposed to that, but I think that makes more sense for lower-level tools. In my case, I could see the argument for having pylibcudf behave this way, but it's kind of on the edge. More likely than not we will at least attempt a model like rapidsai/cudf#17626 where a default stream choice is still made for the user and they don't have to specify a stream. For cuda.parallel (or more generally for the cuda.* libraries) though I think it would be reasonable to tip the other way. I should note that it may be a bit incongruous if the Python libraries do this while the underlying C++ libraries still have a default though. We may want to push to eventually synchronize every library at the same level in the stack to follow the same default stream behavior.