[AIR] Address UserWarning from Torch training

[bveeramani]

Why are these changes needed?

Related issue number

Fixes #28003

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[clarkzinzow]

I don't think that copying the ndarray here is the right solution, since we want to be able to reuse the shared memory tensor data buffers without incurring unnecessary copies. We're also under the expectation the user/model will not mutate these input tensors during training and inference, so I don't think that the copy is necessary for correctness.

What if we suppressed the warning instead via something like:

Suggested change

	# NOTE: PyTorch raises a `UserWarning` if `ndarray` isn't writeable. See #28003.
	if not ndarray.flags["WRITEABLE"]:
	ndarray = np.copy(ndarray)
	return torch.as_tensor(ndarray, dtype=dtype, device=device)
	with warnings.catch_warnings():
	warnings.simplefilter("ignore")
	tensor = torch.as_tensor(ndarray, dtype=dtype, device=device)
	return tensor

[bveeramani]

Are we confident that the warning is harmless? If so, I think this is fine/

[clarkzinzow]

If the user tries to mutate the tensor (e.g. with in-place tensor operations) then this will be undefined behavior, but the common case is that this tensor data buffer will be left untouched in Plasma until we either (a) transfer the tensor to the GPU, or (b) we create a new tensor via an operation that creates a copy; in either case, we're not likely to mutate it.

[clarkzinzow]

I'm not sure if we should suppress the warning, though, since it's a useful signal to the user that they will need to .clone() it if they wish to mutate it.

[richardliaw]

It'd be great to suppress the repeated warnings (if that is somehow possible).

[bveeramani]

Closing this PR because I don't know how to suppress repeated warning with multiple workers