Moving the memory space detection outside the Engine classes for the GPU-aware logic

[anagainaru]

The BP5, SST and DataMan engines are GPU aware which means they can receive GPU pointers as well as Kokkos::Views.

For Kokkos::View the memory space is updated based on the type of the View. For raw pointers, the memory space needs to be detected. There is currently no consistent way of detecting the memory space across the three engines and if not handled correctly the engine might propagate a Detect memory space until it's time to copy the memory to our internal buffers (which defaults to Host and a seg fault if the pointer was actually a device pointer). To overcome this problem, the safe way is for the application to call SetMemorySpace to Device or Host for all raw pointers. SST and DataMan are not exhaustively tested so there might be scenarios for them where the memory is not detected.

This issue is an action call to move the detection of the memory space before we reach each particular engine so all behaviors are consistent and the BP5 tests cover SST and DataMan as well.

[eisenhauer]

There are several things that are currently done in at least partially duplicative code across all engines (MemorySelection comes to mind in addition to memory space detection). Each of these is kind of funky and it may be difficult to rationalize this everywhere. However, we might do a better job on testing. Part of the problem is that the tests in engines/bp aren’t structured so that they can be used with streaming engines. However, PR #3823 demonstrated an approach whereby an engines/bp tests could be restructured slightly in order to run it in testing/adios/engines/staging-common so that it can be run against staging engines too. Whatever we do with unifying memory space detection, we could also move towards running the tests we have against all the engines we have.

[anagainaru]

I like your solution to use MPI_Comm_split to move from BP to SST testing. I will try to update the device testing to include something similar.

I would still like to trigger the detection mechanism for memory space as early as possible. There are so many paths that engines take to move the data, I am starting to regret the decision to allow Puts/Gets on different memory space for the same variable. I am thinking of forcing just one (once you send device data to a variable, all data needs to be on the device) since I am having trouble transposing on the other side data if you don't read it on the same space.