Allow (require) execution space for 1st arg of VerifyExecutionCanAccessMemorySpace

[mhoemmen]

This is a copy of #178 , that got closed without being finished. We moved trilinos/Trilinos#1697 from Trilinos to Kokkos today.

[crtrott]

And yes I agree with you this has been bugging me out of my mind for the last three years. Not sure why I didn't do anything about it. Also there should be a public class for this. How about just designing a new one:

template<class ExecSpace, class MemSpace>
SpaceAccessibility {
   enum readwrite;   // Can ExecSpace read and write to MemSpace
   enum write;       // Can ExecSpace write to MemSpace
   enum read;        // Can ExecSpace read from MemSpace
   enum allocatable; // Can ExecSpace allocate in MemSpace
   enum default;     // Is MemSpace the default memory space of ExecSpace
   enum fast;        // Is MemSpace the preferred fast persistent memory of ExecSpace
   enum capacity;    // Is MemSpace the preferred capacity memory of ExecSpace
};

For Example KNL with HBWSpace enabled:

SpaceAccessibility<OpenMP,HostSpace> {
   enum readwrite = 1;
   enum write = 1;
   enum read = 1;
   enum allocatable = 0; // Might be a 1 if we make our allocator threadsafe; 
   enum default = 0; 
   enum fast = 0;
   enum capacity 1 ;
};
SpaceAccessibility<OpenMP,HBWSpace> {
   enum readwrite = 1;
   enum write = 1;
   enum read = 1;
   enum allocatable = 0; // Might be a 1 if we make our allocator threadsafe; 
   enum default = 1; 
   enum fast = 1;
   enum capacity = 0;
};

Note it is not true that at least one of fast or capacity must be true. For CUDA we have CudaSpace, UVMSpace and HostPinnedSpace where CudaSpace would be fast, HostPinned capacity and UVM neither.

[mhoemmen]

Alas, default is a C++11 keyword. Not sure what would be better there.

I vote for "native" or "local" instead of "fast." Fast could mean different things (latency? bandwidth? good atomics but slow non-atomic access? accelerated vector scatter / gathers but slow scalar access?) depending on context.

Does "allocatable" mean "can allocate memory from that memory space inside a parallel execution region from that execution space"?

[hcedwar]

There is the currently undocumented public meta-function-class
Kokkos::SpaceAccessibility

template< typename AccessSpace , typename MemorySpace >
struct SpaceAccessibility {
  enum { accessible };  // AccessSpace can access MemorySpace
  enum { assignable };  // Can assign View<...,AccessSpace,...> = View<...,MemorySpace,...>
  enum { deep_copy };  // Can deep copy to AccessSpace::memory_space from MemorySpace
};

[mhoemmen]

That's an improvement, yes. It's still annoying to me that the interface requires allowing AccessSpace to be either a memory space or an execution space (otherwise assignable and deep_copy wouldn't work). However, this should at least allow AccessSpace to be an execution space, and the name isn't so wrong ;-) . Thanks!

[mhoemmen]

@hcedwar May we consider SpaceAccessibility part of the public interface of Kokkos?

[hcedwar]

@mhoemmen Yep - it is in the Kokkos namespace.

Holding this issue open until wiki user guide is updated.