Scalability of MPI_Win_allocate_shared

[ashwinraghu]

A field report noted the execution time of MPI_Win_allocate_shared() increasing with the amount of memory being allocated. e.g. on a single node with 4 ranks, it takes 3 secs for 8GB, 6 secs for 16GB and 12 secs for 32GB.
I did some instrumentation of the 3.4.x implementation using callgrind. The resulting report shows the implementation spending a good part of the time executing instructions in the system call msync() called in turn from
MPIDIU_get_shm_symheap()->generate_random_addr().->check_maprange_ok()

(Column 1 is the number of instructions)
520,123,460 (96.79%) < ???:MPIDIU_get_shm_symheap (1x)
251,658,345 (46.83%) * ???:generate_random_addr
218,103,795 (40.59%) > ???:msync (16,777,215x)
50,331,648 ( 9.37%) > ???:__errno_location (16,777,216x)

Based on my reading of check_maprange_ok(), it's trying to detect a virtual address range of a given size that is free/unmapped.
The base address of this range is then bcast'd and used to map the shared memory across all processes.

The question: what is the motive for trying to ensure that the base address is identical across all processes, given that any access via load and store operations on the shared memory is to be done only after fetching the base address via MPI_Win_shared_query()?
It's also apparent that the fall-back method uses any available address returned by mmap() anyway.

[hzhou]

Symmetrical allocation affords some optimizations, for example, skip the base address lookup and translations in RMA. I believe this is also needed to support OpenSHM's symmetric heap (https://pmodels.github.io/oshmpi-www/). In either case, it's an optional optimization. It looks like the current code always tries symmetrical heap allocation before fallback. As this can be costly per your report, we should add a configure option or CVAR to allow users to disable it.

[hzhou]

skip the base address lookup and translations in RMA

If you mean TLB lookups, aren't they unique to process context?

No, not TLB lookups. If it is symmetrical, then we can skip the base address lookup and translate the actual virtual address.

[ashwinraghu]

On re-reading the specification, I understand that the need/relevance of a symmetric allocation starting at the same virtual address across all processes is mentioned under MPi_Win_allocate(). On the other hand, for MPI_Win_allocate_shared(), what seems more important is that the shared memory is contiguous across ranks.
A process will load/store from/into another process's memory using it's own virtual address translations and I don't see why the base address has to be the same across ranks for this. Contiguity of the memory on the other hand is key.

Secondly, there doesn't seem to be any easy means for the application to know if MPI_Win_allocate_shared() was able to allocate the shared memory and attach at the same base address on all ranks.

Finally, the first step in win_allocate_shared() of checking the address range and the subsequent steps of creating and attaching the memory using mmap() don't seem "atomic". It's quite possible that a second thread for instance could use up virtual address ranges in between. Makes me wonder even more of the real value/efficacy of check_map_range() then.

[hzhou]

I agree that MPI_Win_allocate_shared has no point for symmetric allocations.

[ashwinraghu]

I believe there is a case for making MPI_Win_allocated_shared lighter. Is an "issue" in order?

[hzhou]

Yes :)