mmap shared memory transport #1086
Conversation
|
|
||
| /* Write all current contents of the data segment to the file */ | ||
| FILE *fp = fdopen(fd, "wb"); | ||
| size_t ret = fwrite(base, shm_size, 1, fp); |
There was a problem hiding this comment.
Can you share a reference to docs for this? As far as I've been able to track down, fdopen on a descriptor returned by shm_open is described as having undefined behavior.
There was a problem hiding this comment.
I can't find anything referencing this either, probably because it's a complete hack ;).
BTW, there's a good chance we won't merge this PR, at least as it is, for this very reason. Maybe it should just be a topic branch until the symmetric data segment is deprecated or something. The reason this WIP PR exists is because I'd like to evaluate some internal systems where there's no chance XPMEM will ever be installed, so we have to think out of the box.
Yes, manpages say fdopen on a shared memory object is undefined, but I think that's because there are many operations one may not be able to do on shared memory "files" (like seek, lock via fcntl, chmod, etc.) and possibly because there could be other processes writing to the shared region while the file is being opened. So it's usually a very bad idea to do this.
But I'm wondering if it may be relatively harmless to do this within SOS init on Linux, because we're not doing any fancy ops on the file (just writing to it and then truncating it) and no other PE or thread will write to the region until after shmem_init (but I think there's a risk of say some crazy interrupt or something leading to changes to the data segment before it's mmap'd).
I'm surprised this approach always seems to work on my end (based on the unit tests) - but I've only tried a few flavors of Linux. MacOS definitely doesn't like it at all, I get a failure on fwrite.
There was a problem hiding this comment.
Yeah, this is the only mention I could find: https://stackoverflow.com/questions/23347877/map-char-pointer-to-in-memory-file-with-file-descriptor
| if (shm_size == 0) return NULL; | ||
|
|
||
| shm_unlink(key); | ||
| int fd = shm_open(key, O_RDWR | O_CREAT | O_TRUNC, 0666); |
There was a problem hiding this comment.
Can permissions be tighter, say 0600?
|
@markbrown314 - do you think it's possible to use |
davidozog
force-pushed
the
wip/mmap_xpmem_heap
branch
from
e4465b4 to
974c41f
Compare
Change-Id: Id52557bfc490c364fe812ef5de3d9bd23c0c4921
Change-Id: Id9499dbd50ed9d95eef70dff389f8f00c3ca1fc0
davidozog
force-pushed
the
wip/mmap_xpmem_heap
branch
from
974c41f to
cbe7bf6
Compare
This new on-node transport mmap's the symmetric data and heap segments of all other PE's, much like XPMEM does. The intent is to provide an alternative to XPMEM for shared memory transport when XPMEM is not available. The RMA performance looks very similar to XPMEM.
To create a file-backed mmap of the data segment, I copy the whole segment to the file during initialization. This may be risky if another thread/entity can somehow update the data segment before the mmap succeeds. But I haven't seen any problems in practice. (MacOS does not support this because writing to the file fails for some reason.)
Thanks to @halitdogan and Pardo for the help on this!