Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add MPI_File_fence #2

Open
wants to merge 3 commits into
base: dev-unify
Choose a base branch
from
Open

Conversation

adammoody
Copy link

@adammoody adammoody commented Feb 10, 2022

@roblatham00 , I was hacking together a prototype for MPI_File_fence. We talked about defining two more calls: MPI_File_flush and MPI_File_fetch, but I'll just stick with fence to start with. I think fence is likely the call HDF would eventually use in place of:

MPI_File_sync()
MPI_Barrier()
MPI_File_sync()

Recall the fence synchronizes with the file system servers to expose newly written data, but it does not require the data to be flushed to disk. The data only need to be made visible for reads by other procs. Fence also implies a global barrier across ranks so that all ranks know that all other ranks have also flushed their data upon returning from fence.

One intrusive change is that it requires adding a new field to the struct of ADIO function pointers. I've not completed that for all ADIO implementations. I just added it in a couple places to bring up the point.

@adammoody
Copy link
Author

adammoody commented Feb 15, 2022

With the second commit in this PR, this at least now compiles for me. It's still not added to the function pointer table across all ADIO implementations yet. It's now defined for NFS, TESTFS, UFS, Lustre, GPFS, and UNIFY.

@adammoody
Copy link
Author

adammoody commented Feb 15, 2022

I extended the src/mpi/romio/test/perf.c test as a check and verified that it works on NFS.

@adammoody
Copy link
Author

@roblatham00 , for POSIX-compliant file systems like Lustre and GPFS, the MPI_File_fence only needs to execute a barrier. That's how I implemented the GEN_Fence.

However, for file systems that are not POSIX-compliant like NFS, ADIO_Fence may need to do more, perhaps even fall back to call ADIO_Flush for an fsync. I could use your help in adding it to the other backends.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant