add fortran interface for mount and unmount

[adammoody]

New files unifycrf.{c,h} define Fortran wrappers to call unifycr_mount/unmount calls.

Description

I have some simple test cases not included in this current PR. I can see that UNIFYCR_MOUNT from Fortran calls unifycr_mount just fine. However, my Fortran call to open() is not intercepted by our open wrapper as I was hoping for. Will need to figure out what's going on there.

Motivation and Context

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the UnifyCR code style requirements.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted.

[adammoody]

@MichaelBrim , I dropped the MPI wrapper portion of this PR to just provide the Fortran interface.

[adammoody]

Example Fortran code (named test_write.F):

      program test_ckpt_F

      implicit none

      include 'mpif.h'
      include 'unifycrf.h'

      character*1024 :: basefname = "file"
      character*1024 :: fname, file_suffix
      character*1024 :: prefix = "/unifycr"
      integer(kind=4) :: flag;
      integer(kind=4) :: outflags;
      integer(kind=4) :: valid;

      integer, parameter :: ni=20, nj=30, nk=45
      integer :: loop_count=5

      integer(kind=8), dimension(ni,nj,nk) :: W1, R1

      integer :: ierr, errors, all_errors, nprocs, mynod, ios
      integer :: i,j,k,loop

      integer :: writeunit, readunit
      integer(kind=8) :: nodeoff

!      integer (kind = 8) :: total_bytes_transferred
      real (kind = 8) :: total_bytes_transferred
      real (kind = 4) overall_transfer_rate
      real (kind = 8) time0, time1, iter_time0, iter_time1

      call MPI_INIT(ierr)
      call MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, ierr)
      call MPI_COMM_RANK(MPI_COMM_WORLD, mynod, ierr)

      call UNIFYCR_MOUNT(prefix, mynod, nprocs, 0, ierr)

      nodeoff=2**21

!      readunit = mynod
!      write(file_suffix, '(i5.5)') readunit
!      fname = trim(ckptname) // "/" // 
!     +    trim(basefname) // trim(file_suffix) // ".ckpt"
      fname = trim(prefix) // "/" //
     +    trim(basefname) // ".ckpt"

!      R1 = 0
!      open(unit=readunit,file=fname,form='unformatted',
!     +    action='read')
!      read(readunit,iostat=ios) R1
!      close(readunit)

        forall(i=1:ni,j=1:nj,k=1:nk) W1(i,j,k) =
     +      nodeoff*mynod+i+ni*(j-1+nj*(k-1))

        open(unit=writeunit,file=fname,form='unformatted',
     +      action='write')
        write(writeunit,iostat=ios) W1
        close(writeunit)

      call UNIFYCR_UNMOUNT(ierr)

      call MPI_FINALIZE(ierr)

      end program test_ckpt_F

Build

. spack/share/spack/setup-env.sh

spack load leveldb
spack load [email protected]
spack load mercury
spack load argobots
spack load margo
spack load flatcc
spack load rankstr

mpif90 -g -O0 -o test_write test_write.F -I./install/include -L./install/lib -Wl,-rpath,./install/lib -lunifycrf -lunifycr_gotcha

Run

spack load mercury
spack load argobots
spack load margo
spack load gotcha
spack load leveldb
spack load flatcc

#touch /var/tmp/unifycr.conf
export UNIFYCR_CONFIGFILE=/var/tmp/unifycr.conf

mkdir /tmp/$USER/spillover
export UNIFYCR_SPILLOVER_DATA_DIR=/tmp/$USER/spillover

mkdir /tmp/$USER/spillover
export UNIFYCR_SPILLOVER_META_DIR=/tmp/$USER/spillover

totalview test_write

[adammoody]

Last I tried, I could see that the fortran interface was successfully passing control from UNIFYCR_MOUNT() to unifycr_mount() and that the mount was returning with success.

However, it then failed to intercept the Fortran open() call, which of course led to an I/O error.

[adammoody]

@MichaelBrim , would you mind taking a stab with this one? My hope is that we can use gotcha to still pick up Fortran as it calls down into the C I/O libraries so that we don't have to add a separate set I/O wrappers for the Fortran calls.

[MichaelBrim]

@adammoody sure I'll take a stab. I already managed to figure out how to add fortran build support to our examples Makefile.am. Any objection to adding your simple test program as a real example?

[adammoody]

Great, thanks! Sure, go ahead and add this example. I don't know whether this is the real way Fortran programs do I/O, so I can't vouch for how legit it is.

[MichaelBrim]

@adammoody On Summitdev, I see an error about failing to wrap stat(), but the rest of the I/O calls are successfully wrapped and executed. There's a segfault after unmount at the end of the program, but that's consistent across all examples using libunifycr_gotcha.

[adammoody]

First, I found that I had a bug in Fortran example. I was not setting the writeunit to a valid value. Setting this to "mynod" works.

After fixing that, with the Intel compiler, I get the following:

+ test_write
forrtl: Bad file descriptor
forrtl: severe (108): cannot stat file, unit 0, file /unifycr/file.ckpt
Image              PC                Routine            Line        Source             
test_write         000000000040B0CE  for__io_return        Unknown  Unknown
test_write         000000000041A0C5  for_open              Unknown  Unknown
test_write         0000000000408A51  Unknown               Unknown  Unknown
test_write         000000000040866E  Unknown               Unknown  Unknown
libc-2.17.so       00002AAAAC09D3D5  __libc_start_main     Unknown  Unknown
test_write         0000000000408569  Unknown               Unknown  Unknown

Good news is that I was able to see that this was in fact running through our I/O wrappers. The for_open was eventually calling into our open64 wrapper. It's possible that the Intel Fortran I/O library may not like the out-of-range file descriptor we are returning.

[adammoody]

Runs without error when using gfortran. The stack trace in open here goes:

OPEN --> st_open --> new_unit --> open_external --> open wrapper

Our open wrapper returns the same file descriptor value here. The gfortran example appears to work.

[adammoody]

Given that this works for gfortran, we could merge it since it will be useful to some people. I'll open an issue to track the Intel compiler problem.

[adammoody]

Issue to track Intel error: #300

[MichaelBrim]

Nice catch on the common headers too