Compile scorpio with adios2, got error on PIO_MAX_CACHED_STEPS_FOR_ADIOS #554
Comments
|
Can you include your complete configure/make log (you can copy it in a github gist and include the link to gist here)? |
|
I found this error caused by the previously compiled libgptl.a of scorpio is reused. So this problem is gone, but after the successful compilation, I got a lot of fails on ctest. So I still share the cmake log here and ctest log. Could you please tell me how I can pass all ctest? Thanks! https://gist.github.com/halehawk/1acad191f06c3d805edf2a6cf9e020b2 |
|
Can you try launching one of the tests in a batch file (without using ctest, directly using mpiexec/aprun etc)? It looks like ctest support for this machine (derecho?) is not added in SCORPIO yet. It should be a small patch that we can provide to you for testing that should get the ctest testing working for you. Please try running the tests manually in a batch file and let us know the results (Also include the batch file in the issue). |
|
I requested an interactive node on derecho, ran the ctest and single test as well. Some tests can pass now, but more cannot and even stuck at test #71. |
|
@halehawk Your ADIOS2 lib is quite old (version 2.7.1) For latest ADIOS read support we do require ADIOS2 2.9.0 or higher versions. I think you can install ADIOS2 2.9.1 for testing (we have not tested latest 2.9.2 or 2.10.0-rc1 so far). FYI, you can install ADIOS2 2.9.1 with the commands below (please change /path/to/your/adios/installation accordingly) |
|
@halehawk It seems that you configured SCORPIO with NetCDF 4.9.2: Our nightly builds use an older but more stable NetCDF 4.8.0 lib for testing (higher versions have some known issues for the unit tests). In fact, you do not need NetCDF lib for testing ADIOS type (PnetCDF lib is sufficient). FYI, you can use "-DWITH_NETCDF=OFF" to configure SCORPIO without NetCDF support (you can safely remove -DNetCDF_C_PATH=XXXX and -DNetCDF_Fortran_PATH=XXXX in this case). |
|
@halehawk As a sanity check on that machine, I would suggest that you first run the full SCORPIO unit tests with PnetCDF type only (SCORPIO is configured without NetCDF or ADIOS support): PIO_ENABLE_EXAMPLES is set to ON to enable testing some C examples. |
|
Sure, I will try your last suggestion first.Sent from my iPhoneOn Jan 4, 2024, at 9:24 AM, dqwu ***@***.***> wrote:
@halehawk As a sanity check, I would suggest that you run the full SCORPIO unit tests with PnetCDF type only (SCORPIO is configured without NetCDF or ADIOS support):
git clone https://github.com/E3SM-Project/scorpio.git
cd scorpio
mkdir build
cd build
CC=mpicc CXX=mpicxx FC=mpif90 cmake -Wno-dev \
-DWITH_NETCDF=OFF \
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr \
-DPIO_USE_MALLOC=ON \
-DPIO_ENABLE_TESTS=ON \
-DPIO_ENABLE_EXAMPLES=ON \
..
make -j4
make tests
ctest
PIO_ENABLE_EXAMPLES is set to ON to enable testing some C examples.
PIO_USE_MALLOC is set to ON (recommended) to use native malloc (instead of bget package).
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@halehawk ADIOS IO type in SCORPIO has some known limitations, see #553 for detailed information. For ADIOS type, you should not run all of the unit tests (C and Fortran). In fact, our nightly builds only test a subset of the Fortran unit tests (C unit tests do not support ADIOS type so far), plus C example test_adios.c. After you install a custom ADIOS 2.9.1 lib on that machine, I would suggest that you test ADIOS write feature first (ADIOS read support has more restrictions). FYI, below are the commands to test ADIOS write feature: Note that we only run a subset of the Fortran unit tests and test_adios example with the ctest command. Also, ADIOS_BP2NC_TEST is set to ON to perform implicit file conversion (ADIOS to NetCDF) such that we actually read converted .nc files for testing. |
|
@halehawk For testing ADIOS read feature, we only run test_adios.c example so far (the shorter list of applicable Fortran unit tests has not been determined yet). Simply remove "-DADIOS_BP2NC_TEST=ON" and only run test_adios with the ctest command: |
|
I tested on pnetcdf with scorpio. Until test#69, all tests passed. |
|
I tried to compile scorpio with esmf. I included scorpio include path and adios2 include path. But I still got this error: It looks like spio_hash.h is not in my scorpio include path, how can I change cmake option to install this header file? |
It might take some time for us to fix this issue. If you manually copy spio_hash.h to /glade/.../scorpio/install/include, does it work? |
|
Yes, this error is gone. Then I got this error
/glade/.../esmf/src/Infrastructure/Mesh/src/ESMCI_UGRID_Util.C:357:11:
error: use of undeclared identifier 'PIOc_InitDecomp_ReadOnly'
piorc = PIOc_InitDecomp_ReadOnly(pioSystemDesc, PIO_INT,
Is there any macro I should define to avoid this error?
…On Wed, Jan 10, 2024 at 3:53 PM dqwu ***@***.***> wrote:
I tried to compile scorpio with esmf. I included scorpio include path and
adios2 include path. But I still got this error: In file included from
/glade/.../esmf/src/Infrastructure/IO/src/ESMCI_IO_Handler.C:36: In file
included from
/glade/.../esmf/src/Infrastructure/IO/src/../include/ESMCI_PIO_Handler.h:45:
/glade/.../scorpio/install/include/pio.h:71:10: fatal error: In file
included from
/glade/.../esmf/src/Infrastructure/IO/src/ESMCI_PIO_Handler.C:'spio_hash.h'
file not found27:
It looks like spio_hash.h is not in my scorpio include path, how can I
change cmake option to install this header file?
It might take some time for us to fix this issue. If you manually copy
spio_hash.h to /glade/.../scorpio/install/include, does it work?
—
Reply to this email directly, view it on GitHub
<#554 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFAZC63EKDM745JTX6DYN4LV7AVCNFSM6AAAAABBKVPKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVHA3TGMZRGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
And this erro /glade/.../esmf/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:219:13: error: use of undeclared identifier 'PIOc_free_iosystem'; |
|
@halehawk It seems that PIOc_InitDecomp_ReadOnly is an API of NCAR PIO, which is not supported by SCORPIO. |
|
ESMCI_Mesh_FileIO.C might need to be updated to use SCORPIO APIs. |
|
PIOc_InitDecomp_ReadOnly is a read decomp, can Scorpio read nc files
directly?
…On Wed, Jan 10, 2024 at 4:13 PM dqwu ***@***.***> wrote:
ESMCI_Mesh_FileIO.C might need to be updated to use SCORPIO APIs.
—
Reply to this email directly, view it on GitHub
<#554 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFBYFCKPPEIXUUKRL2DYN4OADAVCNFSM6AAAAABBKVPKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVHA4TIOJUGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@halehawk I think you can simply use PIOc_InitDecomp() in ESMCI_Mesh_FileIO.C |
|
After I replaced PIOc_InitDecomp_ReadOnly with PIOc_InitDecomp. I still have the error: And this erro /glade/.../esmf/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:219:13: error: use of undeclared identifier 'PIOc_free_iosystem'; |
|
@halehawk You can use PIOc_finalize() instead of PIOc_free_iosystem(). FYI, NCAR PIO added PIOc_free_iosystem() as a duplicate for PIOc_finalize():
|
|
OK, I commented free_iosystem out for continuing compilation. I will change to PIOc_finalize. Now Scorpio static libraries cannot be linked its path to load. Do you know where I can add -fPIC in your CMakeLists.txt to build a shared library? Thanks! |
|
AFAIK, ESMF currently cannot use SCORPIO (The PIO library added some APIs specifically for ESMF that are currently not available in SCORPIO. It has been in our todo list but hasn't been a high priority task yet.) |
|
@halehawk E3SM uses static scorpio lib without any build issues. I think you should be able to build your application with static scorpio lib as well. @jayeshkrishna Any thoughts on this? |
|
Using static lib should work fine, however as I mentioned above you cannot use SCORPIO with ESMF right now. |
|
So I just used the scorpio static lib with esmf, and changed the previous two APIs, now at least esmf compiled. |
|
Though I built scorpio enabled esmf. But when I tried to build this esmf with cesm, I got the following errors: /glade/.../tmp/ifortAc6Qfb.i: error #5286: Ambiguous generic interface PIO_SETERRORHANDLING: previously declared specific procedure SPIO_ERR::PIO_SETERRORHANDLING_IOSYS is not distinguishable from this declaration. [PIOLIB_MOD::SETERRORHANDLINGIOSYSTEM] /glade/.../my_cesm_sandbox1/components/cam/src/dynamics/mpas/dycore/src/framework/mpas_io.F(313): error #6405: The same named entity from different modules and/or program units cannot be referenced. [PIO_OPENFILE] Is this caused by I included spio header files? If not, how can I use from PIO mode instead of SPIO mod? |
You might try configuring scorpio with -DPIO_USE_FORTRAN_LEGACY_LIB=ON |
|
After I enable -DPIO_USE_FORTRAN_LEGACY_LIB=ON, can I still use PIO_IOTYPE_ADIOS2? |
Yes you can. |
|
Then why are there USE_LEGACY_PIO? What is the difference between using it and not using it? |
Please see PR #513 for detailed information. If CESM or ESMF does not support the new Fortran interface of SCORPIO, you might switch back to the legacy interface as a workaround. |
|
@halehawk Please notice that there might be more issues with your testing. As @jayeshkrishna has mentioned, ESMF does not work with SCORPIO so far. |
|
So far, the legacy pio worked on cesm/esmf. I just hope scorpio has few issues with esmf and can find replace APIs easily. By the way, I want to know if scorpio can read netcdf files if the pio_iotype is set to adios. |
When the pio_iotype is set to adios, if only netcdf files exist, scorpio will switch adios type to pnetcdf (or netcdf) type to read them. |
|
Now I used scorpio to build with cesm, cesm has pio_short type, but scorpio doesn't have it. Do you have any replacement type for pio_short, if you do, which one? Thanks! |
We will fix this issue later. For now, you can manual edit src/flib/pio_types.F90 to add 3 new lines: |
|
@halehawk Maybe updating pio_types.F90 is not sufficient. Please wait for further updates, thanks. |
|
Yes, thank you. It looks like you need to add PIO_SHORT in genf90.pl as well. Another thing, currently I compiled scorpio in adios write mode as you suggested which is enabled BP2NC mode. But the cesm has problem as follows: |
Are you using PIO_SHORT for these calls? Maybe the Fortran interface pio_get_var (and some others) needs to be updated to support PIO_SHORT as well. |
|
NCAR PIO has added support for PIO_SHORT, and it seems that there are quite a few files updated: |
|
No, you can see the data type is double1d, double2d, real1d, real2d in
these calls. I will update scorpio and try again.
…On Fri, Mar 22, 2024 at 8:30 AM dqwu ***@***.***> wrote:
Yes, thank you. It looks like you need to add PIO_SHORT in genf90.pl as
well. Another thing, currently I compiled scorpio in adios write mode as
you suggested which is enabled BP2NC mode. But the cesm has problem as
follows: error #6284: There is no matching specific function for this
generic function reference. [PIO_GET_VAR] rcode = pio_get_var(pioid,
varid,start=(/1,1,1,nt/),count=(/1,1,1,1/), ival=data_real2d) rcode =
pio_get_var(pioid, varid,start=(/1,1,nt/),count=(/1,1,1/),
ival=data_real1d) rcode = pio_get_var(pioid, varid,start=(/1,1,1,nt/),
count=(/1,1,1,1/), ival=data_dbl2d) rcode = pio_get_var(pioid,
varid,start=(/1,1,nt/), count=(/1,1,1/), ival=data_dbl1d) Do you know if it
caused by BP2NC mode or netcdf disabled? Thanks!
Are you using PIO_SHORT for these calls? Maybe the Fortran interface
pio_get_var (and some others) needs to be updated to support PIO_SHORT as
well.
—
Reply to this email directly, view it on GitHub
<#554 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAPEFDXABZ2LWYOZ6AG5H3YZQ6A3AVCNFSM6AAAAABBKVPKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVGIZDSMZZGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Now I am trying to build the scorpio latest version, I got the following error even after I set PIO_ENABLE_LOGGING=ON |
|
It's not compiler problem, I tested with a simple c code, it can create a symbolic link on my machine by ICC or ICX. |
|
I double checked with a simple testcase for scorpio pio_get_var with arguments (file, varid,start, count,val), it cannot find matching functions, but if I called pio_get_var with (file,varid, val), it works. So is scorpio supposed to not support pio_get_var(file,varid,start,count,val) or pio_get_var(file, vardesc,start,count, val)? If scorpio does support, I suspect that the declaration of pio_get_var(file,varid,start,count,val) is not correct after you generate spio_get_var using genf90.pl. Could you please have a look? |
pio_get_var(file,varid,start,count,val) should be supported by SCORPIO. Maybe it is the type of val that is not supported (e.g. PIO_SHORT)? As we've mentioned before, ESMF currently isn't compatible with SCORPIO, and we don't recommend further testing. Even after resolving some known issues (such as PIO_SHORT not being supported), you're likely to encounter additional unexpected issues while testing ESMF with SCORPIO. |
|
@halehawk For pio_get_var(), you might also try legacy Fortran interface. Edit scorpio/CMakeLists.txt and set PIO_USE_FORTRAN_LEGACY_LIB to ON by default: |
|
The situation is I want to use adios to write and netcdf to read, if I turn on fortran legacy, it looks like I cannot use adios to write in fortran code. |
|
I didn't mean pio_get_var for PIO_SHORT, I mean pio_get_var for PIO_REAL and PIO_DOUBLE, you declared four modules in the pio_get_var for 1d_real, I tested I only can use pio_get_var with (file,varid, val) or (file,vardesc,val), but not pio_get_var with (file,varid,start,count,val) or (file,vardesc,start,count,val). And also I didn't get this problem from esmf, but from "use pio: pio_get_var" directly. So that's why I want to ask for your help. |
Hello,
I tried to compile scorpio with adios2 at derecho.ucar.edu, I got the following error:
/glade/derecho/scratch/haiyingx/scorpio/src/clib/pioc_support.c:360:58: error: expected expression
file->max_step_calls = PIO_MAX_CACHED_STEPS_FOR_ADIOS;
It looks like PIO_MAX_CACHED_STEPS_FOR_ADIOS is set by during cmake as the follows, do I miss anything in the cmake options?
Thanks, Haiying
Here is the partial cmake log:
CC=mpicc CXX=mpicxx FC=mpif90 cmake -DNetCDF_C_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.25/oneapi/2023.0.0/wzol/
-DNetCDF_Fortran_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.25/oneapi/2023.0.0/wzol/
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr/
-DWITH_ADIOS2=ON -DADIOS2_DIR=/glade/work/haiyingx/ADIOS2/installintelde/lib64/cmake/adios2
-DCMAKE_INSTALL_PREFIX=/glade/derecho/scratch/haiyingx/scorpio/install
-DPIO_ENABLE_TESTS=ON
..
-- ===== Configuring SCORPIO... =====
-- Enabling SCORPIO I/O performance statistics collection (default)
-- Using BGET to allocate memory for caching data in SCORPIO
-- Disabling debug logging in SCORPIO (default)
-- Disabling use/check of the MPI serial library (default)
-- Disabling saving I/O decompositions (default)
-- No limit on the number of cached I/O regions (default)
-- Limit on the number of Lustre OSTs, PIO_MAX_LUSTRE_OSTS, is not set (default)
-- Filesystem striping unit is not set (default)
-- Using PnetCDF independent data mode to read variables in SCORPIO (default)
-- Reserving some extra space in the header when creating NetCDF files, requested bytes = 10240 (default)
-- Setting the maximum number of I/O decompositions registered with ADIOS type to PIO_MAX_ADIOS_DECOMPS = 65536 (default)
-- Setting the maximum number of cached application steps for ADIOS type to PIO_MAX_CACHED_STEPS_FOR_ADIOS = 128 (default)
-- Disabling code coverage... (use -DPIO_ENABLE_COVERAGE:BOOL=ON to enable coverage, only GNU is supported for now)
The text was updated successfully, but these errors were encountered: