CCPP physics_init step failed #30
Comments
|
Ufuk, I would like to suggest that you repeat the run with the updated code in ufs_public_release ufs-community/ufs-weather-model#15. This code will support the two physics suite for the release: GFSv15p2 and GFSv16beta. You will find those SDFs in https://github.com/NOAA-EMC/fv3atm/tree/ufs_public_release/ccpp/suites (all other SDFs have been removed since they are not supported for use with the UFS in this release). You will find the namelists for the C96 configuration here: The error you got may have been related to the changes you made in the namelist. Setting a parametrization to false in the namelist but still having it in the SDF often times does not work: those two files (SDF and namelist) must be compatible. Did you say you are missing an input file? Please let me know which one you are missing. |
|
@ufuk I am updating the cime build to work with the latest ufs_weather_model release code now, should have something for you later today. |
|
@mzhangw @JulieSchramm @llpcarson Pls refer to these SDFs and namelists for documentation |
|
@jedwards4b okay that is great. BTW, I have some update in CIME side and I'll make a PR for it. There is no major changes and I just updated input data directory. So, if we merge it with CIME master then we could also update it and point original CIME repository. |
|
@ligiabernardet Thanks for your help. @jedwards4b will update the model source and I could try again. |
|
@ligiabernardet if i compare both nml files i could see some difference as following, while we are creating entire namelist automatically using CIME. I need to decide which parameter is resolution and CCPP dependent and which one is the common. So, i could define CIME xml namelist file based on this information and we just need to define the modifications for specific case. Do you have any idea about it? |
|
I might miss something but i noticed that there is no option as interp_method in interpolator_nml (under FMS) Are these namelists are up-to-date with code? Any idea? |
|
Ufuk,
To my knowledge the namelists are up to date with the code. Jack Kain and
Fanglin gave DTC the namelists via Google Doc, which Dom committed to the
repository with minor changes to run C96. We tested them and they run fine.
The two namelists provided reflect how EMC runs the configurations of the
GFS (operational v15.2 and developmental v16). As you noted, there are
differences in the physical parameterizations but also in other options
(e.g., variables d2_bg_k1 and d2_bg_k2 in &fv_core_nml have different
values in the two configurations).
I have no additional information about namelist &interpolator_nml. The
settings you see are how GFS runs.
The two namelists Dom committed to the repository are for the C96
resolution. Regarding how the namelist options should vary with resolution,
I know the following variables must change with resolution in
&fv_core_nml: layout, npx, npy. I do not know whether/how other variables
should be changed.
@philip Pegion - NOAA Affiliate <[email protected]> @climbfuji @Arun
Chawla - NOAA Federal <[email protected]> Do you know if/how the
namelist should vary with resolution?
…On Tue, Dec 17, 2019 at 12:27 PM Ufuk Turunçoğlu ***@***.***> wrote:
I might miss something but i noticed that there is no option as
interp_method in interpolator_nml (under FMS)
&interpolator_nml
interp_method = 'conserve_great_circle'
/
Are these namelists are up-to-date with code? Any idea?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AE7WQATWRIIDOG2YPHONF2LQZER3JA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHDVQ3A#issuecomment-566712428>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAQSL7VXE437OLT5Y5DQZER3JANCNFSM4J3S5IKA>
.
|
|
I can only say that in the regression tests some of the dynamics/physics parameters vary with resolution (which makes sense), but I do not know which values are the "correct" ones to use for which resolution (and for which version of the suite). |
|
@ligiabernardet It is fine! we will use those two namelist for the release. |
|
@ligiabernardet @climbfuji @KateFriedman-NOAA @yangfanglin We also found some inconsistencies in the input files especially for global_o3prdlos.f77. The file which is found in the FTP site https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad/global_o3prdlos.f77 is not compatible with CCPP and the latest version of the model gives following error when we use the file from the FTP While we are getting all input files from the FTP site, this creates a problem for us. It would be nice to have input files consistent with the CCPP on the FTP. @ligiabernardet created a input directories for us to test the CCPP and that folders contains different global_o3prdlos.f77 files (at least the md5 hash are different). |
|
This has been a confusion point all along when configuring/using this
model, and I am glad CIME is here to help. You need to use
the global_o3prdlos.f77 we provided because it is compatible with the
version of NRL ozone parameterization that is used in those suites (NRL
2016). There is an older version of this parameterization (NRL 2005) that
uses a different (smaller) file by the same name. Can the correct ozone
file please be placed in the ftp site?
…On Tue, Dec 17, 2019 at 3:15 PM Ufuk Turunçoğlu ***@***.***> wrote:
@ligiabernardet <https://github.com/ligiabernardet> @climbfuji
<https://github.com/climbfuji> @KateFriedman-NOAA
<https://github.com/KateFriedman-NOAA> @yangfanglin
<https://github.com/yangfanglin> We also found some inconsistencies in
the input files especially for global_o3prdlos.f77. The file which is found
in the FTP site
https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad/global_o3prdlos.f77
is not compatible with CCPP and the latest version of the model gives
following error when we use the file from the FTP
Now getting a runtime error:
22:An error occurred in ccpp_physics_init
22:An error occured in GFS_phys_time_vary_init: Value error in GFS_phys_time_vary_init: oz_coeff from read_o3data does not match value in GFS_typedefs.F90: 4 /= 6
22:
22:FATAL from PE 22: Call to CCPP physics_init step failed
While we are getting all input files from the FTP site, this creates a
problem for us. It would be nice to have input files consistent with the
CCPP on the FTP.
@ligiabernardet <https://github.com/ligiabernardet> created a input
directories for us to test the CCPP and that folders contains different
global_o3prdlos.f77 files (at least the md5 hash are different).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AE7WQAUMA4V3EGLCMQNBJ4DQZFFQBA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHEENWI#issuecomment-566773465>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAWT36PFNCTUK3ZING3QZFFQBANCNFSM4J3S5IKA>
.
|
|
@ligiabernardet okay thanks for the clarification. It is hard to decide which input file is correct which one is old without any data stamp on the files. @KateFriedman-NOAA @yangfanglin is it possible to put those directories to the FTP site? By this way, we could still use FTP site. |
|
NRL 2006 parameterization is not supported in this release. Only NRL 2015
is supported. So we only need one global_o3prdlos.f77 in the ftp site (the
bigger one).
…On Tue, Dec 17, 2019 at 3:26 PM Ufuk Turunçoğlu ***@***.***> wrote:
@ligiabernardet <https://github.com/ligiabernardet> okay thanks for the
clarification. It is hard to decide which input file is correct which one
is old without any data stamp on the files. @KateFriedman-NOAA
<https://github.com/KateFriedman-NOAA> @yangfanglin
<https://github.com/yangfanglin> is it possible to put those directories
to the FTP site? By this way, we could still use FTP site.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AE7WQASP4JC2GMV7DIUY2TDQZFGXRA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHEFJRA#issuecomment-566777028>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAVH5JGYUHKHJR3I3GTQZFGXRANCNFSM4J3S5IKA>
.
|
|
In our ftp archive we have the following two files
global_o3prdlos.f77
ozprdlos_2015_new_sbuvO3_tclm15_nuchem.f77
In our forecast script exglobal_fcst_nemsfv3gfs.sh you can find
if [ ${new_o3forc:-YES} = YES ]; then
O3FORC=ozprdlos_2015_new_sbuvO3_tclm15_nuchem.f77
else
O3FORC=global_o3prdlos.f77
fi
So there is no confusion had users been using our system (the workflow), or
the other workflow has been correctly set up.
Regards,
Fanglin
.....
$NLN $FIX_AM/${O3FORC} $DATA/global_o3prdlos.f77
On Tue, Dec 17, 2019 at 5:26 PM Ufuk Turunçoğlu ***@***.***> wrote:
@ligiabernardet okay thanks for the clarification. It is hard to decide
which input file is correct which one is old without any data stamp on the
files. @KateFriedman-NOAA @yangfanglin is it possible to put those
directories to the FTP site? By this way, we could still use FTP site.
…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; [email protected]
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
|
|
@yangfanglin Do we have those both file in the FTP? I think we have only one of them and the file that we need is missing. If we have both file then i could make the required changes in the CIME side. |
|
I am working on putting together a trimmed down version of the fix
directory for UFS. It will have these two files included.
Fanglin
…On Tue, Dec 17, 2019 at 5:32 PM Ufuk Turunçoğlu ***@***.***> wrote:
@yangfanglin <https://github.com/yangfanglin> Do we have those both file
in the FTP? I think we have only one of them and the file that we need is
missing. If we have both file then i could make the required changes in the
CIME side.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AKY5N2I2JF3Q3BTXMN62ZYDQZFHQRA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHEF3HA#issuecomment-566779292>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKY5N2K3COGOCVSHPFNI3LTQZFHQRANCNFSM4J3S5IKA>
.
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; [email protected]
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
|
|
Thanks @yangfanglin for making the trimmed down version. All, I have replaced the fix directory on our ftp server with the trimmed down version: https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/ The version (v20191213) corresponds to our latest set on WCOSS from which this set was made. The two O3FORC files are here: https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/global_o3prdlos.f77 |
|
@laurie Carson <[email protected]> @juliescharmm Please make sure these files
have a 1-line of documentation in the UG
…On Wed, Dec 18, 2019 at 6:57 AM Kate Friedman ***@***.***> wrote:
Thanks @yangfanglin <https://github.com/yangfanglin> for making the
trimmed down version.
All, I have replaced the fix directory on our ftp server with the trimmed
down version:
https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/
fix_am.v20191213
fix_fv3_gmted2010.v20191213
The version (v20191213) corresponds to our latest set on WCOSS from which
this set was made.
The two O3FORC files are here:
https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/global_o3prdlos.f77
https://ftp.emc.ncep.noaa.gov/EIB/UFS/global/fix/fix_am.v20191213/ozprdlos_2015_new_sbuvO3_tclm15_nuchem.f77
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AE7WQARPPAPMGJFMT2QE5A3QZIT3HA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHGGALI#issuecomment-567042093>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAWDTGITFPF7NTKSPILQZIT3HANCNFSM4J3S5IKA>
.
|
|
@yangfanglin BTW, i could not find the new_o3forc if statement that you mentioned in the model code. i just checked release/v0/scripts/exglobal_fcst_nemsfv3gfs.sh the top level hash for the mrweather-model is 7a4a7f3d in the following repository and this version of code just has following copy command $NLN $FIX_AM/global_o3prdlos.f77 $DATA/INPUT/global_o3prdlos.f77 |
|
@uturuncoglu @KateFriedman-NOAA Has the new input fix directories solved this? |
|
You need to use the new script from our workflow
…On Wed, Dec 18, 2019 at 1:27 PM Ufuk Turunçoğlu ***@***.***> wrote:
@yangfanglin <https://github.com/yangfanglin> BTW, i could not find the
new_o3forc if statement that you mentioned in the model code. i just checked
release/v0/scripts/exglobal_fcst_nemsfv3gfs.sh
the top level hash for the mrweather-model is 7a4a7f3d in the following
repository
https://github.com/ufs-community/ufs-weather-model/
and this version of code just has following copy command
$NLN $FIX_AM/global_o3prdlos.f77 $DATA/INPUT/global_o3prdlos.f77
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AKY5N2NZHXW7NAPRMMG6VKTQZJTRDA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHBEMI#issuecomment-567153201>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKY5N2OZXXBKWWNESOZXXRDQZJTRDANCNFSM4J3S5IKA>
.
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; [email protected]
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
|
|
@arunchawla-NOAA I am trying to modify the CIME scripts because folders in FTP changed. Then, i'll test with new file. @yangfanglin If you don't mind, could you point me the place of the new script from your workflow? Thanks for your help. |
|
Please see https://github.com/NOAA-EMC/global-workflow/tree/feature/gfsv16
The forecast script is
https://github.com/NOAA-EMC/global-workflow/blob/feature/gfsv16/scripts/exglobal_fcst_nemsfv3gfs.sh
…On Wed, Dec 18, 2019 at 2:50 PM Ufuk Turunçoğlu ***@***.***> wrote:
@arunchawla-NOAA <https://github.com/arunchawla-NOAA> I am trying to
modify the CIME scripts because folders in FTP changed. Then, i'll test
with new file.
@yangfanglin <https://github.com/yangfanglin> If you don't mind, could
you point me the place of the new script from your workflow? Thanks for
your help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AKY5N2MRPSIY7BSGX5WBE5TQZJ5J7A5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHI65Q#issuecomment-567185270>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKY5N2KP7OALDI3OVZZRYRTQZJ5J7ANCNFSM4J3S5IKA>
.
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; [email protected]
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
|
|
Correction
https://github.com/NOAA-EMC/global-workflow/tree/feature/gfsv16b
https://github.com/NOAA-EMC/global-workflow/blob/feature/gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh
On Wed, Dec 18, 2019 at 3:36 PM Fanglin Yang - NOAA Federal <
[email protected]> wrote:
… Please see
https://github.com/NOAA-EMC/global-workflow/tree/feature/gfsv16
The forecast script is
https://github.com/NOAA-EMC/global-workflow/blob/feature/gfsv16/scripts/exglobal_fcst_nemsfv3gfs.sh
On Wed, Dec 18, 2019 at 2:50 PM Ufuk Turunçoğlu ***@***.***>
wrote:
> @arunchawla-NOAA <https://github.com/arunchawla-NOAA> I am trying to
> modify the CIME scripts because folders in FTP changed. Then, i'll test
> with new file.
>
> @yangfanglin <https://github.com/yangfanglin> If you don't mind, could
> you point me the place of the new script from your workflow? Thanks for
> your help.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#30?email_source=notifications&email_token=AKY5N2MRPSIY7BSGX5WBE5TQZJ5J7A5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHHI65Q#issuecomment-567185270>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AKY5N2KP7OALDI3OVZZRYRTQZJ5J7ANCNFSM4J3S5IKA>
> .
>
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; ***@***.***
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
--
Fanglin Yang, Ph.D.
Physical Scientist
Environmental Modeling Center
National Centers for Environmental Prediction
301-6833722; [email protected]
http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/
http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
|
|
I implemented almost all the logic for cold start and and CCPP v15p2 and try to run the model but i am getting following error in microphysics, The initial conditions are coming from https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad/INPUT/ and all input namelist files are generated automatically by CIME (it seems that they are identical with https://github.com/ufs-community/ufs-weather-model/blob/ufs_public_release/parm/ccpp_v15p2_c96.nml.IN, except default options, they are not included to the namelist). i 'll also test with the input files found in static directory provided by @ligiabernardet PS: My run directory is in /glade/scratch/turuncu/ufs-mrweather-app/run |
|
Ufuk, Linlin Pan did a couple of successful runs today and transferred the
directories to Cheyenne
cheyenne4:/glade/p/ral/jntp/lpan/updated/v15p2_run/fv3_gfs_v15p2_repro_ccpp
cheyenne4:/glade/p/ral/jntp/lpan/updated/v16beta_run/fv3_gfs_v16beta_ccpp
Can you compare against you runs to see if it sheds any light?
…On Thu, Dec 19, 2019 at 3:14 PM Ufuk Turunçoğlu ***@***.***> wrote:
I implemented almost all the logic for cold start and and CCPP v15p2 and
try to run the model but i am getting following error in microphysics,
102:MPT: #2 MPI_SGI_stacktraceback (
102:MPT: ***@***.***=0x7ffe6f97c340 "MPT ERROR: Rank 102(g:102) received signal SIGBUS(7).\n\tProcess ID: 25772, Host: r2i3n20, Program: /glade/scratch/turuncu/ufs-mrweather-app/bld/ufs.exe\n\tMPT Version: HPE MPT 2.19 02/23/19 05:30:09\n") at sig.c:340
102:MPT: #3 0x00002ae603dc3fb2 in first_arriver_handler ***@***.***=7,
102:MPT: ***@***.***=0x2ae6109a0080) at sig.c:489
102:MPT: #4 0x00002ae603dc434b in slave_sig_handler (signo=7, siginfo=<optimized out>,
102:MPT: extra=<optimized out>) at sig.c:564
102:MPT: #5 <signal handler called>
102:MPT: #6 gfdl_cloud_microphys_mod_mp_wqs2_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/module_gfdl_cloud_microphys.F90:3878
102:MPT: #7 0x0000000000cad36f in gfdl_cloud_microphys_mod_mp_subgrid_z_proc_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/module_gfdl_cloud_microphys.F90:2131
102:MPT: #8 0x0000000000cac1f2 in gfdl_cloud_microphys_mod_mp_icloud_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/module_gfdl_cloud_microphys.F90:2003
102:MPT: #9 0x0000000000c9ea5d in gfdl_cloud_microphys_mod_mp_mpdrv_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/module_gfdl_cloud_microphys.F90:971
102:MPT: #10 0x0000000000c98529 in gfdl_cloud_microphys_mod_mp_gfdl_cloud_microphys_mod_driver_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/module_gfdl_cloud_microphys.F90:476
102:MPT: #11 0x0000000000b9fceb in gfdl_cloud_microphys_mp_gfdl_cloud_microphys_run_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/physics/gfdl_cloud_microphys.F90:231
102:MPT: #12 0x0000000000b743fe in ccpp_fv3_gfs_v15p2_physics_cap_mp_fv3_gfs_v15p2_physics_run_cap_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/ccpp_FV3_GFS_v15p2_physics_cap.F90:1548
102:MPT: #13 0x0000000000b0a534 in ccpp_static_api_mp_ccpp_physics_run_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/physics/ccpp_static_api.F90:150
102:MPT: #14 0x0000000000b0c0b6 in ccpp_driver_mp_ccpp_step_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/ccpp/driver/CCPP_driver.F90:234
102:MPT: #15 0x00000000004b966e in atmos_model_mod_mp_update_atmos_radiation_physics_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/atmos_model.F90:368
102:MPT: #16 0x00000000004afd63 in module_fcst_grid_comp_mp_fcst_run_phase_1_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/module_fcst_grid_comp.F90:705
102:MPT: #17 0x00002ae5ff5d87de in ESMCI::FTable::callVFuncPtr(char const*, ESMCI::VM*, int*) ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #18 0x00002ae5ff5dc39b in ESMCI_FTableCallEntryPointVMHop ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #19 0x00002ae5ffa9f2d5 in ESMCI::VM::enter(ESMCI::VMPlan*, void*, void*) ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #20 0x00002ae5ff5d9e3a in c_esmc_ftablecallentrypointvm_ ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #21 0x00002ae5ffcceb4d in esmf_compmod_mp_esmf_compexecute_ ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #22 0x00002ae5ffec7e31 in esmf_gridcompmod_mp_esmf_gridcomprun_ ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #23 0x00000000004a2cdc in fv3gfs_cap_mod_mp_modeladvance_ ()
102:MPT: at /glade/scratch/turuncu/ufs-mrweather-app/bld/atm/obj/FV3/fv3_cap.F90:998
102:MPT: #24 0x00002ae5ff946d59 in ESMCI::MethodElement::execute(void*, int*) ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #25 0x00002ae5ff946c8a in ESMCI::MethodTable::execute(std::string, void*, int*, bool*) ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #26 0x00002ae5ff946172 in c_esmc_methodtableexecute_ ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #27 0x00002ae5ffb7a592 in esmf_attachmethodsmod_mp_esmf_methodgridcompexecute_
102:MPT: ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
102:MPT: #28 0x00002ae600397ff9 in nuopc_modelbase_mp_routine_run_ ()
102:MPT: from /glade/work/turuncu/PROGS/esmf/8.0.0/mpt/2.19/intel/19.0.2/lib/libO/Linux.intel.64.mpt.default/libesmf.so
The initial conditions are coming from
https://ftp.emc.ncep.noaa.gov/EIB/UFS/RT/fv3_gfdlmprad/INPUT/ and all
input namelist files are generated automatically by CIME (it seems that
they are identical with
https://github.com/ufs-community/ufs-weather-model/blob/ufs_public_release/parm/ccpp_v15p2_c96.nml.IN,
except default options, they are not included to the namelist).
i 'll also test with the input files found in static directory provided by
@ligiabernardet <https://github.com/ligiabernardet>
PS: My run directory is in /glade/scratch/turuncu/ufs-mrweather-app/run
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30?email_source=notifications&email_token=AE7WQAW5NGUTDSU43GTFQALQZPW5NA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHLG7DI#issuecomment-567701389>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE7WQAQKOC756GTSFCZVUYLQZPW5NANCNFSM4J3S5IKA>
.
|
|
@ligiabernardet Okay that is great. Comparing the directories will definitely help. There is also some inconsistencies between the file and if you use the script to find the resolution of the static files, it must be t190.384.192 but in the namelist file t126.384.190 is used. So, i am not sure which one is correct. I also found that ccpp_v15p2_c96.nml.IN file includes also namelist variables that are same with the default values in the code, which i am not including them to input.nml. Now, i am going through to find the diffferncies between ccpp_v15p2_c96.nml.IN and input.nml produced by CIME. |
|
@uturuncoglu, I'm looking in your directory on cheyenne, and I don't see the suite definition file. The crash is occurring in a portion of the microphysics that seems to catch unphysical temperatures. And answering @ligiabernardet questions from earlier. The other options that vary with resolution is cdmbgwd. The global workflow config/config.fv3 show the resolution dependent parameters. |
|
@pjpegion do we need to copy suit definition file (XML one) also to the run directory? |
|
I'm not 100% sure, but if it is no there, how are you telling the model where it is? @ligiabernardet would know for sure. |
|
Okay, to test it i copied to my run directory manually. So, i'll let you know about it. It is still in the queue. |
|
The SDF does not have to be copied to the run directory.
CCPP is compiled in static mode, which means that the SDF needs to be present only during build time. The name of the suites presented at compile time (in the xml attribute name="...") are compiled into the executable, and the value obtained from CCPP_suite=... in input.nml will be used to run a certain suite.
… On Dec 19, 2019, at 3:49 PM, Phil Pegion ***@***.***> wrote:
I'm not 100% sure, but if it is no there, how are you telling the model where it is? @ligiabernardet <https://github.com/ligiabernardet> would know for sure.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=AB5C2RNPWULRRCHNEH32WCDQZP3AFA5CNFSM4J3S5IKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHLJRQI#issuecomment-567711937>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RKABSK7JOZFS2J5YKLQZP3AFANCNFSM4J3S5IKA>.
|
|
@climbfuji thanks for clarifying. @uturuncoglu can you point me to your SDF, I want to see if it consistent with the namelist. |
|
@climbfuji Thanks for the clarification. I also tested by copying to the run directory and it fails with the same way. The only difference seems that i am using different resolution for the static files. So, i'll try with the lower resolutions again. @yangfanglin There is no file called seaice_newland.grb in the global/fix/fix_am.v20191213. Actually this is used by fv3_gfs_v15p2_repro_ccpp/input.nml that @ligiabernardet pointed. If you don't mind could you also put it to the FTP. |
|
@pjpegion it is in /glade/scratch/turuncu/ufs-mrweather-app/run/suite_FV3_GFS_v15p2.xml and i copied from my source /glade/u/home/turuncu/EMC/ufs-mrweather-app.dec17/src/model/FV3/ccpp/suites/suite_FV3_GFS_v15p2.xml |
|
I tested following configurations;
Thanks to all for your help. |
|
@ligiabernardet @climbfuji @yangfanglin @pjpegion When i set h2o_phys = .true. (it was .false.) the model runs without any problem. I had also found problem with d_ext option, it was in its default value. So, now it seems it is working. I'll also test with CCPP v16beta. Thanks again for your great help. |
|
@uturuncoglu is this resolved now? Can this issue be closed? |
|
@arunchawla-NOAA Yes, the issue can be closed now. I could able to run the model without any problem using both CCPP v15p2 and v16beta suites. In this case, all namelist files are automatically generated by CIME. |
arunchawla-NOAA
added this to the Internal testing of end to end system using CIME milestone
@ligiabernardet I am getting following error when i try to run the model with CCPP (by default the model builds with CCPP FV3_GFS_v15)
The full log is in
/glade/scratch/turuncu/ufs-mrweather-app-master/run/ufs.log.9506878.chadmin1.ib0.cheyenne.ucar.edu.191216-144606
Following commands can be used to reproduce the error,
I just wonder is there any flag to debug CCPP and get more information about problem?
It might be a missing file or configuration option but i am not sure.
In this case, all the input files are coming from the https://ftp.emc.ncep.noaa.gov/EIB/UFS and input.nml and model_configure are generated by CIME. So, i am not copying them from any static input directory. I also tested with input.nml (i just set h2o_phys to .false because the required file does not exist in the FTP site) provided by @ligiabernardet which is in /glade/work/turuncu/FV3GFS/fv3_gfs_v15_repro_ccpp but i am getting same error.
The text was updated successfully, but these errors were encountered: