NEMS update to run standalone atm using coupled executable

[junwang-noaa]

NEMS code needs to be updated to allow standalone atm to run with coupled executable. Currently the code has dependency in CMEPS when CMEPS component is defined in the executable while the standalone atm run does not need CMEPS.

[junwang-noaa]

Jun was keeping the CMEPS in nems.configure. Moorthi added an if statement in module_EARTH_GRID_COMP.F90

#ifdef CMEPS
        inst_suffix = ""

        if (componentCount > 1) then
          ! obtain driver attributes (for CMEPS)
          call ReadAttributes(driver, config, "DRIVER_attributes::", formatprint=printattr, rc=rc)
          if (ChkErr(rc,__LINE__,u_FILE_u)) return

          call ReadAttributes(driver, config, "ALLCOMP_attributes::", formatprint=printattr, rc=rc)
          if (ChkErr(rc,__LINE__,u_FILE_u)) return
        endif
#endif

Some conversation on running coupled executable with coupled CCPP suite files for standalone atm, this is beyond the changes in NEMS.

Jun: I am actually concerned to use the same executable with coupled ccpp suite file to run atmosphere only test. It can run, but there is no sst/ice update from ocn/ice components and no atm internal ice model will be called either as the standalone atm used to. We need to compile the code with an atmosphere only ccpp suite file and use that suite file for atmosphere only test. @dom Heinzeller I want to confirm we don't have the choice to use same ccpp suite file, but call sfc_cice for coupled (cplflx is .true.) and call sfc_sice for standalone fv3 (cplflx is .false), right?

Dom: It's possible to make this work, but counterintuitive: call both sfc_cice and sfc_sice in the SDF, but bail out in one of them if cplflx is true (and of the other if false). But what then if someone wants to use both?What is in the suite definition file should run, in my opinion.

Jun: So basically that means we can not run the coupled executable compiled with coupled suite file to run standalone atm.

Dom: Correct.

[junwang-noaa]

@rsdunlapiv @DeniseWorthen @SMoorthi-emc @yangfanglin @arunchawla-NOAA @climbfuji Here are some discussions on using coupled executable for standalone atm.

[SMoorthi-emc]

I think Dom is not correct. It depends on the suite you use.
I compile with suite with NSST and sfc_sice; so when I run standalone FV3, it is correct.
Again, I want to emphasize that i is a consequence of abandoning IPD. One need to know what they are doing with CCPP.

[SMoorthi-emc]

With my update to NEMS you also need
"! clean-up
! --------
deallocate(petList)
#ifdef CMEPS
if (componentCount > 1) then
! Perform restarts if appropriate
call InitRestart(driver, rc)
if (ChkErr(rc,LINE,u_FILE_u)) return

        call AddAttributes(comp, driver, config, i+1, trim(prefix), inst_suffix, rc=rc)
        if (ChkErr(rc,__LINE__,u_FILE_u)) return
      endif

#endif

    enddo

[SMoorthi-emc]

Regarding "sfc_cice" it is used only when ice model is coupled. Otherwise, it is ignored.
"sfc_sice" is used for both coupled and uncoupled model. When coupled, it is used only for lakes.
When uncoupled, it is used for all ice. The coupled executable as uncoupled works perfectly fine if the SDF is set correctly.
It is unfortunate that mis-statements are being made.

[climbfuji]

I think Dom is not correct. It depends on the suite you use.
I compile with suite with NSST and sfc_sice; so when I run standalone FV3, it is correct.
Again, I want to emphasize that i is a consequence of abandoning IPD. One need to know what they are doing with CCPP.

Moorthi, it is possible to do that if you add logic inside the scheme to bail out based on certain flags being true or false. We have presence for this practice in the current code, but I don't think it is good: nsst bails out if nstf_name (1) is zero, for example. Rayleigh damping (which should be removed from all suites, since it is never used in the physics) bails out if the damping parameter is zero. Possible that sfc_sice and sfc_cice are set up in that way, too. (as you just said). My point is that we should consider whether this is the right approach or not. I don't think it is. We could just stuff all possible physics in a suite (for example all microphysics options) and add logic at the top of each to bail out if imp_physics doesn't match that scheme's value. This does not make sense at all.

[rsdunlapiv]

It is clear we need to make sure any CMEPS related calls are conditional in module_EARTH_DRIVER.F90. However, we should try to do this without any #ifdef CMEPS statements since I don't think those are really needed and it reduces the flexibility of the driver. The driver is designed to allow you to change the run sequence at runtime from one with only an ATM to one with several components. Ideally, the driver does not know that much about any of the components, including CMEPS, so I advocate removing all #ifdef CMEPS

We can add a check for the existence of DRIVER_attributes:: and ALLCOMP_attributes:: in nems.configure and only ingest those into the subcomponent if they are present.

This looks the most problemmatic:

#ifdef CMEPS
        use med_internalstate_mod , only : med_id
#endif

We need to look into why CMEPS needs this med_id in the first place - we should definitely not be accessing the internal state of any component like this.

[rsdunlapiv]

@climbfuji at one time I was thinking that you built the model with a set of CCPP suites and then selected a particular suite at runtime. Is this not the case?

[DeniseWorthen]

@rsdunlapiv I talked to Mariana about this and was one of the things I needed to resolve in the 2nd phase of the nems driver cleanup. I have not had time to work on it though

[climbfuji]

@climbfuji at one time I was thinking that you built the model with a set of CCPP suites and then selected a particular suite at runtime. Is this not the case?

Absolutely, this is how we are supposed to do this, and UFS-CCPP supports this completely. The idea to stick everything in one suite runs counter the design of the system and is confusing in my opinion.

[SMoorthi-emc]

While debugging a coupled model, running the coupled executable in uncoupled mode is very useful. That does not mean everyone should be using such capability.

[rsdunlapiv]

@DeniseWorthen The only use of med_id in CMEPS is specific to CESM:

#ifdef CESMCOUPLED
    io_subsystem => shr_pio_getiosys(med_id)
    pio_iotype   =  shr_pio_getiotype(med_id)
    pio_ioformat =  shr_pio_getioformat(med_id)
#else

First, I opened an issue to see if it is really needed at all:
ESCOMP/CMEPS#211

In either case, it seems that you can simply remove the setting of med_id in UFS entirely.

[rsdunlapiv]

@SMoorthi-emc are you saying we should put everything in one SDF?

Why isn't the solution here to compile the model with a SDF for coupled and a SDF for uncoupled and then select the right one at runtime?

[DeniseWorthen]

@DeniseWorthen The only use of med_id in CMEPS is specific to CESM:

#ifdef CESMCOUPLED
    io_subsystem => shr_pio_getiosys(med_id)
    pio_iotype   =  shr_pio_getiotype(med_id)
    pio_ioformat =  shr_pio_getioformat(med_id)
#else

First, I opened an issue to see if it is really needed at all:
ESCOMP/CMEPS#211

In either case, it seems that you can simply remove the setting of med_id in UFS entirely.

I believe I had tried this and my recollection is that it didn't work. The problem wasn't on the CMEPS side, but on the reading of attributes on the NEMS side. But it could have been user error.

[SMoorthi-emc]

Rocky,
No, I am not recommending anyone to put everything in one SDF. However, we need to have the ability to run the coupled executable in uncoupled mode if we want. I have been doing that and will continue to do so as I am aware of what I am doing.
Those who do not want to use that ability are free not to use it. Disabling that possibility is not the right thing. I find it to be very helpful in my work. CCPP is hard enough to debug in the first place.

[rsdunlapiv]

@SMoorthi-emc if we address the CMEPS dependency issues in the driver and compile in both uncoupled and coupled physics suites, it seems we can accomplish what you want. I agree that it is good to be able to run in both modes with a single executable (and really it is more than two, since "coupled" can be a variety of different component options) .

[SMoorthi-emc]

Rocky, if you switch SDF, even if it is in the same executable, the execution does not go through the same path. That is not good enough. I do not understand why this resistance. I can already run with the same executable and same suite. I am not even sure why we are discussing this.

…

On Thu, Jul 1, 2021 at 3:14 PM Rocky Dunlap ***@***.***> wrote: @SMoorthi-emc <https://github.com/SMoorthi-emc> if we address the CMEPS dependency issues in the driver and compile in both uncoupled and coupled physics suites, it seems we can accomplish what you want. I agree that it is good to be able to run in both modes with a single executable (and really it is more than two, since "coupled" can be a variety of different component options) . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYSHO2GE34UEXR5CZG3TVS5BZANCNFSM47U6DINA> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[SMoorthi-emc]

Just to clarify, I am not advocating that every coupled component should be able to run in uncoupled mode using the coupled executable. I am not sure if it is doable. I am only interested in runnnig the standalone atmosphere using coupled executable of ATM/OCN/ICE/WAVE.

[BinLiu-NOAA]

@SMoorthi-emc @rsdunlapiv, For what it might be worth noting, for the UFS-HAFS application, we were also able to retain the this capability. The coupled FV3ATM-HYCOM executable is able to run FV3ATM only, as well as run coupled FV3ATM-HYCOM (for both CMEPS-based and NUOPC based coupling actually) forecast configurations. Whether the ocean component is on or not is controlled by the nems.configure file. And even when both model components (FV3ATM and HYCOM) are on, we have further config/namelist options to control the coupling mode between the two model components (side-by-side no coupling, one-way coupling, or two-way coupling). Besides, for the ongoing HAFS coupling developments adding the WW3 component, I believe we are taking the same strategy and will again try to maintain this backward compatibility (the single executable being able to run the forecast in different modes, with/without coupling).

Thanks a lot for the great collaboration and support among/from EMC and the ESMF team (especially from @danrosen25 @uturuncoglu @rsdunlapiv, etc.) to make this possible. This is at least kind of critical for the HAFS application, since the forecast model could be running with different coupling configurations for different tropical cyclone basins. Also, this provide alternative fallback options for operational failures. Plus, I personally think this is also beneficial for community researchers to promote easier R2O transitions.

A note is that, when building the forecast executable, currently HAFS does build with two or more CCPP suites (with and without the nsst module for example). And of course, the HAFS application workflow knows how to configure and run the forecast executable for different ATMonly/coupling modes.

[uturuncoglu]

It is clear we need to make sure any CMEPS related calls are conditional in module_EARTH_DRIVER.F90. However, we should try to do this without any #ifdef CMEPS statements since I don't think those are really needed and it reduces the flexibility of the driver. The driver is designed to allow you to change the run sequence at runtime from one with only an ATM to one with several components. Ideally, the driver does not know that much about any of the components, including CMEPS, so I advocate removing all #ifdef CMEPS

In the existing NEMS PR (https://github.com/NOAA-EMC/NEMS/pull/106/files) I am checking _modelio:: attribute and reading it based on the return value (isPresent). We could use the same approach for other attribute groups related to CMEPS and try to remove CMEPS CPP flags as mush as possible.

I am not sure why do we need to use med_id but since we restructure the PIO initialization and switch to component level PIO initialization, may be we could get rid of med_id now. If you look at here med_id is only used by CESM model. If you look at also rest of CMEPS it is used by the CMPES driver. So, it could be safe to remove it under UFS. @DeniseWorthen when did you try to remove it? did you try to remove it by using recent version of CMEPS?

[junwang-noaa]

I am not sure if we want to remove CMEPS CPP as it is not needed for some applications (when building pure standalone FV3). Currently UFS has the capability to build only with required components, standalone FV3, FV3-WW3 and FV3-GOCART are built without CMEPS. Without the CMEPS CPP, all those applications need to be built with extra CMEPS component which is not used at all.

…

On Fri, Jul 2, 2021 at 1:39 AM Ufuk Turunçoğlu ***@***.***> wrote: It is clear we need to make sure any CMEPS related calls are conditional in module_EARTH_DRIVER.F90. However, we should try to do this without any #ifdef CMEPS statements since I don't think those are really needed and it reduces the flexibility of the driver. The driver is designed to allow you to change the run sequence at runtime from one with only an ATM to one with several components. Ideally, the driver does not know that much about any of the components, including CMEPS, so I advocate removing all #ifdef CMEPS In the existing NEMS PR (https://github.com/NOAA-EMC/NEMS/pull/106/files) I am checking _modelio:: attribute and reading it based on the return value (isPresent). We could use the same approach for other attribute groups related to CMEPS and try to remove CMEPS CPP flags as mush as possible. I am not sure why do we need to use med_id but since we restructure the PIO initialization and switch to component level PIO initialization, may be we could get rid of med_id now. If you look at here <https://github.com/ESCOMP/CMEPS/blob/d49d38bb49c0eaf442374524303badb739439187/mediator/med_io_mod.F90#L172> med_id is only used by CESM model. If you look at also rest of CMEPS it is used by the CMPES driver. So, it could be safe to remove it under UFS. @DeniseWorthen <https://github.com/DeniseWorthen> when did you try to remove it? did you try to remove it by using recent version of CMEPS? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TJYJZCD7WDZGVZASBLTVVGIFANCNFSM47U6DINA> .