opnReqTest does not work with cpld_bmark_p7

[JessicaMeixner-NOAA]

Description

The opnReqTest does not work with cpld_bmark_p7

To Reproduce:

from test directory, run:

./opnReqTest -n cpld_bmark_p7 -k -c mpi,thr,dcp

Additional context

Add any other context about the problem here.
Directly reference any issues or PRs in this or other repositories that this is related to, and describe how they are related. Example:

• needs to be fixed also in noaa-emc/nems/issues/<issue_number>
• needed for noaa-emc/fv3atm/pull/<pr_number>

Output

./opnReqTest -n cpld_bmark_p7 -k -c mpi,thr,dcp
hfe11
Machine:  hera.intel     Account:  nems
test name: cpld_bmark_p7
cases to compile: std
compiling std with compile option -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1
done compiling std
cases to run: std_base mpi thr dcp
Running test for std_base with
    THRD: 1; INPES: 6; JNPES: 8; TASKS: 560; TPN: 40
Coupled application not yet implemented for mpi
Died with error code 1

[MinsukJi-NOAA]

@JessicaMeixner-NOAA can you try without mpi? ./opnReqTest -n cpld_bmark_p7 -k -c thr,dcp. Since you must have the executable already created, you can add -x flag as well to skip compilation

[junwang-noaa]

@JessicaMeixner-NOAA <https://github.com/JessicaMeixner-NOAA> cpld_bmark_p7 has the wave component, we don't expect it will work with dcp either. Are you running the test for debugging?

…

On Thu, Nov 4, 2021 at 3:55 PM Minsuk Ji ***@***.***> wrote: @JessicaMeixner-NOAA <https://github.com/JessicaMeixner-NOAA> can you try without mpi? ./opnReqTest -n cpld_bmark_p7 -k -c thr,dcp — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#899 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TJZ37LEP5NKL2RVBJ3UKLXJ3ANCNFSM5HMHTZPQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JessicaMeixner-NOAA]

Trying to be proactive in showing that WW3 can pass more of these tests now with some upcoming code changes. I have also added new tests, but had added those before the opnReqTest was here to use so wanted to try this as well as extra tests might not be wanted in the repository.

I'll try just:
./opnReqTest -n cpld_bmark_p7 -k -c thr
and report back

[JessicaMeixner-NOAA]

@MinsukJi-NOAA

$ ./opnReqTest -n cpld_bmark_p7 -k -c thr,dcp
hecflow01
Machine:  hera.intel     Account:  nems
test name: cpld_bmark_p7
cases to compile: std
compiling std with compile option -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1
done compiling std
cases to run: std_base thr dcp
Running test for std_base with
    THRD: 1; INPES: 6; JNPES: 8; TASKS: 560; TPN: 40
Running test for thr with
    THRD: 2; INPES: 3; JNPES: 4; TASKS: 138; TPN: 20
Died with error code 1

Should I try cpld_control_wave_p7 instead? Or just cpld_control_p7 works?

[MinsukJi-NOAA]

@MinsukJi-NOAA

$ ./opnReqTest -n cpld_bmark_p7 -k -c thr,dcp
hecflow01
Machine:  hera.intel     Account:  nems
test name: cpld_bmark_p7
cases to compile: std
compiling std with compile option -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1
done compiling std
cases to run: std_base thr dcp
Running test for std_base with
    THRD: 1; INPES: 6; JNPES: 8; TASKS: 560; TPN: 40
Running test for thr with
    THRD: 2; INPES: 3; JNPES: 4; TASKS: 138; TPN: 20
Died with error code 1

Should I try cpld_control_wave_p7 instead? Or just cpld_control_p7 works?

@JessicaMeixner-NOAA
The number of tasks does not look right for thr. It may have been hard coded for cpld_control_p7. Can you please try cpld_control_wave_p7, which I believe should have the same tile layout as cpld_control_p7? If that does not work, I will work on making cpld_control_wave_p7 work with opnReqTest script; I am not sure if the benchmark test is a good candidate for the opnReqTest script.

[JessicaMeixner-NOAA]

Looks like the cpld_control_wave_p7 also does not work:

./opnReqTest -n cpld_control_wave_p7 -k -c thr
hecflow01
Machine:  hera.intel     Account:  nems
test name: cpld_control_wave_p7
cases to compile: std
compiling std with compile option -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1
done compiling std
cases to run: std_base thr
Running test for std_base with
    THRD: 1; INPES: 3; JNPES: 8; TASKS: 200; TPN: 40
Running test for thr with
    THRD: 2; INPES: 3; JNPES: 4; TASKS: 120; TPN: 20
Died with error code 1

I'm curious as to why the benchmark test would not be a good idea? Sometimes the components have features that are resolution dependent, so I would think having the capability to run any resolution might be important.

[MinsukJi-NOAA]

I agree that different resolutions could be important. However, I am not sure if the operation test should deal with the high number of PE's used by benchmark to meet forecast time requirement..

[JessicaMeixner-NOAA]

So is the recommendation then that we stick to manual testing of the higher resolution test cases to ensure the resolutions meet the operational requirements?

Here are two cases where resolution has an impact (there are likely more, and likely other reasons and arguments for just testing the lower):

• Ocean physics is different at the 1 deg versus 1/4 deg resolution
• Currently, the wave model passes the MPI test at the lower resolution, but not at the higher resolution which exposed an issue. So, in this case, the wave model would appear to pass the operational requirements, but for operational resolutions, it would in fact not pass.

[junwang-noaa]

Thanks, I think the opnReqTest needs to support high resolution tests too, e,g, the cpld_bmark cases.  But please keep in mind that the developer won't be able to run those tests or need to require a special queue if the high resolution tests require a large amount of nodes that developers may not be able to get access to.

[MinsukJi-NOAA]

I agree that different resolutions could be important. However, I am not sure if the operation test should deal with the high number of PE's used by benchmark to meet forecast time requirement..

@JessicaMeixner-NOAA Maybe my previous comment was not clear. I agree with you on testing different resolutions. My uncertainty was in regards to using the benchmark test; here is an example. If cpld_control_p7 and cpld_bmark_p7 have the same resolutions but different number of PETs (since benchmark needs to run faster to meet the run time requirement), should we test operation requirement on cpld_control_p7?

[JessicaMeixner-NOAA]

The cpld_control_p7 is a C96/1 deg test, whereas cpld_bmark_p7 is C384/0.25 deg + waves test, so I think the different resource requirements between the two is due to the resolution differences. The cpld_control_c384_p7 and cpld_bmark_p7 have the same atm/ocn resolution but the bmark test also includes waves.

[MinsukJi-NOAA]

ORT changes for cpld_bmark_p7 has been merged via #931