add ufs-datm-lnd RT baseline check in ctest

[clouden90]

Description

The goal of this PR is to add ufs-datm-lnd RT baseline check in ctest. A new test test_ufs_datm_land with baseline check is added to ctest list. A data retrieve script is also added for non tier-1 machines or within container. The new test test_ufs_datm_land requires at least 7 cores so users will have to run this new test in computing node on NOAA tier-1 machines.

Issue:

Currently JEDI_EXECDIR env is set as ${JEDI_INSTALL}/jedi-bundle/build/bin in test/runtime_vars.sh. However there is no jedi-bundle folder on both Hera and Orion. @jkbk2004 or @ulmononian can you double check that?

How to test

This feature branch has been tested on Hera, Orion, and PW-Clouds within container.

Below is the instruction to run test on Orion:

$ git clone -b feature/baseline-datm-lnd --recursive https://github.com/ufs-community/land-DA_workflow.git
$ cd land-DA_workflow
$ module use modulefiles
$ module load landda_orion.intel
$ mkdir build && cd build
$ ecbuild ../ && make -j4
$  ../test/retrieve_data.sh /work2/noaa/epic-ps/ycteng/case/20230501/20230503/land-DA_workflow
$ salloc --ntasks 8 --exclusive --qos=debug --partition=debug --time=00:30:00 --account=epic-ps
$ module use ../modulefiles && module load landda_orion.intel
$ export JEDI_EXECDIR=${JEDI_INSTALL}/fv3-bundle/build/bin
$ export RTPWD=/work2/noaa/epic-ps/ycteng/case/20230501/20230503/inputs/NEMSfv3gfs/develop-20230413/INTEL
$ export INPUTDATA_ROOT=/work2/noaa/epic-ps/ycteng/case/20230501/20230503/inputs/NEMSfv3gfs/input-data-20221101
$ ctest

Test project /work2/noaa/epic-ps/ycteng/case/20230501/20230503/land-DA_workflow/build
    Start 1: test_vector2tile
1/7 Test #1: test_vector2tile .................   Passed    1.46 sec
    Start 2: test_create_ens
2/7 Test #2: test_create_ens ..................   Passed    1.94 sec
    Start 3: test_letkfoi_snowda
3/7 Test #3: test_letkfoi_snowda ..............   Passed   38.12 sec
    Start 4: test_apply_jediincr
4/7 Test #4: test_apply_jediincr ..............   Passed    4.97 sec
    Start 5: test_tile2vector
5/7 Test #5: test_tile2vector .................   Passed    2.85 sec
    Start 6: test_land_driver
6/7 Test #6: test_land_driver .................   Passed    8.61 sec
    Start 7: test_ufs_datm_land
7/7 Test #7: test_ufs_datm_land ...............   Passed   58.79 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 116.78 sec

Below is the instruction to run test using docker on PW-Clouds:

# pull docker container
$sudo docker pull noaaepic/ubuntu20.04-intel-ue-landda:unified-dev-testmp

# start container
$sudo docker run --rm -v /lustre:/lustre -it docker.io/noaaepic/ubuntu20.04-intel-ue-landda:unified-dev-testmp bash --login

# build land-da
$ cd /lustre
$ source /opt/spack-stack/spack-stack-1.3.0/.bashenv
$ ulimit -s unlimited
$ apt-get install rsync -y
$ git clone -b feature/baseline-datm-lnd --recursive https://github.com/ufs-community/land-DA_workflow.git
$ cd land-DA_workflow
$ mkdir build && cd build
$ module use ../modulefiles
$ module load landda_container.intel
$ export JEDI_EXECDIR=/opt/jedi-bundle/build/bin
$ export JEDI_STATICDIR=/opt/jedi-bundle/fv3-jedi/test/Data
$ export TEST_BASEDIR=/opt/landda-data/cycle_land/DA_GHCN_test/mem000/restarts/vector
$ export RTPWD=/lustre/inputs/NEMSfv3gfs/develop-20230413/INTEL
$ export INPUTDATA_ROOT=/lustre/inputs/NEMSfv3gfs/input-data-20221101
$ ecbuild ../
$ make -j4

# retrieve input data for all tests
$ ../test/retrieve_data.sh /lustre/land-DA_workflow

# run ctest
$ module purge
$ source /opt/spack-stack/spack-stack-1.3.0/.bashenv
$ ctest

Test project /lustre/land-DA_workflow/build
    Start 1: test_vector2tile
1/7 Test #1: test_vector2tile .................   Passed    0.35 sec
    Start 2: test_create_ens
2/7 Test #2: test_create_ens ..................   Passed    0.52 sec
    Start 3: test_letkfoi_snowda
3/7 Test #3: test_letkfoi_snowda ..............   Passed    2.47 sec
    Start 4: test_apply_jediincr
4/7 Test #4: test_apply_jediincr ..............   Passed    0.54 sec
    Start 5: test_tile2vector
5/7 Test #5: test_tile2vector .................   Passed    0.77 sec
    Start 6: test_land_driver
6/7 Test #6: test_land_driver .................   Passed    7.09 sec
    Start 7: test_ufs_datm_land
7/7 Test #7: test_ufs_datm_land ...............   Passed   42.12 sec

100% tests passed, 0 tests failed out of 7

Subcomponents involved:

• DA_update
• vector2tile
• ufs-land-driver
• [ X ] none

Linked PR's and Issues:

Depends on #7

Testing (for CM's):

• RDHPCS
  • Hera
  • Orion
  • Jet
  • Gaea
  • Cheyenne
• CI
  • Completed
• PW-Clouds
  • AWS
  • AZURE
  • GCP

[jkbk2004]

@clouden90 yes, I am aware of hera/orion local jedi-bundle issue. Still WIP to perfectly set up local modulefiles. But tests all good thru singularity. Good to see you show test runs ok thru docker run as well. We can revisit the case for local jedi-bundle build directly thru cmake. I am merging this pr.

[jkbk2004]

@clouden90 yes, I am aware of hera/orion local jedi-bundle issue. Still WIP to perfectly set up local modulefiles. But tests all good thru singularity. Good to see you show test runs ok thru docker run as well. We can revisit the case for local jedi-bundle build directly thru cmake. I am merging this pr.

@ulmononian FYI