Version 2.4 Release

[gdicker1]

This release features a few improvements and changes:

• EarthWorks is now based on the cesm3_0_beta03 tag from ESCOMP/CESM. The changes from this include:
  • Replaced manage_externals with the git-fleximod tool to finish fetching code after cloning EarthWorks. After cloning EarthWorks, users should now use the command ./bin/git-fleximod update to finish fetching the code.
  • Improvements to externals from their upstreams, especially new input data and access to the CTSM60 and CAM70 configurations.
• Now utilizing a non-hydrostatic version of MPAS-O. This serves as a base for GPU acceleration in future work.
• GPU improvements:
  • GPU CLUBB answer improvements (also available upstream in ESCOMP/CAM:cam_development).
  • Re-enable MPAS-A on GPUs for simple-physics cases only. This is functional with further work coming to improve the results and allow non-simple physics cases to run to completion.
  • Initial GPU acceleration is added to MPAS-O as part of the non-hydrostatic update. This is experimental and may only work in a narrow range of cases.
• Fix MPAS-O and MPAS-SI logging to use appropriate ocn.log.* and ice.log.* files instead of fort.* files.

Description of Model Configurations (Compsets)

See EarthWorks Supported Configurations in the GitHub wiki for more details.

Testing

Tested Systems

NSF NCAR’s Derecho Supercomputer

The tests in this release occurred on Derecho.

CPU-only hardware: Derecho’s CPU-only nodes consisted of dual-socket, 64-core, 3rd Gen AMD EPYC™ 7763 Milan processors with 256 GB of DDR4 memory.

CPU/GPU hybrid hardware: Derecho has GPU nodes consisting of single-socket, 64-core, 3rd Gen AMD EPYC™ 7763 Milan processor with 512 GB of DDR4 memory plus 4 NVIDIA A100 GPUs each with 40 GB of onboard memory.

Tested Software Stacks

Compiler Versions

Derecho:

• ifort (Intel Classic compiler version 2023.2.1)
• ifx (Intel OneAPI version 2023.2.1)
• Nvfortran (NVHPC fortran compiler version 24.3)
• Gnufortran (compiler version 12.2.0)

Libraries

Derecho:

Intel, NVHPC, and GNU compilers

• MPI (Cray MPICH version 8.1.27)
• Parallel-NetCDF (version 1.12.3)
• PIO2 (version 2.6.2)
• ESMF (version 8.6.0)

Testing Results

Derecho create_test Results

To test this release, tests were carried out on Derecho using the ew-rel, ew-pr, and (new) ew-pr-gpu categories as described in the v2.2 release.

ew-rel Results

11 day exact restart tests, CPU-only

• - 120km, 32L CHAOS2000dev with NVHPC and Intel (Overall: FAIL)
• - 120km, 58L CHAOS2000dev with NVHPC and Intel (Overall: FAIL)
• - 60km CHAOS2000dev with NVHPC and Intel (Overall: FAIL)
• - 30km CHAOS2000dev with NVHPC and Intel (Overall: FAIL)
  • For the above 4 tests, NVHPC restart runs failed during initialization (see “CAM runs with MPAS-A, restart issue” in Known Issues below).
  • For the above 4 tests, Intel runs failed when comparing the “base” run to the “restart” run (see “Compsets involving MPAS-O and MPAS-SI” in Known Issues below).
• 15km, 58L CHAOS2000dev with NVHPC and Intel (Overall: FAIL)
  • These tests didn’t make it far through simulation (about 1 day), a reported CLUBB error will be investigated in future releases.

ew-pr Results

11 day exact restart tests, CPU-only

• 120km FHS94 with NVHPC (Overall: FAIL)
• 120km CHAOS2000dev with NVHPC (Overall: FAIL)
  • NVHPC restart runs failed during initialization (see “CAM runs with MPAS-A, restart issue” in Known Issues below).
• 120km FHS94 with Intel, and GNU (Overall: PASS)
• 120km CHAOS2000dev with Intel, and GNU (Overall: FAIL)
  • This is an expected reported failure, the builds and runs failed when comparing the “base” run to the “restart” run (see “Compsets involving MPAS-O and MPAS-SI” in Known Issues below).

5 day smoke tests, CPU-only

• 120km FKESSLER with NVHPC (Overall: PASS)
• 120km QPC6 with NVHPC (Overall: PASS)
• 120km F2000climoEW with NVHPC (Overall: PASS)
• 120km FullyCoupledEW with NVHPC (Overall: PASS)
• 120km CHAOS2000dev with NVHPC, Intel, and GNU (Overall: PASS)

ew-pr-gpu Results

These new tests check some important functionality for GPU compsets. Currently this separately checks a GPU CAM-MPAS dycore-only run and another GPU CAM run that uses the Spectral Element dycore with the GPU physics suites.

• 5 day smoke test, 120km FKESSLER with NVHPC on GPU (Overall: PASS)
• 9 step restart test, 1-deg SE-dycore F2000dev with NVHPC on GPU (Overall: PASS)
  • NOTE: since this test doesn’t use the MPAS-A dycore in CAM, it restarts as expected.

Known Issues

• Attempts to run 15km CHAOS2000dev failed while testing this release
  • Additional details: The Intel run of this compset indicated that CLUBB caused the abort (both NVHPC and Intel abort). This is likely an oversight on “out of the box” configuration and will be addressed in future changes. Please reach out to the development team for help working around this issue.
  • Work around: Message dev. team for exact configuration settings needed.
  • See also: EarthWorksOrg/EarthWorks PR #83
• Compsets involving MPAS-O and MPAS-SI fail to match between default runs and restart runs
  • Additional details: Runs involving the MPAS-O and MPAS-SI components in EarthWorks that begin from restart files of a previous run do not produce the same results. For example, the results are different when comparing a ‘base’ run of 11 simulated days (that writes a restart on day 6) to a ‘restart’ run from day 6 to day 11.
  • Work around: None currently
  • See also: EarthWorksOrg/EarthWorks Issue #66
• Compsets using MPAS-A on GPUs are only supported for simple physics in this release.
  • Additional details: Compsets like FHS94 and FKESSLER are supported but non-simple physics compsets like F2000dev currently abort during run-time when using MPAS-A OpenACC due to the results becoming unrealistic.
  • Work around: Don’t use the MPAS-A dycore in CAM with non-simple physics and compiling to run on GPUs.
  • See also: EarthWorksOrg/EarthWorks Issue #82
• Compsets using MPAS-O on GPUs may fail
  • Additional details: The port of MPAS-O using OpenACC in this release is in an initial, experimental state - not many cases are expected to succeed currently. Cases tested for this release failed during run-time when MPAS-O detects NaNs in its fields and aborts.
  • Work around: Don’t use MPAS-O in cases running on GPUs with this release.
  • See also: EarthWorksOrg/EarthWorks Issue #84
• Memory leak for atm+lnd compsets on Derecho with NVHPC/24.9 stacks or newer
  • Additional details: Investigation showed increasing memory usage per simulated day in cases that had active atmosphere and land components when using nvhpc/24.9 and nvhpc/24.11 software stacks. Testing on Derecho has shown stable memory usage with intel-based software stacks and nvhpc/24.3 software stacks.
  • Resolutions affected: all supported resolution/level combinations.
  • See also: EarthWorksOrg/EarthWorks Issue #90
• NVHPC compilers (tested nvfortran version 24.3, continues from previous releases): Initializing CAM-MPAS from restart fails.
  • Additional details: Any configuration using MPAS-A dycore (tested with QPC6, F2000ClimoEW, FullyCoupledEW) that attempts to restart from a previous run will fail in CAM subroutine dyn_init.
  • Resolutions affected: all supported resolution/level combinations.
  • Work around: Run without restart.
  • See also: EarthWorksOrg/EarthWorks Issue #21

---

This discussion was created from the release Version 2.4 Release.