Skip to content

Commit

Permalink
Merge branch 'feature/remove_mct'
Browse files Browse the repository at this point in the history 
Start bounds at 1; remove references to MCT

Users: take note of (1), and in particular the caveat for users noted
below (in the "Notes of particular relevance for users").

(1) All global index arrays on a given processor now have a starting index of 1
  - bounds_proc for each subgrid level has a starting index of 1 for each level
  - bounds_clump for each subgrid level has a starting index of 1 for just
    the first clump on the processor - but all the other clumps on the
    processor do not start at 1 - but rather are offset with the number of
    gridcells, columns, ...etc on the preceeding clumps

(2) There are no longer any references to any mct data structures other than in the mct cap
  - All references to gsmap have been removed from decompMod.F90 and
    replaced with new global index arrays for the various subgrid levels
  - decompInitMod has been refactored to calculated these global index
    arrays using pure MPI rather than mct
  - ncdio_pio_F90.in has been refactored to use the new global index
    arrays rather than the gsmap data structures
  - the data struture ldecomp is no longer needed
  - the module spmdGathScatMod.F90 is no longer needed and has been removed
  • Loading branch information
@billsacks
billsacks committed Jul 8, 2021
2 parents e0e8451 + 9ea37c3 commit 515a31b
Show file tree
Hide file tree
Showing 26 changed files with 929 additions and 1,501 deletions.
97 changes: 97 additions & 0 deletions doc/ChangeLog
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,101 @@
===============================================================
Tag name: ctsm5.1.dev047
Originator(s): mvertens (Mariana Vertenstein)
Date: Thu Jul 8 12:03:31 MDT 2021
One-line Summary: Start bounds at 1; remove references to MCT

Purpose and description of changes
----------------------------------

Users: take note of (1), and in particular the caveat for users noted
below (in the "Notes of particular relevance for users").

(1) All global index arrays on a given processor now have a starting index of 1
- bounds_proc for each subgrid level has a starting index of 1 for each level
- bounds_clump for each subgrid level has a starting index of 1 for just
the first clump on the processor - but all the other clumps on the
processor do not start at 1 - but rather are offset with the number of
gridcells, columns, ...etc on the preceeding clumps

(2) There are no longer any references to any mct data structures other than in the mct cap
- All references to gsmap have been removed from decompMod.F90 and
replaced with new global index arrays for the various subgrid levels
- decompInitMod has been refactored to calculated these global index
arrays using pure MPI rather than mct
- ncdio_pio_F90.in has been refactored to use the new global index
arrays rather than the gsmap data structures
- the data struture ldecomp is no longer needed
- the module spmdGathScatMod.F90 is no longer needed and has been removed

Significant changes to scientifically-supported configurations
--------------------------------------------------------------

Does this tag change answers significantly for any of the following physics configurations?
(Details of any changes will be given in the "Answer changes" section below.)

[Put an [X] in the box for any configuration with significant answer changes.]

[ ] clm5_1

[ ] clm5_0

[ ] ctsm5_0-nwp

[ ] clm4_5


Bugs fixed or introduced
------------------------
Issues fixed (include CTSM Issue #):
- Addresses the first part of ESCOMP/CTSM#293 (Rework threading to be
done at a higher level, simplifying array argument passing)

Notes of particular relevance for users
---------------------------------------
Caveats for users (e.g., need to interpolate initial conditions):
- Until now, if the model produced an error message with a gridcell or
subgrid index (e.g., "Error at p = 1234"), you could rerun the model
with write statements in conditionals like "if (p == 1234) then". With
the changes in this tag, this simple conditional is no longer
possible. Instead, you will need to also reference the processor
number that produces the error.

If the error message is produced in the lnd log file, then you can
have a conditional like:
if (masterproc .and. p == 1234) then

If the error message is produced in the cesm log file, then you can
have a conditional like:
if (iam == X .and. p == 1234) then

where X is the processor number that produced the message. On some
machines (including cheyenne), this processor number is printed at the
start of each line in the cesm log file. In addition, some aborts will
now print "proc_id = X", and that value can be used.

Soon we will update
https://escomp.github.io/ctsm-docs/versions/master/html/users_guide/trouble-shooting/trouble-shooting.html
to document the new recommended procedure.

Testing summary:
----------------
regular tests (aux_clm: https://github.com/ESCOMP/CTSM/wiki/System-Testing-Guide#pre-merge-system-testing):

cheyenne ---- ok
izumi ------- ok

Answer changes
--------------

Changes answers relative to baseline: NO

Other details
-------------
Pull Requests that document the changes (include PR ids):
https://github.com/ESCOMP/CTSM/pull/1420

===============================================================
===============================================================
Tag name: ctsm5.1.dev046
Originator(s): rgknox (Ryan Knox,,,)
Date: Fri Jul 2 14:36:44 MDT 2021
Expand Down
1 change: 1 addition & 0 deletions doc/ChangeSum
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
Tag Who Date Summary
============================================================================================================================
ctsm5.1.dev047 mvertens 07/08/2021 Start bounds at 1; remove references to MCT
ctsm5.1.dev046 rgknox 07/02/2021 Updating external fates has to tag sci.1.46.2_api.16.1.0
ctsm5.1.dev045 slevis 06/29/2021 Include CWD in heterotrophic respiration
ctsm5.1.dev044 mvertens 06/24/2021 New stream functionality when using NUOPC or LILAC
Expand Down
4 changes: 2 additions & 2 deletions src/cpl/lilac/lnd_comp_esmf.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ module lnd_comp_esmf

! ctsm code
use spmdMod , only : masterproc, spmd_init, mpicom
use decompMod , only : bounds_type, ldecomp, get_proc_bounds
use decompMod , only : bounds_type, get_proc_bounds
use domainMod , only : ldomain
use controlMod , only : control_setNL
use clm_varorb , only : eccen, obliqr, lambm0, mvelpp
Expand Down Expand Up @@ -342,7 +342,7 @@ subroutine lnd_init(comp, import_state, export_state, clock, rc)
call ESMF_LogWrite(subname//"ctsm initialize1 done...", ESMF_LOGMSG_INFO)

!----------------------
! Initialize decomposition (ldecomp) and domain (ldomain) types and generate land mesh
! Initialize decomposition and domain (ldomain) types and generate land mesh
!----------------------
! TODO: generalize this so that a mask mesh is read in like for nuopc/cmeps
! For now set the meshfile_mask equal to the model_meshfile
Expand Down
122 changes: 60 additions & 62 deletions src/cpl/mct/FireDataBaseType.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,19 @@ module FireDataBaseType
! module for handling of fire data
!
! !USES:
use shr_kind_mod , only : r8 => shr_kind_r8, CL => shr_kind_CL
use shr_strdata_mod , only : shr_strdata_type, shr_strdata_create, shr_strdata_print
use shr_strdata_mod , only : shr_strdata_advance
use shr_log_mod , only : errMsg => shr_log_errMsg
use clm_varctl , only : iulog, inst_name
use spmdMod , only : masterproc, mpicom, comp_id
use fileutils , only : getavu, relavu
use decompMod , only : gsmap_lnd_gdc2glo
use domainMod , only : ldomain
use abortutils , only : endrun
use decompMod , only : bounds_type
use shr_kind_mod , only : r8 => shr_kind_r8, CL => shr_kind_CL
use shr_strdata_mod , only : shr_strdata_type, shr_strdata_create, shr_strdata_print
use shr_strdata_mod , only : shr_strdata_advance
use shr_log_mod , only : errMsg => shr_log_errMsg
use clm_varctl , only : iulog, inst_name
use spmdMod , only : masterproc, mpicom, comp_id
use fileutils , only : getavu, relavu
use domainMod , only : ldomain
use abortutils , only : endrun
use decompMod , only : bounds_type
use FireMethodType , only : fire_method_type
use lnd_set_decomp_and_domain, only : gsmap_global
use mct_mod
use FireMethodType , only : fire_method_type
!
implicit none
private
Expand All @@ -38,7 +38,6 @@ module FireDataBaseType
type(shr_strdata_type) :: sdat_hdm ! Human population density input data stream
type(shr_strdata_type) :: sdat_lnfm ! Lightning input data stream


contains
!
! !PUBLIC MEMBER FUNCTIONS:
Expand Down Expand Up @@ -171,7 +170,6 @@ subroutine hdm_init( this, bounds, NLFilename )
character(len=CL) :: popdensmapalgo = 'bilinear' ! mapping alogrithm for population density
character(len=CL) :: popdens_tintalgo = 'nearest'! time interpolation alogrithm for population density
character(len=CL) :: stream_meshfile_popdens ! not used

character(*), parameter :: subName = "('hdmdyn_init')"
character(*), parameter :: F00 = "('(hdmdyn_init) ',4a)"
!-----------------------------------------------------------------------
Expand Down Expand Up @@ -225,31 +223,31 @@ subroutine hdm_init( this, bounds, NLFilename )

call clm_domain_mct (bounds, dom_clm)

call shr_strdata_create(this%sdat_hdm,name="clmhdm", &
pio_subsystem=pio_subsystem, &
pio_iotype=shr_pio_getiotype(inst_name), &
mpicom=mpicom, compid=comp_id, &
gsmap=gsmap_lnd_gdc2glo, ggrid=dom_clm, &
nxg=ldomain%ni, nyg=ldomain%nj, &
yearFirst=stream_year_first_popdens, &
yearLast=stream_year_last_popdens, &
yearAlign=model_year_align_popdens, &
offset=0, &
domFilePath='', &
domFileName=trim(stream_fldFileName_popdens), &
domTvarName='time', &
domXvarName='lon' , &
domYvarName='lat' , &
domAreaName='area', &
domMaskName='mask', &
filePath='', &
call shr_strdata_create(this%sdat_hdm,name="clmhdm", &
pio_subsystem=pio_subsystem, &
pio_iotype=shr_pio_getiotype(inst_name), &
mpicom=mpicom, compid=comp_id, &
gsmap=gsmap_global, ggrid=dom_clm, &
nxg=ldomain%ni, nyg=ldomain%nj, &
yearFirst=stream_year_first_popdens, &
yearLast=stream_year_last_popdens, &
yearAlign=model_year_align_popdens, &
offset=0, &
domFilePath='', &
domFileName=trim(stream_fldFileName_popdens), &
domTvarName='time', &
domXvarName='lon' , &
domYvarName='lat' , &
domAreaName='area', &
domMaskName='mask', &
filePath='', &
filename=(/trim(stream_fldFileName_popdens)/) , &
fldListFile='hdm', &
fldListModel='hdm', &
fillalgo='none', &
mapalgo=popdensmapalgo, &
calendar=get_calendar(), &
tintalgo=popdens_tintalgo, &
fldListFile='hdm', &
fldListModel='hdm', &
fillalgo='none', &
mapalgo=popdensmapalgo, &
calendar=get_calendar(), &
tintalgo=popdens_tintalgo, &
taxmode='extend' )

if (masterproc) then
Expand Down Expand Up @@ -381,31 +379,31 @@ subroutine lnfm_init( this, bounds, NLFilename )

call clm_domain_mct (bounds, dom_clm)

call shr_strdata_create(this%sdat_lnfm,name="clmlnfm", &
pio_subsystem=pio_subsystem, &
call shr_strdata_create(this%sdat_lnfm,name="clmlnfm", &
pio_subsystem=pio_subsystem, &
pio_iotype=shr_pio_getiotype(inst_name), &
mpicom=mpicom, compid=comp_id, &
gsmap=gsmap_lnd_gdc2glo, ggrid=dom_clm, &
nxg=ldomain%ni, nyg=ldomain%nj, &
yearFirst=stream_year_first_lightng, &
yearLast=stream_year_last_lightng, &
yearAlign=model_year_align_lightng, &
offset=0, &
domFilePath='', &
domFileName=trim(stream_fldFileName_lightng), &
domTvarName='time', &
domXvarName='lon' , &
domYvarName='lat' , &
domAreaName='area', &
domMaskName='mask', &
filePath='', &
filename=(/trim(stream_fldFileName_lightng)/),&
fldListFile='lnfm', &
fldListModel='lnfm', &
fillalgo='none', &
tintalgo=lightng_tintalgo, &
mapalgo=lightngmapalgo, &
calendar=get_calendar(), &
mpicom=mpicom, compid=comp_id, &
gsmap=gsmap_global, ggrid=dom_clm, &
nxg=ldomain%ni, nyg=ldomain%nj, &
yearFirst=stream_year_first_lightng, &
yearLast=stream_year_last_lightng, &
yearAlign=model_year_align_lightng, &
offset=0, &
domFilePath='', &
domFileName=trim(stream_fldFileName_lightng), &
domTvarName='time', &
domXvarName='lon' , &
domYvarName='lat' , &
domAreaName='area', &
domMaskName='mask', &
filePath='', &
filename=(/trim(stream_fldFileName_lightng)/), &
fldListFile='lnfm', &
fldListModel='lnfm', &
fillalgo='none', &
tintalgo=lightng_tintalgo, &
mapalgo=lightngmapalgo, &
calendar=get_calendar(), &
taxmode='cycle' )

if (masterproc) then
Expand Down
45 changes: 19 additions & 26 deletions src/cpl/mct/SoilMoistureStreamMod.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,26 +16,24 @@ module SoilMoistureStreamMod
! Read in soil moisture from data stream
!
! !USES:
use shr_strdata_mod , only : shr_strdata_type, shr_strdata_create
use shr_strdata_mod , only : shr_strdata_print, shr_strdata_advance
use shr_kind_mod , only : r8 => shr_kind_r8
use shr_kind_mod , only : CL => shr_kind_CL, CXX => shr_kind_CXX
use shr_log_mod , only : errMsg => shr_log_errMsg
use decompMod , only : bounds_type
use abortutils , only : endrun
use clm_varctl , only : iulog, use_soil_moisture_streams, inst_name
use clm_varcon , only : grlnd
use controlMod , only : NLFilename
use decompMod , only : gsMap_lnd2Dsoi_gdc2glo
use domainMod , only : ldomain
use fileutils , only : getavu, relavu
use LandunitType , only : lun
use ColumnType , only : col
use SoilStateType , only : soilstate_type
use WaterStateBulkType, only : waterstatebulk_type
use perf_mod , only : t_startf, t_stopf
use spmdMod , only : masterproc
use spmdMod , only : mpicom, comp_id
use shr_strdata_mod , only : shr_strdata_type, shr_strdata_create
use shr_strdata_mod , only : shr_strdata_print, shr_strdata_advance
use shr_kind_mod , only : r8 => shr_kind_r8
use shr_kind_mod , only : CL => shr_kind_CL, CXX => shr_kind_CXX
use shr_log_mod , only : errMsg => shr_log_errMsg
use decompMod , only : bounds_type
use abortutils , only : endrun
use clm_varctl , only : iulog, use_soil_moisture_streams, inst_name
use clm_varcon , only : grlnd
use controlMod , only : NLFilename
use domainMod , only : ldomain
use LandunitType , only : lun
use ColumnType , only : col
use SoilStateType , only : soilstate_type
use WaterStateBulkType , only : waterstatebulk_type
use perf_mod , only : t_startf, t_stopf
use spmdMod , only : masterproc, mpicom, comp_id
use lnd_set_decomp_and_domain , only : gsMap_lnd2Dsoi_gdc2glo
use mct_mod
use ncdio_pio
!
Expand Down Expand Up @@ -127,8 +125,7 @@ subroutine PrescribedSoilMoistureInit(bounds)

! Read soilm_streams namelist
if (masterproc) then
nu_nml = getavu()
open( nu_nml, file=trim(NLFilename), status='old', iostat=nml_error )
open( newunit=nu_nml, file=trim(NLFilename), status='old', iostat=nml_error )
call find_nlgroup_name(nu_nml, 'soil_moisture_streams', status=nml_error)
if (nml_error == 0) then
read(nu_nml, nml=soil_moisture_streams,iostat=nml_error)
Expand All @@ -139,7 +136,6 @@ subroutine PrescribedSoilMoistureInit(bounds)
call endrun(subname // ':: ERROR finding soilm_streams namelist')
end if
close(nu_nml)
call relavu( nu_nml )
endif

call shr_mpi_bcast(stream_year_first_soilm, mpicom)
Expand Down Expand Up @@ -170,11 +166,8 @@ subroutine PrescribedSoilMoistureInit(bounds)

call clm_domain_mct (bounds, dom_clm, nlevels=nlevsoi)

!
! create the field list for these fields...use in shr_strdata_create
!
fldList = trim(soilmString)

if (masterproc) write(iulog,*) 'fieldlist: ', trim(fldList)

call shr_strdata_create(sdat_soilm,name="soil_moisture", &
Expand Down
Loading

0 comments on commit 515a31b

Please sign in to comment.