Carbon simulation - new restart files needed for GCHP

[Aosrain]

Your name

Longtao Wu

Your affiliation

JPL

What happened? What did you expect to happen?

I’m running a GCHP carbon simulation with single CO2 tracer initialized at 2019/01/01. The simulation runs successfully, but the output CO2 values appear unusual as shown in CO2.txt. It seems that the CO2 values are not being initialized from the restart file. Do you have any idea what might be causing this issue? Thanks.

CO2.txt
gchp.20190101_0000z.log.txt
setCommonRunSettings.sh.txt
ExtData.rc.txt
HEMCO_Config.rc.txt

What are the steps to reproduce the bug?

Setup GCHP carbon simulation with updated CO2 emissions.

Please attach any relevant configuration and log files.

No response

What GCHP version were you using?

GCHP 14.4.0

What environment were you running GCHP on?

Other (please explain below)

What compiler and version were you using?

ifort 19.1.3.304 20200925

What MPI library and version were you using?

MPT: libxmpi.so 'HPE MPT 2.28 04/24/23 12:22:36-mraymond'

Will you be addressing this bug yourself?

Yes

Additional information

No response

[yantosca]

Thanks for writing @Aosrain, I see at the surface level (index 71) the CO2 concentration is 0.6 mol/mol dry. This would be 6e5 ppm if my math is correct. That seems a bit off too.

You could try turning on debug output in CAP.EXTDATA (see logging.yml) on the root core. That should generate output about which files are being read in.

Also tagging @lizziel @hannahnesser

[Aosrain]

Thank you, @yantosca. The allPEs.log file with the duplicate lines removed is attached.
allPEs.log.txt

[lizziel]

Hi @Aosrain - The species in the restart file are being found since they do not appear in the GCHP log file as being bootstrapped, meaning set to zero. (As an aside, the variables that are bootstrapped are probably not needed for carbon. We should bracket those out with simulation type to reduce the size of the non-fullchem restart files).

 Character Resource Parameter: GCHPchem_INTERNAL_RESTART_FILE:gchp_restart.nc4
 Character Resource Parameter: MAPL_ENABLE_BOOTSTRAP:YES
 Using parallel NetCDF for file: gchp_restart.nc4
   Bootstrapping Variable: ARCHV_DRY_TOTN in gchp_restart.nc4
   Bootstrapping Variable: ARCHV_WET_TOTN in gchp_restart.nc4
   Bootstrapping Variable: AREA in gchp_restart.nc4
   Bootstrapping Variable: AeroH2O_SNA in gchp_restart.nc4
   Bootstrapping Variable: DEP_RESERVOIR in gchp_restart.nc4
   Bootstrapping Variable: DRYPERIOD in gchp_restart.nc4
   Bootstrapping Variable: DryDepNitrogen in gchp_restart.nc4
   Bootstrapping Variable: GCCTROPP in gchp_restart.nc4
   Bootstrapping Variable: GWET_PREV in gchp_restart.nc4
   Bootstrapping Variable: H2O2AfterChem in gchp_restart.nc4
   Bootstrapping Variable: JNO2 in gchp_restart.nc4
   Bootstrapping Variable: JOH in gchp_restart.nc4
   Bootstrapping Variable: KPPHvalue in gchp_restart.nc4
   Bootstrapping Variable: LAI_PREVDAY in gchp_restart.nc4
   Bootstrapping Variable: ORVCSESQ in gchp_restart.nc4
   Bootstrapping Variable: PARDF_DAVG in gchp_restart.nc4
   Bootstrapping Variable: PARDR_DAVG in gchp_restart.nc4
   Bootstrapping Variable: PFACTOR in gchp_restart.nc4
   Bootstrapping Variable: SO2AfterChem in gchp_restart.nc4
   Bootstrapping Variable: STATE_PSC in gchp_restart.nc4
   Bootstrapping Variable: T_DAVG in gchp_restart.nc4
   Bootstrapping Variable: T_PREVDAY in gchp_restart.nc4
   Bootstrapping Variable: WetDepNitrogen in gchp_restart.nc4

There are also no messages in the log that they are being set to background values which further supports that they are being properly read.

That being said, we have not done a GCHP carbon simulation benchmark so there may be bugs. @yantosca, @hannahnesser: are there plans for validating the GCHP carbon simulations via comparison with GC-Classic?

[Aosrain]

The goal of this GCHP carbon simulation is to compare it with Hannah's GC-classic simulation. Could you please suggest how to troubleshoot this issue in GCHP, as I am relatively new to using it? @lizziel @yantosca @hannahnesser

[lizziel]

That's great you are comparing to GC-Classic. Here are a few ideas for things to look at:

1. Compare the emissions collection diagnostics to check that the emissions match up correctly. Compare other diagnostic collections as well looking for differences.
2. Compare initial and final restart files for a very short run, e.g. 1hr or even 20min. Do they look very different?
3. Turn off all components in geoschem_config.yml (and emissions in HEMCO_Config.rc). Does GC-Classic and GCHP match now? Then turn on chemistry, rerun and compare. Isolate which component has the issue in this way.

[Aosrain]

@lizziel @yantosca I have conducted a 20-minute simulation, which confirms that the unusual values are coming from the restart file. I have checked multiple restart files on the GCHP website (http://geoschemdata.wustl.edu/ExtData/GEOSCHEM_RESTARTS/), and none of them have CO2 values in the range of 400 ppm. Could you provide a GCHP restart file with accurate CO2 values? Alternatively, I could try interpolating Hannah's GC-Classic restart file from 47 levels to 72 levels. Do you have a code for this interpolation? Thank you.

[lizziel]

I diffed the GC-Classic carbon restart file with the GCHP c24 restart file and they are the same apart from regridding noise.

Are you seeing different values than these in the restart file you are using?

[Aosrain]

@lizziel Apologies, I checked a wrong restart file. The values in the carbon restart file appear correct, as shown in the left panel of the figure. The middle panel displays the CO2 values at the bottom layer 10 minutes post-initialization, with all components in geoschem_config.yml disabled. The right panel shows the CO2 emissions. It seems to me that the restart file isn't utilized in the simulation. Any suggestions on how to resolve this issue?

[lizziel]

HI @Aosrain, I was able to reproduce this and investigate. I think the issue is application of the restart variable DELP_DRY within subroutine AIRQNT. It is used to adjust mixing ratios for mass conservation. The factor is a ratio of delta pressure in the restart file and new delta pressure (current time). Somehow these two quantities are vertically flipped relative to each other. I will let you know when I find the fix.

[lizziel]

Following up, I traced the problem to the lev dimension values in the restart file. For GCHP they should be indexes 1 to 72, but they are instead pressure levels in hPa. @yantosca is creating new carbon restart files for GCHP.

[lizziel]

@yantosca created new restart files and replaced the old ones. You can download the files at http://geoschemdata.wustl.edu/ExtData/GEOSCHEM_RESTARTS/v2023-01/.

[yantosca]

@Aosrain, I've confirmed the new files are there. We recenty updated the GCPy file_regrid.py script. I think the problematic files in the v2023-01 folder may have been generated with another method. @lizziel and I have confirmed that the regenerated files have the proper coordinate values for lev (indices instead of pressures).

[Aosrain]

@lizziel @yantosca Thank you very much! It works well now.