Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write TPW variables into a new file every timestep #9

Merged
merged 1 commit into from
Feb 7, 2024

Conversation

hollyhan
Copy link
Collaborator

@hollyhan hollyhan commented Jan 2, 2024

Previously, the sea-level model variables related to true polar wander (TPW) at every time step were written out to a single text file. While this way poses no problem and makes the output folder less crowded, it is problematic when simulations have to be restarted in the middle of run because the variables are appended to the existing file rather than replacing the lines that have been written previously. For example, if a run dies at timestep 5 and gets restarted at timestep 3, the TPW variables calculated at timestep 3 in the restart run will be appended to the TPW variables at timestep 4 written from the previous run before it died. Unfortunately, Fortran doesn't allow overwriting specific lines of files in its sequential file I/O functionality, and the most straightforward & easiest solution is to write a new file for the TPW variables at every single timestep such the new file replaces the old file if it exists.

Below is the testing result on Chicoma (/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel)
Inputs: cylindrical ice sheet retreating at rate of 1 km/yr
length of simulation: 4 years for every 1 year time interval (i.e. 4 timesteps)

TPW file indicating TPW variables at each timestep (0,1,2,3,4) using the original code before the changes in this PR:

(dev_compass_1.2.0) hollyhan@ch-fe1:/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel> ls OUTPUT_SLM/TPW*
OUTPUT_SLM/TPW

(dev_compass_1.2.0) hollyhan@ch-fe1:/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel> vi OUTPUT_SLM/TPW 

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00
     0.00000000E+00     0.00000000E+00     0.00000000E+00
     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     6.19660267E+29     1.14036696E+29     4.63872184E+29
     2.38865866E-06     4.39587230E-07     4.01506090E-09
     5.78175313E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00    -2.58283154E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     9.41311183E-02    -1.73230434E-02     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -1.08616061E-07     4.13788374E-08

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     1.42700389E+30     1.09646876E+29     8.79664576E+29
     5.51025650E-06     4.24406288E-07     7.61778706E-09
     1.09836602E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00    -4.89730082E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     2.17145554E-01    -1.67248001E-02     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -5.94714791E-07     9.21579378E-08

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     1.76286578E+30     1.70720170E+29     1.14045829E+30
     6.82508543E-06     6.61204407E-07     9.88121339E-09
     1.42524925E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00    -6.35121512E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     2.68959705E-01    -2.60564272E-02     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -9.09222159E-07     1.77837340E-07

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     2.30958745E+30     4.72491673E+28     1.28543831E+30
     8.95440405E-06     1.87322423E-07     1.11424425E-08
     1.60912347E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00    -7.15750314E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     3.52870876E-01    -7.38191254E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -1.57918278E-06     6.61006268E-08

(Note: each block of outputs represent the TPW values at each timestep)
Example visualization: changes in the gravitational field at timestep 4
Screen Shot 2024-01-02 at 3 45 48 PM

TPW# file at timestep 1 and 4 using the code in this PR:

(dev_compass_1.2.0) hollyhan@ch-fe1:/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel> ls OUTPUT_SLM_TEST/TPW*
OUTPUT_SLM_TEST/TPW0  OUTPUT_SLM_TEST/TPW1  OUTPUT_SLM_TEST/TPW2  OUTPUT_SLM_TEST/TPW3	OUTPUT_SLM_TEST/TPW4

*Note that separate files are created at each timestep.

 
(dev_compass_1.2.0) hollyhan@ch-fe2:/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel/OUTPUT_SLM_TEST> vi TPW1

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     6.19660267E+29     1.14036696E+29     4.63872184E+29
     2.38865866E-06     4.39587230E-07     4.01506090E-09
     5.78175313E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00    -2.58283154E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     9.41311183E-02    -1.73230434E-02     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -1.08616061E-07     4.13788374E-08
(dev_compass_1.2.0) hollyhan@ch-fe2:/lustre/scratch4/turquoise/hollyhan/compass/MALI-Dev/components/mpas-albany-landice/src/SeaLevelModel/OUTPUT_SLM_TEST> vi TPW4

     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     2.30958745E+30     4.72491673E+28     1.28543831E+30
     8.95440405E-06     1.87322423E-07     1.11424425E-08
     1.60912347E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00    -7.15750314E-04     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00     3.52870876E-01    -7.38191254E-03     0.00000000E+00     0.00000000E+00     0.00000000E+00     0.00000000E+00    -1.57918278E-06     6.61006268E-08

Example visualization: changes in the gravitational field at timestep 4

Screen Shot 2024-01-02 at 3 47 41 PM

Difference in the Gfield (original code result minus the modified code result):
image

Copy link

@matthewhoffman matthewhoffman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hollyhan , thanks for describing these changes so carefully. I made one comment that you may want to address or not, but I'm otherwise happy for you to proceed with merging when you're ready.

@@ -1296,6 +1273,20 @@ subroutine sl_solver(itersl, iter, dtime, starttime, mali_iceload, mali_mask, sl
write(201,'(I4)') nmelt
close(201)

! number of iteration it takes for the inner convergence
open(unit = 201, file = trim(outputfolder)//'numiter', form = 'formatted', access ='sequential', &
& status = 'old', position='append')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this file have the same issue - that if a restart causes some time steps to be rerun there will be multiple copies of those lines? Maybe that doesn't matter much for this file but wanted to point it out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewhoffman
Yes - this file will have the same issue, along with other files like times as well. These two files won't affect the SLM calculation itself but affect post-processing if used. But those files are rarely used nor looked at. So maybe I can either just stop writing those into a file in the first place, or we just keep it as is for now and address this issue in a new PR. The new PR could be even on making all the I/O to happen in the netcdf-format (because it's been proven that the text-formatted files take up way too much space (as we've recently seen in my project directory on Chicoma), and the remaining issue will be automatically taken care of. What do you think?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with that limitation with this file for now if you are. I don't imagine I will be looking at them, so if you don't think they are critical, that's ok with me.

@hollyhan hollyhan merged commit 4e4e077 into MALI-Dev:main Feb 7, 2024
@hollyhan hollyhan deleted the fix_IO_when_restart branch February 7, 2024 04:10
matthewhoffman added a commit to MALI-Dev/E3SM that referenced this pull request Mar 1, 2024
…Dev/develop

Previously, when the regional sea-level prediction capability was added
to MALI (#21), the restart config option for the sea-level model was not
added. This led the sea-level model to get initialized to Timestep zero
when coupled MALI-SLM simulations are being restarted, forgetting about
the ice loading changes and associated viscoelastic solid earth deformation
that happened in the timesteps prior to current model time. This PR
fixes the problem by allowing the sea-level model to resume where it
was left off. Note in parallel to this PR, the version of the SLM needs
to incorporate the changes made in the following accompanying
PR (MALI-Dev/1DSeaLevelModel_FWTW#9)

* hollyhan/add_restart_functionality_slm:
  Don't call SLM on init of a restart
  Add addl info on restart about the calculated time since last SLM call
  Allow restarts at any interval when using SLM
  Add missing error flag so model actually dies when error occurs
  Add missing arguments to log write statement
  Update restart check to also use time interval division
  Adjust check if adaptive dt is on or not
  Update checks using interval division
  Improve error handling, correct other usage of config_uplift_method
  Improve synchronization of timesteps between MALI and SLM
  Add restart option when the SLM is coupled to MALI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants