File was corrupted on glade

[ekluzek]

Brief summary of bug

This will likely cause answers for the ERP_D_Ld3.f09_g17.I2000Clm50SpGs.cheyenne_intel.clm-prescribed tests to change.

General bug information

CTSM version you are using: This is outside of ctsm and is for a file but applies to ctsm1.0.dev113
Does this bug cause significantly incorrect results in the model's science? Yes

Configurations affected: With use_lai_streams=T

Details of bug

The file with the problem is: $DIN_LOC_ROOT/lnd/clm2/lai_streams/MODISPFTLAI_0.5x0.5_c140711.nc
This is from the email I sent Simone Tilmes...

We noticed that this file...

ls -l /glade//p/cesmdata/inputdata/lnd/clm2/lai_streams/MODISPFTLAI_0.5x0.5_c140711.nc

-rw-rw-r-- 1 tilmes cseg 1820327936 May 27 20:07 /glade//p/cesmdata/inputdata/lnd/clm2/lai_streams/MODISPFTLAI_0.5x0.5_c140711.nc

has your name on it as being overwritten on my birthday this year. My guess is that this was an accidental birthday gift ;-) -- as the file is different than the version that's in svn or on CGD.

Do you remember working with this file, or running any simulations with it? And have any idea of what happened for it to be overwritten? The metadata on the file is unchanged, and only 5 of the 28 fields have differences.

Note, that Simone doesn't know how this happened. Also we can only see the change, modify and creation dates and they might not correspond to the author of the file. The author should have been Peter Lawrence or Stephen Muszala, as it was added by them in something like clm4_5_71. Stephen's author stamp is on the file in inputdata.

[ekluzek]

I checked other files with Simone's author stamp and most made sense that they had her name on it. I checked one SSP file to make sure it was identical to what's in inputdata and it was. So I'm reasonably satisfied that this problem isn't more pervasive. We still don't understand quite why it happened, so we can't prevent it.

But, @billsacks suggestion is that we periodically run the following aliases in inputdata to make sure files are read-only...

alias inputdata_perms_dir='find . -user sacks -type d -exec chmod a+rx,ug+w,g+s {} ;'
alias inputdata_perms_fil='find . -user sacks -type f -exec chmod a+r,a-w {} ;'

[ekluzek]

As I thought pointing to the correct file does give an apparent change in answers for this test. Any simulations done with LAI streams on would be suspect at this point. When I looked at the differences in the files, they are large for a few fields and changes are all over the globe. There was one field that must have been different at only a few points, but the other fields were different all over.

This also means when we correct the file, any tests using previous baselines (since May/27th) which means ctsm1.0.dev098 to ctsm1.0.dev113 will show answer differences for this specific test.

[ekluzek]

OK, I copied the file over. So this will appear to change answers for the next CTSM tag. To prevent that I'll rerun this test for both ctsm1.0.dev113 and ctsm_release-cesm2.2.01.

[ekluzek]

OK, I reran those two baselines. So any comparisons to dev113 or cesm2.2.01 won't show an answer change. As such I can close this issue.