sync_data.sh does not concatenate and copy daily ice output

[hakaseh]

When I do 'qsub sync_data.sh', it successfully copies the output/restart over, except that 'ice/OUTPUT' in the destination is empty.

In the output file (sync.o19960106), there is this message that might prevent from using cdo commands:

"Fatal error: modules not available."

Any idea what is wrong?

My control directory is: /scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro

And the destination is: /g/data/ik11/outputs/access-om2/1deg_jra55_iaf_iamip2_pro

[aekiss]

I'll take a look - can you enable group read on /scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/sync* please?

[hakaseh]

Thanks @aekiss.
I just changed the permission.

[aekiss]

The problem is that the daily ice concatenation assumes that there are monthly output files, which is not the case for this run, so nothing gets concatenated. rsync is set up to not sync unconcatenated ice dailies, so nothing gets copied. I'll try to think of a better way to do this.

The messages about modules are unrelated and to do with creating the run summary.

[aekiss]

I've made a (hopefully) improved version of the concatenation part.

Copy /home/156/aek156/concat-dailies.sh to your control directory and try running it.

This should concatenate the daily files into archive/output*/ice/OUTPUT/*-daily.nc
and rename the individual daily files to archive/output*/ice/OUTPUT/*-DELETE

Let me know how it goes. It might take a while to concatenate everything. If all goes well you should be able to re-run the sync script and have it copy the files you want.

[hakaseh]

/home/156/aek156/concat-dailies.sh is working well, thanks!

it would be great if this could be implemented into sync_data.sh as well, although saving only daily might be rare.

[aekiss]

Yep, that was the plan.

Looks like one month didn't work:
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output098/ice/OUTPUT/*.????-??-??.nc
if the concatenation job is no longer running you should delete
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output098/ice/OUTPUT/iceh.2056-02-IN-PROGRESS
and run concat-dailies.sh again.

[aekiss]

It's also odd that run_summary.py couldn't find f90nml. What do you get if you type this?

module use /g/data/hh5/public/modules
module load conda/analysis3
conda list | grep f90nml

[hakaseh]

It's also odd that run_summary.py couldn't find f90nml. What do you get if you type this?

module use /g/data/hh5/public/modules
module load conda/analysis3
conda list | grep f90nml

$ conda list | grep f90nml
f90nml 1.2 pyh9f0ad1d_0 conda-forge

[hakaseh]

Yep, that was the plan.

Looks like one month didn't work:
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output098/ice/OUTPUT/*.????-??-??.nc
if the concatenation job is no longer running you should delete
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output098/ice/OUTPUT/iceh.2056-02-IN-PROGRESS
and run concat-dailies.sh again.

Thanks for catching this. For my knowledge, could you tell me how you found out that this particular output did not work??

[aekiss]

I just did
ls /scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output098/ice/OUTPUT/*.????-??-??.nc
to find unconcatenated daily files.

[aekiss]

when you're satisfied the concatenated files are ok you can delete the unconcatenated ones
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output*/ice/OUTPUT/*-DELETE
and this tmp file
/scratch/v45/hh0162/access-om2-iamip2/control/1deg_jra55_iaf_iamip2_pro/archive/output*/ice/OUTPUT/*.tmp