Published datasets

[aidanheerdegen]

The data published in cj50 needs to be added to the master database.

@AndyHoggANU did you say this was ready to go?

[aekiss]

I think we agreed I should delete ty_trans_int_z but otherwise I think it's probably good to go.
If cj50 is part of the automatic update I guess it will be fixed when those files are deleted, so indexing could start now?

[AndyHoggANU]

Agreed - let's do it.

[aidanheerdegen]

So all these model directories under /g/data/cj50/access-om2/?

├── v20171212
│   ├── access-om2
│   │   ├── 1deg_jra55v13_ryf0304_RCP45
│   │   ├── 1deg_jra55v13_ryf0304_spinup
│   │   ├── 1deg_jra55v13_ryf8485_RCP45
│   │   ├── 1deg_jra55v13_ryf8485_spinup
│   │   ├── 1deg_jra55v13_ryf9091_RCP45
│   │   └── 1deg_jra55v13_ryf9091_spinup
│   ├── access-om2-025
│   │   ├── 025deg_jra55v13_ryf8485_RCP45
│   │   └── 025deg_jra55v13_ryf8485_spinup
│   ├── license.txt
│   └── readme.txt
├── v20191202
│   ├── access-om2
│   │   └── jra55v13_iaf
│   ├── access-om2-01
│   │   └── jra55v13_iaf
│   ├── access-om2-025
│   │   └── jra55v13_iaf
│   ├── license.txt
│   └── readme.txt
└── v20200608
    ├── access-om2-01
    │   ├── jra55v13_ryf8485
    │   ├── jra55v13_ryf8485_freshRCP45
    │   └── jra55v13_ryf8485_freshRCP85
    ├── license.txt
    └── readme.txt

There are no metadata.yaml files in any of those directories. I would suggest we not index them until those are present.

There are 201 unreadable directories in /g/data/cj50/access-om2/v20171212/access-om2/1deg_jra55v13_ryf8485_spinup and this temporary file is unreadable and should be removed in any case /g/data/cj50/access-om2/v20191202/access-om2-01/jra55v13_iaf/ocean/u/.u_access-om2-01_200607_200612.nc.rt8HEW.

[AndyHoggANU]

Yes - we will only include data under raw-output into the database for now.

[aidanheerdegen]

Didn't respond to this, apologies.

I'll add that directory to the index.

[AndyHoggANU]

We haven't yet included the v20191202 and v20200608 directories into the COSIMA database. @aekiss - any chance you can make the metadata.yaml files for these experiments so that they can be included?

[aekiss]

I've had a bash at the IAF ones - do these look ok? (@aidanheerdegen are one and access-om2-1 standard keywords?)
/g/data/cj50/access-om2/v20191202/access-om2*/jra55v13_iaf/metadata.yaml

I've also put a draft RYF one here
/g/data/cj50/access-om2/v20200608/access-om2-01/jra55v13_ryf8485/metadata.yaml
@AndyHoggANU can you check this and use it to create the metadata for the perturbed runs? (@aidanheerdegen is there a standard keyword for perturbations?)

[aidanheerdegen]

Standard is up to you guys, what is deemed useful. The utility of tenth/quarter would be is there were some non-ACCESS-OM2 runs that didn't also use access-om2-025. If there aren't any, or noone uses it, then probably less useful. one is a bit problematic, as it doesn't have the instant recognition that quarter or tenth do.

There is no keyword for perturbations. Make one up and use it.

[AndyHoggANU]

Done my bits -- @aidanheerdegen if you get time to add these to the database that would be great.

[aidanheerdegen]

Sorry this dropped off my radar.

Does this render this moot?

#2

So we should remove them from the database and replace with these?

[aidanheerdegen]

Cleaning up issues. This seems to have been fixed with 2089092

Indexes the following datasets:

$ find /g/data/cj50/access-om2/cf-compliant/ -mindepth 3 -maxdepth 3 -type d
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf9091_spinup
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf0304_spinup
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf8485_RCP45
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf9091_RCP45
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf0304_RCP45
/g/data/cj50/access-om2/cf-compliant/access-om2/v20171212/1deg_jra55v13_ryf8485_spinup
/g/data/cj50/access-om2/cf-compliant/access-om2/v20191202/jra55v13_iaf
/g/data/cj50/access-om2/cf-compliant/access-om2-01/v20200608/jra55v13_ryf8485_freshRCP45
/g/data/cj50/access-om2/cf-compliant/access-om2-01/v20200608/jra55v13_ryf8485_freshRCP85
/g/data/cj50/access-om2/cf-compliant/access-om2-01/v20200608/jra55v13_ryf8485
/g/data/cj50/access-om2/cf-compliant/access-om2-01/v20191202/jra55v13_iaf
/g/data/cj50/access-om2/cf-compliant/access-om2-025/v20191202/jra55v13_iaf
/g/data/cj50/access-om2/cf-compliant/access-om2-025/v20171212/025deg_jra55v13_ryf8485_spinup
/g/data/cj50/access-om2/cf-compliant/access-om2-025/v20171212/025deg_jra55v13_ryf8485_RCP45

$ find /g/data/cj50/access-om2/raw-output/ -mindepth 2 -maxdepth 2 -type d
/g/data/cj50/access-om2/raw-output/access-om2-025/025deg_jra55v13_SAMx_0203_from_ryf9091_yr160_gmredi6
/g/data/cj50/access-om2/raw-output/access-om2-025/025deg_jra55v13_SAMx_9192_from_ryf9091_yr160_gmredi6
/g/data/cj50/access-om2/raw-output/access-om2-025/025deg_jra55v13_SAMx_1011_from_ryf9091_yr160_gmredi6
/g/data/cj50/access-om2/raw-output/access-om2-025/025deg_jra55v13_ryf9091_gmredi6
/g/data/cj50/access-om2/raw-output/access-om2-025/025deg_jra55v13_SAMx_9899_from_ryf9091_yr160_gmredi6
/g/data/cj50/access-om2/raw-output/access-om2/1deg_jra55_SAMextr_ryf9192_from9091_kds50
/g/data/cj50/access-om2/raw-output/access-om2/1deg_jra55_ryf9091_kds50_bkd
/g/data/cj50/access-om2/raw-output/access-om2/1deg_jra55_SAMextr_ryf9899_from9091_kds50
/g/data/cj50/access-om2/raw-output/access-om2/1deg_jra55_SAMextr_ryf1011_from9091_kds50
/g/data/cj50/access-om2/raw-output/access-om2/1deg_jra55_SAMextr_ryf0203_from9091_kds50
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55_SAMextr_1011_from9091
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55_SAMextr_0203_from9091
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle2
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55_SAMextr_9899_from9091
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle4
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55_SAMextr_9192_from9091