Runs netcdf ioda converters in parallel

[AndrewEichmann-NOAA]

Sets up the ocean obs prep task to run the NetCDF to IODA converter executable in parallel by obs space, node permitting. Also tries to detect and notify of failures.

Tested with ctests on Hera, including pointing DMPDIR at /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/. Runtime was reduced only from 16 minutes to 15 minutes, but that is probably because only rads and metop b GHRSST are processed, with the latter being the bottleneck.

Partially addresses #886

[guillaumevernieres]

that doesn't make sense @AndrewEichmann-NOAA . How many sst retrievals did you test with? It should be as fast as the slowest sst retrieval.

[AndrewEichmann-NOAA]

that doesn't make sense @AndrewEichmann-NOAA . How many sst retrievals did you test with? It should be as fast as the slowest sst retrieval.

@guillaumevernieres That's 187 input files, 2810778 obs after the conversion - I have it go 4 cycles back. Too far? :-)

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Tue Jan 30 14:49:43 CST 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jan 30 15:36:09 CST 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Tue Jan 30 16:03:08 CST 2024
Tests: 100% tests passed, 0 tests failed out of 52

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Tue Jan 30 20:52:28 UTC 2024 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jan 30 21:46:03 UTC 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Tue Jan 30 22:04:47 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 52

[AndrewEichmann-NOAA]

@guillaumevernieres
Tested with the following, using current settings (mostly) in prepobs_config:

(gdasapp) -bash-4.2$ ls gw/testrun/testjjobs/ROTDIRS/gdas_test/gdas.20210701/12/obs/
adt_rads_all2ioda.yaml                     gdas.t12z.sst_avhrr_mb_l3u.2021070112.nc4   sst_avhrr_ma_l3u2ioda.yaml
gdas.t12z.adt_rads_all.2021070112.nc4      gdas.t12z.sst_viirs_n20_l3u.2021070112.nc4  sst_avhrr_mb_l3u2ioda.yaml
gdas.t12z.icec_amsr2_north.2021070112.nc4  gdas.t12z.sst_viirs_npp_l3u.2021070112.nc4  sst_viirs_n20_l3u2ioda.yaml
gdas.t12z.icec_amsr2_south.2021070112.nc4  icec_amsr2_north2ioda.yaml                  sst_viirs_npp_l3u2ioda.yaml
gdas.t12z.sst_avhrr_ma_l3u.2021070112.nc4  icec_amsr2_south2ioda.yaml
(gdasapp) -bash-4.2$

run time 3:41

[guillaumevernieres]

Nice!