Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed-up the obs processing #886

Closed
guillaumevernieres opened this issue Jan 29, 2024 · 1 comment
Closed

Speed-up the obs processing #886

guillaumevernieres opened this issue Jan 29, 2024 · 1 comment
Assignees
@AndrewEichmann-NOAA
Labels
soca

Comments

@guillaumevernieres
Copy link
Contributor

With the addition of the geo-stationary sst's, it take ~12mn to process the satellite obs for 1 cycle. This needs to be parallelized.
Executables of interest:

GDASApp/scripts/exglobal_prep_ocean_obs.py

Line 85 in 0ceff32

subprocess.run([OCNOBS2IODAEXEC, iodaYamlFilename], check=True)

@AndrewEichmann-NOAA AndrewEichmann-NOAA mentioned this issue Jan 30, 2024
Merged
guillaumevernieres pushed a commit that referenced this issue Jan 31, 2024
@AndrewEichmann-NOAA
Runs netcdf ioda converters in parallel (#888)
6023fd2 
Sets up the ocean obs prep task to run the NetCDF to IODA converter
executable in parallel by obs space, node permitting. Also tries to
detect and notify of failures.

Tested with ctests on Hera, including pointing `DMPDIR` at
`/scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/`. Runtime was reduced
only from 16 minutes to 15 minutes, but that is probably because only
rads and metop b GHRSST are processed, with the latter being the
bottleneck.

Partially addresses #886
@guillaumevernieres
Copy link
Contributor Author

I'm getting timing around 230 sec @AndrewEichmann-NOAA , that's with 2 AVHRR and 2 VIIRS but no geostationary and a stride of 15 for binning. Were you able to run a test on your side?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndrewEichmann-NOAA AndrewEichmann-NOAA

Labels
soca
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@guillaumevernieres @AndrewEichmann-NOAA @apchoiCMD