update TEMPEST converter for uplifted data #1610

BenjaminRuston · 2025-01-30T00:25:43Z

Description

The TEMPEST data has been uplifted and is available via NASA Earthdata

Issue(s) addressed

Resolves #1609

Impact

Expected impact on downstream repositories:
enables
https://github.com/JCSDA-internal/ewok/pull/1097
https://github.com/JCSDA-internal/skylab/pull/664

Checklist

I have performed a self-review of my own code
I have run the unit tests before creating the PR

… to update TEMPEST as well

BenjaminRuston · 2025-01-30T00:27:55Z

@ashley314 think this should work in general...

I disabled the QC (not ideal) as I think my sample file was from a period of blackout. the international space station (ISS) has a lot of activities and various obstructions and periods of outage unique from free flying spacecraft.

So will try this for a few more dates and try to see if indeed the sample file is from an inconvenient period

ashley314 · 2025-01-30T15:05:45Z

Thank you! I will try it out with TEMPEST and COWVR today

ashley314 · 2025-01-30T20:17:22Z

I am still getting the following error using data from 2025-01-24.

++ python3 /Users/agriffin/JEDI_developing/jedi-bundle/iodaconv/src/hdf5/cowvr_hdf5_2ioda.py -d 2025012400 -i obs.tempest_iss.20250124T000000Z.0.h5 -o /Users/agriffin/JEDI_developing/workdir/4dcbab/20250124T000000Z/obs_tempest_iss/obs.tempest_iss.20250124T000000Z.0.nc4
 ... starting timer: 1881379.691
Traceback (most recent call last):
 File "/Users/agriffin/JEDI_developing/jedi-bundle/iodaconv/src/hdf5/cowvr_hdf5_2ioda.py", line 443, in <module>
   main(args)
 File "/Users/agriffin/JEDI_developing/jedi-bundle/iodaconv/src/hdf5/cowvr_hdf5_2ioda.py", line 78, in main
   file_obs_data = get_data_from_files(afile)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/agriffin/JEDI_developing/jedi-bundle/iodaconv/src/hdf5/cowvr_hdf5_2ioda.py", line 156, in get_data_from_files
   obs_data = get_tempest_data(f, obs_data)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/agriffin/JEDI_developing/jedi-bundle/iodaconv/src/hdf5/cowvr_hdf5_2ioda.py", line 182, in get_tempest_data
   np.array(f['GeolocationAndFlags']['instr_scan_ang'], dtype='float32'),
            ~^^^^^^^^^^^^^^^^^^^^^^^
 File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
 File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
 File "/Users/agriffin/JCSDA/spack-stack-1.8.0-sonoma/spack-stack/envs/ue.agriffin/install/apple-clang/14.0.3/py-h5py-3.11.0-onv25l2/lib/python3.11/site-packages/h5py/_hl/group.py", line 357, in __getitem__
   oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
 File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
 File "h5py/h5o.pyx", line 241, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'GeolocationAndFlags' doesn't exist)"

ashley314 · 2025-01-30T20:22:00Z

src/hdf5/cowvr_hdf5_2ioda.py

+    obs_data[('solarZenithAngle', metaDataName)] = np.array(f['Geolocation']['sat_solar_zen'], dtype='float32')
+    obs_data[('solarAzimuthAngle', metaDataName)] = np.array(f['Geolocation']['sat_solar_az'], dtype='float32')
+    obs_data[('sensorZenithAngle', metaDataName)] = np.array(f['Geolocation']['earth_inc_ang'], dtype='float32')
+    obs_data[('sensorAzimuthAngle', metaDataName)] = np.array(f['Geolocation']['earth_az_ang'], dtype='float32')
    obs_data[('sensorViewAngle', metaDataName)] = compute_scan_angle(
        np.array(f['GeolocationAndFlags']['instr_scan_ang'], dtype='float32'),


Does this need to be updated to np.array(f['Geolocation']['instr_scan_ang'], dtype='float32'),?

I updated this line and got the converter to successfully run for TEMPEST and COWVR. Thanks!

ashley314 · 2025-01-30T20:29:49Z

The converter is working but the plot for TEMPEST looks like this...

BenjaminRuston · 2025-01-31T00:26:24Z

@ashley314 did you do Jan2025 (or 2024) both of those dates believe the instruments was essentially off

I'm running May2024 and timed out on the first conversion (converted 6 of 7)... notice the files are not much smaller than the original:

(venv_orion) orion-login-3: /hdf5 >ll /work2/noaa/jcsda/bruston/work/JCSDA/JEDI/tmp/ewok/40c309/20240501T000000Z/obs_tempest_iss
total 237704
drwxr-s--- 2 bruston jcsda     4096 Jan 30 17:46 .
drwxr-s--- 3 bruston jcsda     4096 Jan 30 17:30 ..
-rw-r----- 1 bruston jcsda 25635757 Jan 30 17:34 obs.tempest_iss.20240501T000000Z.0.nc4
-rw-r----- 1 bruston jcsda 25583898 Jan 30 17:36 obs.tempest_iss.20240501T000000Z.1.nc4
-rw-r----- 1 bruston jcsda 25469919 Jan 30 17:37 obs.tempest_iss.20240501T000000Z.2.nc4
-rw-r----- 1 bruston jcsda 25507665 Jan 30 17:40 obs.tempest_iss.20240501T000000Z.3.nc4
-rw-r----- 1 bruston jcsda 25916486 Jan 30 17:42 obs.tempest_iss.20240501T000000Z.4.nc4
-rw-r----- 1 bruston jcsda 24950166 Jan 30 17:44 obs.tempest_iss.20240501T000000Z.5.nc4
-rw-r----- 1 bruston jcsda 25374707 Jan 30 17:46 obs.tempest_iss.20240501T000000Z.6.nc4
-rw-r----- 1 bruston jcsda 64915624 Oct 31 00:01 obs.tempest_iss.20240501T000000Z.7.h5

now running with the apply_qc and it's running faster and the files are much smaller:

-rw-r----- 1 bruston jcsda 64915624 Oct 31 00:01 obs.tempest_iss.20240501T000000Z.7.h5
-rw-r----- 1 bruston jcsda 65862789 Oct 31 00:01 obs.tempest_iss.20240501T000000Z.6.h5
drwxr-s--- 3 bruston jcsda     4096 Jan 30 18:10 ..
-rw-r----- 1 bruston jcsda    20858 Jan 30 18:10 response.json
-rw-r----- 1 bruston jcsda  9671505 Jan 30 18:13 obs.tempest_iss.20240501T000000Z.0.nc4
-rw-r----- 1 bruston jcsda  9635534 Jan 30 18:15 obs.tempest_iss.20240501T000000Z.1.nc4

BenjaminRuston · 2025-01-31T01:47:16Z

yay! this is working for a day with valid data:

now will try for a date with potentially all flagged data

ashley314 · 2025-01-31T14:45:06Z

Awesome! Thank you! I will give it a shot as well.

ashley314 · 2025-01-31T18:49:05Z

With these updates it fixes the plot for the same date I ran as before:

ashley314

This is working for COWVR and TEMPEST using https://github.com/JCSDA-internal/skylab/pull/664 and https://github.com/JCSDA-internal/ewok/pull/1097/. Thanks!

haydenlj

Thanks @BenjaminRuston this worked for me

BenjaminRuston added 6 commits January 24, 2025 17:36

update converter for uplifted public released data from COWVR attempt…

5e25444

… to update TEMPEST as well

add a sample of COWVR data

806fc4c

add some new files to CMakeLists

20f95b2

add the ctest into the CMakeLists

8533c87

add the file to CMakeLists in src

1cb9a39

update TEMPEST converter for uplifted data

986035a

BenjaminRuston requested a review from ashley314 January 30, 2025 00:25

Merge branch 'develop' into feature/tempest4ingestsuite

895fa28

BenjaminRuston requested a review from fcvdb January 30, 2025 17:29

ashley314 reviewed Jan 30, 2025

View reviewed changes

fix another GeolocationAndFlags

17fce4c

enable the quality control based on internal flags to reduce data

29a8df5

BenjaminRuston marked this pull request as ready for review January 31, 2025 18:55

BenjaminRuston requested review from ashley314, smaticka and haydenlj January 31, 2025 18:55

ashley314 approved these changes Jan 31, 2025

View reviewed changes

haydenlj approved these changes Feb 5, 2025

View reviewed changes

BenjaminRuston merged commit e984f85 into develop Feb 6, 2025

BenjaminRuston deleted the feature/tempest4ingestsuite branch February 6, 2025 17:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update TEMPEST converter for uplifted data #1610

update TEMPEST converter for uplifted data #1610

BenjaminRuston commented Jan 30, 2025

BenjaminRuston commented Jan 30, 2025

ashley314 commented Jan 30, 2025

ashley314 commented Jan 30, 2025

ashley314 Jan 30, 2025

ashley314 Jan 30, 2025

ashley314 commented Jan 30, 2025

BenjaminRuston commented Jan 31, 2025

BenjaminRuston commented Jan 31, 2025

ashley314 commented Jan 31, 2025

ashley314 commented Jan 31, 2025

ashley314 left a comment

haydenlj left a comment

update TEMPEST converter for uplifted data #1610

update TEMPEST converter for uplifted data #1610

Conversation

BenjaminRuston commented Jan 30, 2025

Description

Issue(s) addressed

Impact

Checklist

BenjaminRuston commented Jan 30, 2025

ashley314 commented Jan 30, 2025

ashley314 commented Jan 30, 2025

ashley314 Jan 30, 2025

Choose a reason for hiding this comment

ashley314 Jan 30, 2025

Choose a reason for hiding this comment

ashley314 commented Jan 30, 2025

BenjaminRuston commented Jan 31, 2025

BenjaminRuston commented Jan 31, 2025

ashley314 commented Jan 31, 2025

ashley314 commented Jan 31, 2025

ashley314 left a comment

Choose a reason for hiding this comment

haydenlj left a comment

Choose a reason for hiding this comment