add ctest for ground-based gnss ipw product #1539

BenjaminRuston · 2024-08-06T23:13:58Z

add a ctest for the ground-based gnss integrated precipitable water (IPW) product

Description

add a ctest for an NCEP BUFR file for IPW from ground-based GNSS

Issue(s) addressed

Resolves #<issue_number>

Impact

Potential to add IPW product to assimilation

Checklist

I have performed a self-review of my own code
I have run the unit tests before creating the PR

…roduct

BenjaminRuston · 2024-08-06T23:17:32Z

@ibanos90 added this to see if it properly passes regression testing

the input file, and resulting output file are too large.. @gthompsnJCSDA may be able to help edit the input BUFR to trim it down to well under a MegaByte

for this file at current resolution got a IODArange.py output of:

file: 20240806T00Z_ipw_gnssgb_ncep.nc
range(/MetaData/dateTime): [2024-08-05 23:59:00, 2024-08-06 00:02:00]
range(/MetaData/heightOfStation): [90.0, 90.0]
range(/MetaData/latitude): [-77.83836364746094, 79.98895263671875]
range(/MetaData/longitude): [-170.72244262695312, 179.19654846191406]
range(/MetaData/stationName): [0001-JMA, ZZON-ROBH]
range(/ObsValue/precipitableWater): [2, 72]

BenjaminRuston · 2024-08-06T23:21:33Z

test/testinput/bufr_gpsipw.yaml

@@ -0,0 +1,82 @@
+# (C) Copyright 2022 NOAA/NWS/NCEP/EMC


Suggested change

# (C) Copyright 2022 NOAA/NWS/NCEP/EMC

# (C) Copyright 2024 NOAA/NWS/NCEP/EMC

emilyhcliu · 2024-08-07T02:13:38Z

test/testinput/bufr_gpsipw.yaml

+
+        - name: "MetaData/stationName"
+          source: variables/stationName
+          longName: "Station Name"


I do not see "stationName" in the IODA data conventions. How about changing "stationName" to "stationIdentification"?

thanks @emilyhcliu appreciate the extra set of eyes.
and as Emily notes there the online document and this too:
https://github.com/JCSDA-internal/ioda/blob/develop/share/ioda/yaml/validation/ObsSpace.yaml

Thanks @emilyhcliu and @BenjaminRuston for letting me help with this. Please proceed with changing"stationName" to "stationIdentification".

One additional change I think didn't get included -

for "heightOfStation", the query value should be "/SELV" instead of "/GNSSRPSQ/ELEV".

BenjaminRuston · 2024-09-04T21:11:00Z

@ibanos90 is the variable that should be a float but is reported as an integer the Observation value itself:
range(/ObsValue/precipitableWater): [2, 72]

will pick this up and try to drag across the finish line

ibanos90 · 2024-09-04T21:58:02Z

@ibanos90 is the variable that should be a float but is reported as an integer the Observation value itself: range(/ObsValue/precipitableWater): [2, 72]

will pick this up and try to drag across the finish line

Yes, that's correct, but @mrixlam can provide more info

mrixlam · 2024-09-04T22:11:47Z

@ibanos90 is the variable that should be a float but is reported as an integer the Observation value itself: range(/ObsValue/precipitableWater): [2, 72]
will pick this up and try to drag across the finish line

Yes, that's correct, but @mrixlam can provide more info

The variable we discussed about is /ObsValue/precipitableWater.

I have some decoded bufr messages here: /glade/derecho/scratch/mrislam/work/singv_ng_mpas/data/gpsipw in case you have time to take a look at them. Decoded bufr messages have floating point values for precipitableWater but in the converted NetCDF file, it becomes integer and we have no idea why.

BenjaminRuston · 2024-09-05T14:59:56Z

test/testinput/bufr_gpsipw.yaml

+
+      globals:
+        - name: "platformCommonName"
+          type: string


@mrixlam ,,, hearing from @gthompsnJCSDA that adding a line like the but type: float to the precipitableWater should force float

Hi @BenjaminRuston we previously tried "type: float" for the variables section but it didnt work. But you are right. When I removed "type: float" from the variables section to the exports section, it works. The converted file now have float values for precipitableWater.

I'm confused @mrixlam I just tried this with and w/o the type: float

when I added it, I got floats it worked correctly:
range(/ObsValue/precipitableWater): [2.299999952316284, 72.0999984741211]

when removed it output the integer values:
range(/ObsValue/precipitableWater): [2, 72]

so going to add it in... please confirm

test/testinput/bufr_gpsipw.yaml

mrixlam · 2024-09-06T19:40:30Z

test/testinput/bufr_gpsipw.yaml

+            query: "*/STSN"
+
+          heightOfStation:
+            query: "*/GNSSRPSQ/ELEV"


query for heightOfStation should be changed to "*/SELV"

BenjaminRuston · 2024-09-06T20:25:36Z

test/testinput/bufr_gpsipw.yaml

+            query: "*/STSN"
+
+          heightOfStation:
+            query: "*/GNSSRPSQ/ELEV"


Suggested change

query: "*/GNSSRPSQ/ELEV"

query: "*/GNSSRPSQ/SELV"

or this?

Suggested change

query: "*/GNSSRPSQ/ELEV"

query: "*/SELV"

BenjaminRuston · 2024-09-06T20:30:32Z

test/testinput/bufr_gpsipw.yaml

+        - name: "ObsValue/precipitableWater"
+          source: variables/precipitableWater
+          longName: "GPS IPW"
+          units: "KG/(METER**2)"


@mrixlam this is consistent with the way units are presented in our verification yaml
https://github.com/JCSDA-internal/ioda/blob/ad84060522f6fa7ab9c75d227f1d9e72c8bf7a28/share/ioda/yaml/validation/ObsSpace.yaml#L570

Suggested change

units: "KG/(METER**2)"

units: "kg m-2"

…t its output to float

i've updated to use stationIdentification thanks Emily

…da-internal/ioda-converters into feature/ipw_gb_gnss_addition

BenjaminRuston · 2024-09-07T01:58:01Z

@mrixlam and @ibanos90 think I've wrapped this one up,, I'm getting floats for the output and I've reduced the sample file size down to just a few observations to make the ctest dataset much smaller

please let me know if this works for you.. thx

mrixlam

Current version looks good to me.

ibanos90

Looks good! Thanks so much @BenjaminRuston!

add a ctest for the ground-based gnss integrated precipitable water p…

4b92879

…roduct

BenjaminRuston added the OBS OBS processing, UFO label Aug 6, 2024

BenjaminRuston requested review from ibanos90, emilyhcliu, huishao-r and fabiolrdiniz August 6, 2024 23:14

BenjaminRuston added the do not merge Something is wrong, do not merge label Aug 6, 2024

BenjaminRuston requested a review from gthompsnJCSDA August 6, 2024 23:15

BenjaminRuston commented Aug 6, 2024

View reviewed changes

emilyhcliu previously requested changes Aug 7, 2024

View reviewed changes

BenjaminRuston marked this pull request as draft August 7, 2024 21:00

BenjaminRuston added the draft label Aug 7, 2024

BenjaminRuston commented Sep 5, 2024

View reviewed changes

test/testinput/bufr_gpsipw.yaml Show resolved Hide resolved

mrixlam reviewed Sep 6, 2024

View reviewed changes

BenjaminRuston commented Sep 6, 2024

View reviewed changes

BenjaminRuston added 2 commits September 6, 2024 13:44

update station height and station identification units for ipw and se…

9fd11d5

…t its output to float

update the testoutput this and input still need to be reduced

d8c2328

BenjaminRuston requested review from emilyhcliu and mrixlam September 6, 2024 20:46

BenjaminRuston added 2 commits September 6, 2024 18:49

add new testoutput for a small sample of data

a4e9979

create a small sample file for ctest

0a7af81

BenjaminRuston removed do not merge Something is wrong, do not merge draft labels Sep 7, 2024

BenjaminRuston marked this pull request as ready for review September 7, 2024 01:52

BenjaminRuston and others added 3 commits September 6, 2024 18:52

Merge branch 'develop' into feature/ipw_gb_gnss_addition

9eb93b7

update the CMakeLists for the ctest using small sample file

5255f24

Merge branch 'feature/ipw_gb_gnss_addition' of https://github.com/jcs…

03d305d

…da-internal/ioda-converters into feature/ipw_gb_gnss_addition

mrixlam approved these changes Sep 7, 2024

View reviewed changes

ibanos90 approved these changes Sep 7, 2024

View reviewed changes

BenjaminRuston added 6 commits September 9, 2024 12:00

remove large IPW output file

dccab35

remove large input IPW BUFR file

8b2c2b0

update the names of files for output of test

8adce11

update the names of obsdataout for ctest

08f5cad

update name of testouput file

0604f90

add ground based IPW input yaml for decoding

a43c374

BenjaminRuston merged commit e0ce37b into develop Sep 9, 2024
2 checks passed

BenjaminRuston deleted the feature/ipw_gb_gnss_addition branch September 9, 2024 23:07

BenjaminRuston mentioned this pull request Sep 19, 2024

Feature/viirs aod smoke dust #1550

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ctest for ground-based gnss ipw product #1539

add ctest for ground-based gnss ipw product #1539

BenjaminRuston commented Aug 6, 2024 •

edited

Loading

BenjaminRuston commented Aug 6, 2024

BenjaminRuston Aug 6, 2024

emilyhcliu Aug 7, 2024 •

edited

Loading

BenjaminRuston Aug 7, 2024

mrixlam Sep 4, 2024

BenjaminRuston commented Sep 4, 2024

ibanos90 commented Sep 4, 2024

mrixlam commented Sep 4, 2024

BenjaminRuston Sep 5, 2024

mrixlam Sep 6, 2024

BenjaminRuston Sep 6, 2024

mrixlam Sep 6, 2024

BenjaminRuston Sep 6, 2024

BenjaminRuston Sep 6, 2024

BenjaminRuston Sep 6, 2024

BenjaminRuston commented Sep 7, 2024

mrixlam left a comment

ibanos90 left a comment

	# (C) Copyright 2022 NOAA/NWS/NCEP/EMC
	# (C) Copyright 2024 NOAA/NWS/NCEP/EMC

add ctest for ground-based gnss ipw product #1539

add ctest for ground-based gnss ipw product #1539

Conversation

BenjaminRuston commented Aug 6, 2024 • edited Loading

Description

Issue(s) addressed

Impact

Checklist

BenjaminRuston commented Aug 6, 2024

Choose a reason for hiding this comment

emilyhcliu Aug 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminRuston commented Sep 4, 2024

ibanos90 commented Sep 4, 2024

mrixlam commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminRuston commented Sep 7, 2024

mrixlam left a comment

Choose a reason for hiding this comment

ibanos90 left a comment

Choose a reason for hiding this comment

BenjaminRuston commented Aug 6, 2024 •

edited

Loading

emilyhcliu Aug 7, 2024 •

edited

Loading