Fix for IGRA dewpoint calculation

[swnesbitt]

There was an order of operations issue where the data being read dewpoint_depression was being subtracted from temperature before the field was flagged for missing values. This pull request simply changes the field being flagged and does the flagging before dewpoint is calculated, resulting in properly flagged dewpoints.

[CLAassistant]

All committers have signed the CLA.

[danielmwatkins]

Thanks for catching that!

[dopplershift]

Hmm...test failure due to lack of dewpoint_depression field. Is that only available sometimes?

[swnesbitt]

All the humidity data is reported in dewpoint depression. This error only arises if there is a valid temperature and invalid dewpoint_depression observation at a level. I'm working with data from South America and this happens sometimes.

[dopplershift]

Hmmm...is there any way to keep it from erring out in that case?

[swnesbitt]

The fix I proposed in the PR shouldn't cause any errors - before it wasn't crashing either, but it was giving ~1000 degree C dewpoints which was somewhat problematic.

-Steve

On Fri, May 4, 2018 at 1:11 PM Ryan May ***@***.***> wrote: Hmmm...is there any way to keep it from erring out in that case? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADIuEC29SqQ_NCoaCiErDZgH-d8Vg0Ogks5tvJm7gaJpZM4Txb_T> .

-- -- Steve Nesbitt, Associate Professor Department of Atmospheric Sciences, University of Illinois at Urbana-Champa ign Natural History Building 3060, 1301 W. Green St, Urbana, IL 61801 USA voice : +1.217.244.3740 fax : +1.217.244.1752 internet : [email protected] ; https://ww w.atmos.illinois.edu/people/snesbitt ; http://publish.illinois.edu/snesbitt Radar Meteorology, A First Course, our 488 page full-color textbook including homework assignments and solutions for instructors, is now available! For information about ordering a hardcopy or e-book, visit: https://bit.ly/ 2qeYNbk

[dopplershift]

Let me clarify. With your changes, we are currently seeing an error in the tests:

            axis = self._get_axis_number(axis)
            agg_axis = 1 - axis
    
            agg_obj = self
            if subset is not None:
                ax = self._get_axis(agg_axis)
                indices = ax.get_indexer_for(subset)
                check = indices == -1
                if check.any():
>                   raise KeyError(list(np.compress(check, subset)))
E                   KeyError: ['dewpoint_depression']
../../../virtualenv/python3.6.3/lib/python3.6/site-packages/pandas/core/frame.py:3489: KeyError

So while maybe in theory your change shouldn't cause any errors, in practice it's causing an error for:

df, header = IGRAUpperAir.request_data(datetime(2014, 9, 10, 0),                           
                                            'USM00070026', derived=True)

[danielmwatkins]

The problem here is that this fix is only relevant to the derived=False case. When derived=True, the retrieved file will never have a dewpoint depression column. Instead it will have relative humidity. This is why the code as is has reported_relative_humidity in the QC step.

 df = df.dropna(subset=('temperature', 'reported_relative_humidity',
                        'u_wind', 'v_wind'), how='all').reset_index(drop=True)

So just take the change out of the `if self.suffix == '-drvd.txt' section, and leave the change in the 'else' section, and it should pass the tests.

[jrleeman]

Thanks @DanielWatkins and @swnesbitt - I just fixed this up a bit and rebased. Assuming that tests pass, we're good to go here.