avoid invalid rows in measurements dataframes

[veenstrajelmer]

• ddlpy version: master
• Python version: 3.11
• Operating System: Windows

Description

When retreiving data, sometimes an invalid row is included in the resulting dataframe. The example code below prints:

     Grootheid.code Grootheid.Omschrijving
0            WATHTE            Waterhoogte
2354           None                   None

However, this dataframe should only contain GrootheidCode=WATHTE

What I Did

import ddlpy
import pandas as pd

locations = ddlpy.locations()
bool_grootheid = locations['Grootheid.Code'].isin(['WATHTE'])
bool_stations = locations.index.isin(["F3"])
selected = locations.loc[bool_grootheid & bool_stations]

time_min = pd.Timestamp("2024-01-01")
time_max = pd.Timestamp("2024-02-01")

measurements = ddlpy.measurements(selected.iloc[0], time_min, time_max)

print()
print(measurements[["Grootheid.code","Grootheid.Omschrijving"]].drop_duplicates())

This is also the case for MSL/1980/2024 for F3PFM/F3/F16/L9PFM/etc

Analysis

This happens because of an inconvenient invalid value filter:
https://github.com/openearth/ddlpy/blob/effcf06e70d7944fb2775f7de6f9e4e458955758/ddlpy/ddlpy.py#L141-L142

Replace this with a check for the Kwaliteitswaardecode (code insipred by hatyan.getonlinedata.ddlpy_to_hatyan():

# first flatten the KwaliteitswaardecodeLijst
import pandas as pd
qc = pd.to_numeric(ddlpy_meas['WaarnemingMetadata.KwaliteitswaardecodeLijst'].str[0],downcast='integer')
# then use it to set all numeric values to nan
bool_nan = qc==99
ddlpy_meas["Meetwaarde.Waarde_Numeriek"].loc[bool_nan] = np.nan
#alternatively check for "["99"]" directly, if possible (without flattening)