Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xarray cfgrib backend changed data variable name #678

Open
emanuel-schmid opened this issue Mar 22, 2023 · 10 comments
Open

xarray cfgrib backend changed data variable name #678

emanuel-schmid opened this issue Mar 22, 2023 · 10 comments
Assignees

Comments

@emanuel-schmid
Copy link
Collaborator

Up to xarray=2023.2.0, cfgrib=0.9.9.1 and eccodes=2.26.0 reading the storm_europe test files named "test_storm_europe_icon_2021012800" created a dataset with a Data variable 'gust'.

With xarray=2023.2.0, cfgrib=0.9.10.3 and eccodes=2.29.0 the (seemingly) same Data variable is called 'i20fg'.

The following code snippet (mostly taken from StormEurope.from_icon_grib) illustrates this:

mport datetime as dt
from pathlib import Path
import bz2
import xarray as xr

from climada.hazard import StormEurope
from climada.util.api_client import Client
from climada.util.dwd_icon_loader import download_icon_grib

apiclient = Client()
ds = apiclient.get_dataset_info(name='test_storm_europe_icon_2021012800', status='test_dataset')
dsdir, _ = apiclient.download_dataset(ds)

event_date = dt.datetime(2021, 1, 28)
model_name = 'test'

file_names = download_icon_grib(
    dt.datetime(2021, 1, 28),
    model_name='test',
    download_dir=dsdir)

# read intensity from files
for ind_i, file_i in enumerate(file_names):
    gripfile_path_i = Path(file_i[:-4])
    with open(file_i, 'rb') as source, open(gripfile_path_i, 'wb') as dest:
        dest.write(bz2.decompress(source.read()))
    ds_i = xr.open_dataset(gripfile_path_i, engine='cfgrib')
    if ind_i == 0:
        stacked = ds_i
    else:
        stacked = xr.concat([stacked,ds_i], 'valid_time')

print(stacked)

This is relevant since the current conda installation comes with the updated versions as above, and consequently runs into a unittest failure for climada.hazard.test.test_storm_europe.TestReader.test_icon_read

@emanuel-schmid
Copy link
Collaborator Author

the cfgrib package can probably excluded from the list of suspects: downgrading didn't help.

@emanuel-schmid
Copy link
Collaborator Author

emanuel-schmid commented Mar 23, 2023

Turned out that the change happened between eccodes 2.26 and 2.28.
In other words downgrading to eccodes 2.26 "solves" the problem of the failing unit test.

The release notes on 2.28 state that some names have changed but the details seem to be hidden in a non-accessible confluence page.

@ThomasRoosli ThomasRoosli self-assigned this Apr 13, 2023
@ThomasRoosli
Copy link
Collaborator

Indeed, the linked issue [ECC-1465] mentions parameter shortname 'i10fg'. I will research how the code could be adjusted, and how this might resolved together with the current restriction on cfgrib and eccodes.

@ThomasRoosli
Copy link
Collaborator

I did not have time yet to start fixing. this issue, but hope to have time soon, so we can increase the eccodes version freely again.

@peanutfun
Copy link
Member

eccodes v2.32 was released in October. If the problem persists, we should find a way to update our code and become compatible with newer versions. @emanuel-schmid @ThomasRoosli Does this only affect StormEurope or are other modules affected as well?

@luseverin
Copy link
Collaborator

Hey so I tried updating the changed parameter name in StormEurope.from_icon_grib() from gust to i10fg and now the integration test TestStormEurope().test_icon_read() seems to be passing under cfgrib=0.9.10.3 and eccodes=2.29.0. Regarding the exact reason for this, the confluence page of eccodes only says that a new i10fg parameter was added in version 2.28.0, and that it corresponds to instantaneous 10 metre wind gust (https://confluence.ecmwf.int/display/MTG2US/Changes+in+ecCodes+version+2.28.0+compared+to+the+previous+version). However, it is not really clear what happened to the gust parameter and if this is the exact replacement to it or not..
As far as I know about the StormEurope module, this should do the trick. Maybe @ThomasRoosli could confirm this. Otherwise I am not aware of other modules being affected by this issue but maybe @emanuel-schmid knows more about it? If you also think this is a sufficient fix, I can set up a pull request so that we can proceed with the updates the eccodes and cfgrib packages.

@peanutfun
Copy link
Member

@luseverin Excellent work, thanks for looking into this. Are these the latest versions of the two dependencies, and if not, can you try with the latest?

cfgrib=0.9.10.3 and eccodes=2.29.0

@luseverin
Copy link
Collaborator

Latest versions of the packages are (to my knowledge) cfgrib>=0.9.14 and eccodes>=2.35.1, however, updating eccodes leads to the following incompatibilities:

`Could not solve for environment specs
The following packages are incompatible
├─ eccodes >=2.35.1  is installable and it requires
│  └─ jasper >=4.2.4,<5.0a0 , which requires
│     └─ libjpeg-turbo >=3.0.0,<4.0a0 , which requires
│        └─ jpeg <0.0.0a , which can be installed;
└─ jpeg is not installable because it conflicts with any installable versions previously reported.`

So I think @emanuel-schmid or someone who knows about it should look into it to update the packages beyond cfgrib=0.9.10.3 and eccodes=2.29.0.

@peanutfun
Copy link
Member

Why jpeg is not installable eludes me here. But I will check if cfgrib=0.9.10.3 and eccodes=2.29.0 would already bring us to Python 3.12.

@peanutfun
Copy link
Member

But I will check if cfgrib=0.9.10.3 and eccodes=2.29.0 would already bring us to Python 3.12.

No help. It looks like we need cfgrib>=0.9.10 and eccodes>=2.31.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants