Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle out-of-order ERA5 time coord issue #189

Merged

Conversation

leifdenby
Copy link
Contributor

It appears that ECMWF's CDS has started returning data where the time-coordinate isn't monotonic anymore and the timesteps are randomly ordered. This commit adds sorting by time during load-time. This may slow down calculations and a fix where input files are overwritten could be made in future

Related to #183

It appears that ECMWF's CDS has started returning data where the
time-coordinate isn't monotonic anymore and the timesteps are randomly
ordered. This commit adds sorting by time during load-time. This may
slow down calculations and a fix where input files are overwritten could
be made in future

Related to EUREC4A-UK#183
@leifdenby leifdenby requested a review from sjboeing December 8, 2022 14:35
@leifdenby
Copy link
Contributor Author

@sjboeing I've manually verified that this works. We can't test it with our test-data because we don't download data from ECMWF during testing. Could you give it a try too and let me know what you think? Thanks :)

@sjboeing
Copy link
Contributor

Hi Leif,

This looks good and it seems to work for me. While working on this, I did run into an issue with domain versioning:
"Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download."
To make it work, I removed the version from:
lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml
Let me know if this sounds familiar, if not I will make an issue for us to investigate.

@leifdenby
Copy link
Contributor Author

"Exception: The domain data in /home/stevenboeing/lagtraj_0922/lagtraj/data/domains/eurec4a_circle_data doesn't match the version stored in the data definition for lagtraj://eurec4a_circle (unversioned != 1.0.0). Please delete the domain data and re-download."
To make it work, I removed the version from:
lagtraj/input_definitions/examples_files/domains/eurec4a_circle.yaml

Hmm.. I think this means you downloaded this domain data before we added the versioning (#87). But this is fine :) Another way to solve it is just create a filename called VERSION in domains/eurec4a_circle_data which just contains 1.0.0

@leifdenby
Copy link
Contributor Author

I'll merge this in once the tests pass

@sjboeing with this change the changelog is getting quite long: https://github.com/leifdenby/lagtraj/blob/fix-era5-time-monotinicity-issue/CHANGELOG.md - how do you feel about me creating release v0.1.2 once this pull-request has been merged?

@leifdenby leifdenby merged commit 20bb79c into EUREC4A-UK:master Dec 12, 2022
@sjboeing
Copy link
Contributor

Ah, thanks Leif. merging this and creating a new release sounds good.

The other issue occurred with data downloaded just this week. I will check and see if it is something I missed and create an issue if it persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants