Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Merged
merged 28 commits into from
Oct 16, 2024

Conversation

dnaviap
Copy link
Contributor

@dnaviap dnaviap commented Jan 11, 2024

Add combined GRIB reader that can be used for both SEVIRI and FCI L2 products.

  • Tests added
  • Fully documented
  • Add @dnaviap to AUTHORS.md

@strandgren
Copy link
Collaborator

Thanks for working on this @dnaviap :) Before reviewing in more detail, I think we need some refactoring of the files:

  1. Instead of having a eum_l2_grib.yaml files I suggest to keep the seviri_l2_grib.yaml file (which is still there) and add the FCI cloud mask dataset in a new fci_l2_grib.yaml file. That way we don't change the user interface to the reader. That's also how it's being done for the BUFR reader (Changes to Eumetsat L2 BUFR reader #2603)
  2. eum_l2_grib.py should be compatible with both FCI and SEVIRI data, meaning that seviri_l2_grib.py should become obsolete and should therefore be removed.
  3. Similarly, the new test file test_eum_l2_grib.py should make test_seviri_l2_grib.py obsolete, meaning it can be removed as well.

@dnaviap
Copy link
Contributor Author

dnaviap commented Jan 12, 2024

The refactoring is complete and tested.

Copy link
Collaborator

@strandgren strandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:

    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """

That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved
satpy/readers/fci_base.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_eum_l2_grib.py Outdated Show resolved Hide resolved
@dnaviap dnaviap requested review from sfinkens and adybbroe as code owners May 13, 2024 08:47
@dnaviap
Copy link
Contributor Author

dnaviap commented May 13, 2024

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:

    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """

That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

@dnaviap dnaviap closed this May 13, 2024
@dnaviap dnaviap reopened this May 13, 2024
Copy link
Collaborator

@strandgren strandgren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes and the refactoring @dnaviap!

I left a couple of minor comments in-line. Could you please also merge the main branch into this branch and resolve the merge conflict such that all unit tests etc. run?

satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved
satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_eum_l2_grib.py Outdated Show resolved Hide resolved
strandgren and others added 4 commits June 13, 2024 14:05
@dnaviap dnaviap requested a review from sjoro as a code owner June 24, 2024 08:43
Copy link

codecov bot commented Jun 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.08%. Comparing base (b8991c0) to head (ba30733).
Report is 30 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2717   +/-   ##
=======================================
  Coverage   96.07%   96.08%           
=======================================
  Files         373      375    +2     
  Lines       54491    54578   +87     
=======================================
+ Hits        52352    52441   +89     
+ Misses       2139     2137    -2     
Flag Coverage Δ
behaviourtests 3.98% <0.00%> (-0.01%) ⬇️
unittests 96.18% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Jul 1, 2024

Pull Request Test Coverage Report for Build 9743868855

Details

  • 168 of 168 (100.0%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.01%) to 96.056%

Totals Coverage Status
Change from base Build 9702977764: 0.01%
Covered Lines: 51732
Relevant Lines: 53856

💛 - Coveralls

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, just a couple of comments and questions

satpy/tests/reader_tests/test_fci_base.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_eum_l2_grib.py Outdated Show resolved Hide resolved
Comment on lines 109 to 110
@mock.patch("satpy.readers.eum_l2_grib.xr")
@mock.patch("satpy.readers.eum_l2_grib.da")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see a bit of mocking. In our experience this can lead to problems down the road, and nowadays, we prefer to write a synthetic testing file to tmp_path and work with that instead. Do you think this would be doable for this format?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I don't know and I also don't have the time to look into it at this point. Would be grateful if we could tackle that in another PR in the future :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, should we make an issue about it then?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, would be a pity if we were to forget it 😃 I created an issue here: #2932

@strandgren
Copy link
Collaborator

pre-commit.ci autofix

@coveralls
Copy link

Pull Request Test Coverage Report for Build 11361065749

Details

  • 168 of 168 (100.0%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.01%) to 96.18%

Totals Coverage Status
Change from base Build 11324402105: 0.01%
Covered Lines: 52677
Relevant Lines: 54769

💛 - Coveralls

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud added enhancement code enhancements, features, improvements component:readers labels Oct 16, 2024
@mraspaud mraspaud merged commit a5a3227 into pytroll:main Oct 16, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants