Add support for Sentinel-2 MSI L2A SAFE datasets #2783

yukaribbba · 2024-04-18T10:34:33Z

Add support for Sentinel-2 MSI L2A SAFE datasets

Tests added
Fully documented

Composites family:

codecov · 2024-04-18T10:49:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.91%. Comparing base (3b9c04e) to head (ab55c4a).
Report is 516 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2783      +/-   ##
==========================================
- Coverage   95.95%   95.91%   -0.04%     
==========================================
  Files         379      366      -13     
  Lines       53888    53504     -384     
==========================================
- Hits        51708    51321     -387     
- Misses       2180     2183       +3

Flag	Coverage Δ
behaviourtests	`4.04% <0.00%> (-0.05%)`	⬇️
unittests	`96.01% <100.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2024-04-18T11:04:50Z

Pull Request Test Coverage Report for Build 9386597470

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

94 of 94 (100.0%) changed or added relevant lines in 2 files are covered.
572 unchanged lines in 61 files lost coverage.
Overall coverage increased (+0.008%) to 96.04%

Files with Coverage Reduction	New Missed Lines	%
satpy/readers/glm_l2.py	1	98.55%
satpy/readers/ascat_l2_soilmoisture_bufr.py	1	97.44%
satpy/tests/multiscene_tests/test_save_animation.py	1	99.1%
satpy/tests/reader_tests/_li_test_utils.py	1	99.26%
satpy/readers/msi_safe.py	1	98.21%
satpy/dataset/metadata.py	1	99.26%
satpy/tests/reader_tests/test_li_l2_nc.py	1	99.76%
satpy/tests/reader_tests/test_ascat_l2_soilmoisture_bufr.py	1	97.59%
satpy/tests/reader_tests/test_mersi_l1b.py	1	99.71%
satpy/readers/mws_l1b.py	1	98.52%

Totals
Change from base Build 9157044280:	0.008%
Covered Lines:	51561
Relevant Lines:	53687

💛 - Coveralls

simonrp84 · 2024-04-18T14:09:54Z

Thanks for adding this, it was on my to-do list but you got there first :-)

yukaribbba · 2024-04-18T15:05:23Z

Thanks for adding this, it was on my to-do list but you got there first :-)

🍺

yukaribbba · 2024-04-20T06:16:48Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

yukaribbba · 2024-04-20T08:35:16Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

mraspaud · 2024-05-27T12:14:51Z

Yes

mraspaud

Very nice that there is so little to change the msi reader python code! I have some questions inline.

satpy/dataset/dataid.py

satpy/etc/composites/msi.yaml

satpy/etc/readers/msi_safe_l2a.yaml

yukaribbba · 2024-05-27T15:39:31Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

mraspaud

Just a question really, why do the band names need to be different in l1 and l2?

satpy/etc/readers/msi_safe_l2a.yaml

yukaribbba · 2024-06-04T15:23:26Z

Well to be honest I'm not really sure about this but I want to make them look different so that users wouldn't mix them up. That's the first thing came into my mind...

yukaribbba · 2024-06-04T15:34:40Z

We have VIIRS SDR/EDR/L1B readers that can be considered as different process levels. L1B and SDR have the same band names while EDR has its own. So.......

yukaribbba · 2024-06-04T15:36:17Z

@djhoese What do you think about this?

djhoese · 2024-06-04T15:53:15Z

@yukaribbba I'm on parental leave so not sure I'll have time to review this. I'll try to take a look tonight, but have other work things I need to get to.

djhoese · 2024-06-05T01:45:36Z

Are you specifically asking me about the naming thing? Here is another example:

satpy/satpy/etc/readers/abi_l2_nc.yaml

Lines 19 to 24 in 7a7740a

    
           CMIP_C01:                          # Cloud Moisture Image Products Channel 1 
        
             name: C01 
        
             wavelength: [0.450, 0.470, 0.490] 
        
             calibration: reflectance 
        
             file_key: CMI 
        
             file_type: [abi_l2_cmip_c01, abi_l2_mcmip]

ABI L2 files include the original L1b bands but calibrated to reflectances and radiances. With discussions like this in the past it has come down to: are users likely to load/use L1 and L2 files as the same time? The answer is typically no. If they have L2 files they're usually looking for the higher-level products (AOD, etc) not the band reflectances and radiances. For the VIIRS example, SDR and L1B are two different software packages from two different organizations producing essentially the same output so there is no need for the products/bands to be named something different. The EDRs are almost always higher level products only available from those higher level files (L2/EDR). Otherwise a good general rule is to give products the names used in the files or by the organization in charge of the spacecraft or algorithm.

I know nothing of these files or this instrument, but I think the suffix (L2A) should be dropped. My main reason is that I'm guessing users will rarely be loading from both readers at the same time. If this is an incorrect assumption then maybe they do need special names. Even then, only the bands should have special names, not the higher level products.

mraspaud · 2024-06-05T06:22:35Z

Are you specifically asking me about the naming thing? Here is another example:

satpy/satpy/etc/readers/abi_l2_nc.yaml

Lines 19 to 24 in 7a7740a

CMIP_C01: # Cloud Moisture Image Products Channel 1

name: C01

wavelength: [0.450, 0.470, 0.490]

calibration: reflectance

file_key: CMI

file_type: [abi_l2_cmip_c01, abi_l2_mcmip]

ABI L2 files include the original L1b bands but calibrated to reflectances and radiances. With discussions like this in the past it has come down to: are users likely to load/use L1 and L2 files as the same time? The answer is typically no. If they have L2 files they're usually looking for the higher-level products (AOD, etc) not the band reflectances and radiances. For the VIIRS example, SDR and L1B are two different software packages from two different organizations producing essentially the same output so there is no need for the products/bands to be named something different. The EDRs are almost always higher level products only available from those higher level files (L2/EDR). Otherwise a good general rule is to give products the names used in the files or by the organization in charge of the spacecraft or algorithm.

I know nothing of these files or this instrument, but I think the suffix (L2A) should be dropped. My main reason is that I'm guessing users will rarely be loading from both readers at the same time. If this is an incorrect assumption then maybe they do need special names. Even then, only the bands should have special names, not the higher level products.

Here is an example for OLCI, where level1 and level2 channel have the same name, but not same modifiers and standard name for example:
https://github.com/pytroll/satpy/blob/main/satpy/etc/readers/olci_l1b.yaml#L61-L74
https://github.com/pytroll/satpy/blob/main/satpy/etc/readers/olci_l2.yaml#L96-L107

yukaribbba · 2024-06-05T10:28:54Z

k, we'll just drop it.

simonrp84 · 2024-06-05T14:09:09Z

The above discussion made me realise that the standard_names for the L2 reflectance datasets are wrong. At present, they are:

reflectance:
        standard_name: toa_bidirectional_reflectance
        units: "%"
      radiance:
        standard_name: toa_outgoing_radiance_per_unit_wavelength
        units: W m-2 um-1 sr-1

But as these are L2, they should instead be boa_... as atmospheric correction has been performed.

yukaribbba · 2024-06-05T14:29:14Z

@simonrp84 Thx for reminding! :)

mraspaud

LGTM, thanks a lot for putting all the efforts in improving the tests also!

yukaribbba added 2 commits April 18, 2024 18:33

initial

086f45f

Update msi_safe.yaml

79106e8

job part1 -- readers

9cdb03e

yukaribbba added 18 commits April 18, 2024 23:56

Update test_msi_safe.py

6f15335

Update test_msi_safe.py

9d9802a

Update msi_safe.yaml

2dcb4e9

Update msi_safe.py

8e98a49

Update test_msi_safe.py

3211d76

Update msi_safe.yaml

59a3f78

Update test_msi_safe.py

d3cbffa

yaml and test

b420b19

Update test_msi_safe.py

da5f04d

Update test_msi_safe.py

504590c

Update test_msi_safe.py

f851334

Update test_msi_safe.py

057e89e

Update test_msi_safe.py

bfa5f4b

Update test_msi_safe.py

c038cfd

Update test_msi_safe.py

e00e4d3

fix

e86b122

Update test_msi_safe.py

6b53da0

Update test_msi_safe.py

77b1753

pre-commit-ci bot and others added 2 commits April 20, 2024 06:17

[pre-commit.ci] auto fixes from pre-commit.com hooks

a788854

for more information, see https://pre-commit.ci

Update test_msi_safe.py

fe8ac5c

[pre-commit.ci] auto fixes from pre-commit.com hooks

80278e1

for more information, see https://pre-commit.ci

mraspaud reviewed May 27, 2024

View reviewed changes

yukaribbba added 4 commits May 27, 2024 21:28

Update msi.yaml

4a052bc

Update msi.yaml

b7c1b8e

dataid issues

d968b40

Update test_msi_safe.py

8194060

pre-commit-ci bot and others added 4 commits May 27, 2024 15:39

[pre-commit.ci] auto fixes from pre-commit.com hooks

911a6b6

for more information, see https://pre-commit.ci

dataid

6d18e86

Update msi.yaml

82f742f

composites

d2e3c9f

yukaribbba requested a review from mraspaud May 28, 2024 03:18

mraspaud reviewed Jun 4, 2024

View reviewed changes

satpy/etc/readers/msi_safe_l2a.yaml Outdated Show resolved Hide resolved

djhoese added enhancement code enhancements, features, improvements component:readers labels Jun 4, 2024

yukaribbba added 2 commits June 5, 2024 22:58

drop l2a suffix

676a5a1

Update msi_safe_l2a.yaml

ab55c4a

yukaribbba requested a review from mraspaud June 5, 2024 15:19

mraspaud approved these changes Jun 7, 2024

View reviewed changes

mraspaud merged commit 367016e into pytroll:main Jun 7, 2024
17 of 19 checks passed

yukaribbba deleted the add_support_msi_l2a branch June 8, 2024 04:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Sentinel-2 MSI L2A SAFE datasets #2783

Add support for Sentinel-2 MSI L2A SAFE datasets #2783

yukaribbba commented Apr 18, 2024 •

edited

Loading

codecov bot commented Apr 18, 2024 •

edited

Loading

coveralls commented Apr 18, 2024 •

edited

Loading

simonrp84 commented Apr 18, 2024

yukaribbba commented Apr 18, 2024

yukaribbba commented Apr 20, 2024

yukaribbba commented Apr 20, 2024

mraspaud commented May 27, 2024

mraspaud left a comment

yukaribbba commented May 27, 2024

mraspaud left a comment

yukaribbba commented Jun 4, 2024

yukaribbba commented Jun 4, 2024

yukaribbba commented Jun 4, 2024

djhoese commented Jun 4, 2024

djhoese commented Jun 5, 2024

mraspaud commented Jun 5, 2024 •

edited

Loading

yukaribbba commented Jun 5, 2024

simonrp84 commented Jun 5, 2024

yukaribbba commented Jun 5, 2024

mraspaud left a comment

Add support for Sentinel-2 MSI L2A SAFE datasets #2783

Add support for Sentinel-2 MSI L2A SAFE datasets #2783

Conversation

yukaribbba commented Apr 18, 2024 • edited Loading

codecov bot commented Apr 18, 2024 • edited Loading

Codecov Report

coveralls commented Apr 18, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9386597470

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

simonrp84 commented Apr 18, 2024

yukaribbba commented Apr 18, 2024

yukaribbba commented Apr 20, 2024

yukaribbba commented Apr 20, 2024

mraspaud commented May 27, 2024

mraspaud left a comment

Choose a reason for hiding this comment

yukaribbba commented May 27, 2024

mraspaud left a comment

Choose a reason for hiding this comment

yukaribbba commented Jun 4, 2024

yukaribbba commented Jun 4, 2024

yukaribbba commented Jun 4, 2024

djhoese commented Jun 4, 2024

djhoese commented Jun 5, 2024

mraspaud commented Jun 5, 2024 • edited Loading

yukaribbba commented Jun 5, 2024

simonrp84 commented Jun 5, 2024

yukaribbba commented Jun 5, 2024

mraspaud left a comment

Choose a reason for hiding this comment

yukaribbba commented Apr 18, 2024 •

edited

Loading

codecov bot commented Apr 18, 2024 •

edited

Loading

coveralls commented Apr 18, 2024 •

edited

Loading

mraspaud commented Jun 5, 2024 •

edited

Loading