Clip negative FCI radiances #3013

gerritholl · 2024-12-09T15:43:47Z

Optionally clip negative FCI radiances.

Closes artefacts in FCI RGBs using 3.8 µm #3009
Tests added
Fully documented

codecov · 2024-12-09T15:49:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.10%. Comparing base (5984c29) to head (d404a79).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3013   +/-   ##
=======================================
  Coverage   96.10%   96.10%           
=======================================
  Files         377      377           
  Lines       55163    55180   +17     
=======================================
+ Hits        53012    53029   +17     
  Misses       2151     2151

Flag	Coverage Δ
behaviourtests	`3.94% <0.00%> (-0.01%)`	⬇️
unittests	`96.19% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2024-12-09T16:03:30Z

Pull Request Test Coverage Report for Build 12391703039

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

22 of 22 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.002%) to 96.208%

Totals
Change from base Build 12227072168:	0.002%
Covered Lines:	53273
Relevant Lines:	55373

💛 - Coveralls

gerritholl

start behave test

rymdulf · 2024-12-13T11:02:30Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:04:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-13T11:08:13Z

rymdulf is lying, the tests were not successful. Failed with:

  Scenario Outline: Compare generated image with reference image -- @1.1         # features/image_comparison.feature:10
    Given I have a airmass reference image file from GOES17 resampled to <area>  # features/steps/image_comparison.py:54
[ WARN:[email protected]] global loadsave.cpp:241 findDecoder imread_('/app/ext_data/reference_images/reference_image_GOES17_airmass_<area>.png'): can't open/read file: check file path/integrity
    When I generate a new airmass image file from GOES17 with abi_l1b for <area> # features/steps/image_comparison.py:64
      Traceback (most recent call last):
        File "/app/venv/lib/python3.10/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/app/venv/lib/python3.10/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/image_comparison.py", line 82, in step_when_generate_image
          ls = scn.resample(area)
        File "/app/repository/satpy/scene.py", line 980, in resample
          self._resampled_scene(new_scn, destination, resampler=resampler,
        File "/app/repository/satpy/scene.py", line 869, in _resampled_scene
          destination_area = self._get_finalized_destination_area(destination_area, new_scn)
        File "/app/repository/satpy/scene.py", line 905, in _get_finalized_destination_area
          destination_area = get_area_def(destination_area)
        File "/app/repository/satpy/resample.py", line 231, in get_area_def
          return parse_area_file(get_area_file(), area_name)[0]
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 147, in parse_area_file
          return _parse_yaml_area_file(area_file_name, *regions)
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 201, in _parse_yaml_area_file
          raise AreaNotFound('Area "{0}" not found in file "{1}"'.format(area_name, area_file_name))
      pyresample.area_config.AreaNotFound: 'Area "<area>" not found in file "[\'/app/repository/satpy/etc/areas.yaml\']"'

gerritholl

start behave test

rymdulf · 2024-12-13T11:56:22Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:59:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T12:11:20Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T12:15:25Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:27:59Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:32:00Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:43:41Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:47:11Z

The testing process was executed successfully. See the test results for this pull request here!

Trying John Cintineos workaround.

gerritholl

run behave test

gerritholl

start behave test

rymdulf · 2024-12-13T14:59:53Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:02:46Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-16T07:29:29Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T07:33:04Z

The testing process was executed successfully. See the test results for this pull request here!

Add a unit test for FCI radiance clipping. The unit test fails, because there is no implementation yet.

Clip negative radiances when a keyword arguments asks for it.

gerritholl

start behave test

rymdulf · 2024-12-16T17:01:28Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T17:05:00Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-16T17:38:40Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T17:42:25Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-16T17:46:22Z

Good news:

Negative radiances are clipped.
No more black holes in the cloudtop RGB.

Bad news:

Still green holes in the night microphysics RGB.

gerritholl

start behave test

rymdulf · 2024-12-16T18:08:37Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T18:12:28Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-17T13:49:16Z

It's clipping space pixels, which we probably don't want? How do ABI and AHI prevent that?

djhoese · 2024-12-17T19:45:20Z

I think ABI, through normal ground processing, have masked space pixels. AHI has space pixels masked by default in Satpy I think (@simonrp84?) so maybe we just don't care? Or is radiance clipping not implemented for AHI?

simonrp84 · 2024-12-18T07:46:04Z

I don't understand your comment, @gerritholl: The space pixels are masked in the raw FCI data so it shouldn't be clipping them. They should already be nodata.

Also, @djhoese IIRc there's no clipping applied to the AHI data. It's the only modern GEO sat that doesn't mask space pixels in the data, but we do by default mask them in satpy.

gerritholl · 2024-12-18T18:34:15Z

I don't understand your comment, @gerritholl: The space pixels are masked in the raw FCI data so it shouldn't be clipping them. They should already be nodata.

Yes, but I had a bug that was not only setting negative radiances to a low radiance, but also nan radiances. It was retaining only pixels where data>lo was True, and this was false for the nan space pixels. Fixed that in d404a79.

Started behave test for FCI cloudtop

576e941

gerritholl added 3 commits December 9, 2024 18:57

Continue development of FCI imagery tests

d4a1e20

Change reference data and area for ABI test

1f562ae

Improve reference data generation script.

2297ee7

gerritholl commented Dec 13, 2024

View reviewed changes

Fix feature syntax and filename setup

02989b9

gerritholl commented Dec 13, 2024

View reviewed changes

import hdf5plugin first despite isort

8ce9a62

gerritholl commented Dec 13, 2024

View reviewed changes

Does it help if I do Meteosat-12 first?

7cd8858

gerritholl commented Dec 13, 2024

View reviewed changes

workaround for failing hdf5lpguni?

4d13d12

Trying John Cintineos workaround.

gerritholl commented Dec 13, 2024

View reviewed changes

gerritholl commented Dec 16, 2024

View reviewed changes

gerritholl added 2 commits December 16, 2024 17:16

Add unit test for FCI radiance clipping

50a6dfc

Add a unit test for FCI radiance clipping. The unit test fails, because there is no implementation yet.

Clip negative radiances

5d0675b

Clip negative radiances when a keyword arguments asks for it.

gerritholl commented Dec 16, 2024

View reviewed changes

gerritholl added 2 commits December 16, 2024 18:22

Clip in image comparison tests

aae848b

revert erroneous removal of **kwargs

5f6598b

gerritholl marked this pull request as ready for review December 16, 2024 17:38

gerritholl requested review from ameraner, djhoese and mraspaud as code owners December 16, 2024 17:38

gerritholl commented Dec 16, 2024

View reviewed changes

gerritholl requested a review from simonrp84 December 17, 2024 07:56

gerritholl marked this pull request as draft December 17, 2024 13:56

gerritholl marked this pull request as ready for review December 17, 2024 14:57

add doc & don't clip space pxiels

d404a79

gerritholl mentioned this pull request Dec 18, 2024

Combine low level moisture with a cloud mask #3022

Draft

3 tasks

Clip negative FCI radiances #3013

Are you sure you want to change the base?

Clip negative FCI radiances #3013

Conversation

gerritholl commented Dec 9, 2024 • edited Loading

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

coveralls commented Dec 9, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12391703039

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 16, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 17, 2024

djhoese commented Dec 17, 2024

simonrp84 commented Dec 18, 2024

gerritholl commented Dec 18, 2024 • edited Loading

gerritholl commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

coveralls commented Dec 9, 2024 •

edited

Loading

gerritholl commented Dec 18, 2024 •

edited

Loading