Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New compositors: MultiFiller and LongitudeMaskingCompositor #1577

Merged
merged 3 commits into from
Mar 12, 2021
Merged

New compositors: MultiFiller and LongitudeMaskingCompositor #1577

merged 3 commits into from
Mar 12, 2021

Conversation

ch-k
Copy link
Contributor

@ch-k ch-k commented Feb 26, 2021 •
edited by mraspaud
Loading

I've added to more compositors. They are necessary to create a "world composite" (global mosaic) as described in PR #1560 .
The MultiFiller is an extension of Filler but works with multiple projectables (including data from optional_datasets). Maybe Filler might be replaced by this one, but I am unsure if it has any side effects.

The LongitudeMaskingCompositor makes it possible to mask areas outside spezified min and max longitude values.

  • Tests added
  • Fully documented

@codecov
Copy link

codecov bot commented Feb 26, 2021 •
edited
Loading

Codecov Report

Merging #1577 (ccad48d) into master (9f135b1) will decrease coverage by 1.07%.
The diff coverage is 98.43%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1577      +/-   ##
==========================================
- Coverage   92.18%   91.10%   -1.08%     
==========================================
  Files         251      251              
  Lines       36777    36865      +88     
==========================================
- Hits        33902    33586     -316     
- Misses       2875     3279     +404
Flag Coverage Δ
behaviourtests 4.50% <7.81%> (+0.01%) ⬆️
unittests 91.62% <98.43%> (-1.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
satpy/composites/__init__.py 88.63% <96.55%> (+0.36%) ⬆️
satpy/tests/test_composites.py 99.88% <100.00%> (+<0.01%) ⬆️
satpy/tests/reader_tests/test_seviri_l2_grib.py 23.07% <0.00%> (-76.93%) ⬇️
satpy/tests/reader_tests/test_seviri_l2_bufr.py 30.35% <0.00%> (-69.65%) ⬇️
satpy/readers/iasi_l2_so2_bufr.py 25.80% <0.00%> (-68.82%) ⬇️
satpy/readers/ascat_l2_soilmoisture_bufr.py 29.87% <0.00%> (-67.54%) ⬇️
satpy/tests/reader_tests/test_iasi_l2_so2_bufr.py 30.98% <0.00%> (-66.20%) ⬇️
...ts/reader_tests/test_ascat_l2_soilmoisture_bufr.py 37.20% <0.00%> (-60.47%) ⬇️
satpy/readers/seviri_l2_grib.py 26.53% <0.00%> (-60.21%) ⬇️
satpy/readers/seviri_l2_bufr.py 34.52% <0.00%> (-59.53%) ⬇️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f135b1...ccad48d. Read the comment docs.

@djhoese
Copy link
Member

djhoese commented Feb 26, 2021

Correct me if I'm wrong but couldn't the MultiFiller be replaced by using the Filler compositor on each of the inputs to an RGB?

RGB ->
  R = Filler -> input1
  G = Filler -> input2
  B = Filler -> input3

Just curious, what do you use the LongitudeFiller for. I know you said for global composites but in what way? Would it make sense to also allow for latitude?

@ghost
Copy link

ghost commented Feb 26, 2021

DeepCode failed to analyze this pull request

Something went wrong despite trying multiple times, sorry about that.
Please comment this pull request with "Retry DeepCode" to manually retry, or contact us so that a human can look into the issue.

@ch-k
Copy link
Contributor Author

ch-k commented Feb 26, 2021

Correct me if I'm wrong but couldn't the MultiFiller be replaced by using the Filler compositor on each of the inputs to an RGB?

RGB ->
  R = Filler -> input1
  G = Filler -> input2
  B = Filler -> input3

Just curious, what do you use the LongitudeFiller for. I know you said for global composites but in what way? Would it make sense to also allow for latitude?

The FillingCompositor fills each of the channels individually. Like a background for all bands. The MultiFiller is more like a chain of the existing Filler compositor. There's only one output. It can also use more than 3 projectables. In my case, I have images of 5 satellites. I'll attach a sample composite yaml in Monday.

@djhoese
Copy link
Member

djhoese commented Feb 27, 2021

Oh I see now. In the past we've used the MultiScene's blend method to stack separate Scene's together. You don't have to do it this way, but I'm curious how you're accomplishing this (combining multiple satellites into one Scene and compositor).

@ch-k
Copy link
Contributor Author

ch-k commented Mar 1, 2021 •
edited
Loading

Oh I see now. In the past we've used the MultiScene's blend method to stack separate Scene's together. You don't have to do it this way, but I'm curious how you're accomplishing this (combining multiple satellites into one Scene and compositor).

Sample use case for LongitudeMaskingCompositor and MultiFiller

sensor_name: world-images

composites:

  world_comp_zds_masked:
    compositor: !!python/name:satpy.composites.LongitudeMaskingCompositor
    prerequisites:
    - image_zds
    standard_name: world_comp_zds_masked     
    lon_min: -37.5
    lon_max: 20.75

  world_comp_goes16_masked:
    compositor: !!python/name:satpy.composites.LongitudeMaskingCompositor
    prerequisites:
    - image_goes16
    standard_name: world_comp_goes16_masked     
    lon_min: -105.
    lon_max: -37.5

  world_comp_goes17_masked:
    compositor: !!python/name:satpy.composites.LongitudeMaskingCompositor
    prerequisites:
    - image_goes17
    standard_name: world_comp_goes17_masked     
    lon_min: -177.15
    lon_max: -105.

  world_comp_iodc_masked:
    compositor: !!python/name:satpy.composites.LongitudeMaskingCompositor
    prerequisites:
    - image_iodc
    standard_name: world_comp_iodc_masked     
    lon_min: 20.75
    lon_max: 91.1

  world_comp_himawari8_masked:
    compositor: !!python/name:satpy.composites.LongitudeMaskingCompositor
    prerequisites:
    - image_himawari8
    standard_name: world_comp_himawari8_masked     
    lon_min: 91.1
    lon_max: -177.15


  dwd_world_comp:
    compositor: !!python/name:satpy.composites.MultiFiller
    prerequisites:
    - name: world_comp_zds_masked
    optional_prerequisites:
    - name: world_comp_goes16_masked
    - name: world_comp_goes17_masked 
    - name: world_comp_iodc_masked
    - name: world_comp_himawari8_masked
    standard_name: dwd_world_comp

The base channels are defined in following reader config:

reader:
    name: world_generic_image
    description: generic image reader
    reader: !!python/name:satpy.readers.yaml_reader.FileYAMLReader
    sensors: [world-images]
    default_channels: [image_goes16, image_goes17]

datasets:
  image_goes16:
    name: image_goes16
    file_type: graphic_goes16
  image_goes17:
    name: image_goes17
    file_type: graphic_goes17
  image_zds:
    name: image_zds
    file_type: graphic_zds
  image_iodc:
    name: image_iodc
    file_type: graphic_iodc
  image_himawari8:
    name: image_himawari8
    file_type: graphic_himawari8

file_types:
    graphic_goes16:
        file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler
        file_patterns:
         - 'goes16_{filename}{start_time:%Y%m%d%H%M}.tif'
         - 'goes16_{filename}.tif'
    graphic_goes17:
        file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler
        file_patterns:
         - 'goes17_{filename}{start_time:%Y%m%d%H%M}.tif'
         - 'goes17_{filename}.tif'
    graphic_zds:
        file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler
        file_patterns:
         - 'ZDS_{filename}{start_time:%Y%m%d%H%M}.tif'
         - 'ZDS_{filename}.tif'
    graphic_iodc:
        file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler
        file_patterns:
         - 'IODC_{filename}{start_time:%Y%m%d%H%M}.tif'
         - 'IODC_{filename}.tif'
    graphic_himawari8:
        file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler
        file_patterns:
         - 'himawari8_{filename}{start_time:%Y%m%d%H%M}.tif'
         - 'himawari8_{filename}.tif'

And this would be an example output:
worldcomp_geotiff

created with:

my_files = ['/vagrant/worldcomp/geotiff-in/goes16/wcm10km/goes16_IR_1035_ir_wcm10km_202102182330.tif', 
            '/vagrant/worldcomp/geotiff-in/goes17/wcm10km/goes17_IR_1035_ir_wcm10km_202102182330.tif',
            '/vagrant/worldcomp/geotiff-in/zds/wcm10km/ZDS_IR_108_ir_wcm10km_202102182330.tif',
            '/vagrant/worldcomp/geotiff-in/iodc/wcm10km/IODC_IR_108_ir_wcm10km_202102182330.tif',
            '/vagrant/worldcomp/geotiff-in/himawari8/wcm10km/himawari8_IR1_ir_wcm10km_202102182330.tif',
            
            ]
scn = Scene(reader="world_generic_image", filenames=my_files)
scn.load(['dwd_world_comp'])
scn.show('dwd_world_comp')

djhoese
djhoese approved these changes Mar 1, 2021
View reviewed changes
Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I understand the need for the longitude masking now. It was just odd to see it in a compositor.

@djhoese djhoese added component:compositors enhancement code enhancements, features, improvements labels Mar 1, 2021
@mraspaud mraspaud added this to the v0.26.0 milestone Mar 11, 2021
@mraspaud mraspaud merged commit ac92157 into pytroll:master Mar 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djhoese djhoese djhoese approved these changes

@mraspaud mraspaud Awaiting requested review from mraspaud

Assignees

@ch-k ch-k

Labels
component:compositors enhancement code enhancements, features, improvements
Projects
None yet
Milestone
v0.26.0
Development

Successfully merging this pull request may close these issues.
3 participants
@ch-k @djhoese @mraspaud