Cannot load ABI or FCI L1 datasets from Scene combining L1 and L2

[gerritholl]

Describe the bug

When I load ABI or FCI L1 and L2 data into the same Scene, there are restrictions on loading L1 datasets.

To Reproduce

ABI example:

from satpy import Scene
from satpy.utils import debug_on; debug_on()

files_l1b = ["/media/nas/x21308/abi/G16-ABI-L1B-RadC/2021/OR_ABI-L1b-RadC-M6C10_G16_s20213332001177_e20213332003561_c20213332004003.nc"]
files_l2 = ["/media/nas/x21308/abi/G16-L2-ACHAC/OR_ABI-L2-ACHAC-M6_G16_s20213332001177_e20213332003550_c20213332007025.nc"]
global_scene = Scene({"abi_l1b": files_l1b, "abi_l2_nc": files_l2})

global_scene.load(["C10"])

FCI example:

import hdf5plugin
from satpy import Scene
from glob import glob
sc = Scene(
        filenames={
            "fci_l1c_nc": glob("/media/nas/x23352/MTG/FCI/L1c/2024/12/17/12/*FDHSI*BODY*O_0073_*.nc"),
            "fci_l2_nc": ["/media/nas/x23352/MTG/FCI/L2-cases/202412171200/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-2-CLM--FD------NC4E_C_EUMT_20241217121531_L2PF_OPE_20241217120000_20241217121000_N__C_0073_0000.nc"]})
sc.load([0.91])

Expected behavior

I expect the loading to work just as it would (and does) when I pass only L1 files upon Scene initiation.

Actual results

It fails to load channel 10 because it says the file types abi_l2_cmip_c10 and abi_l2_cmip are missing. That's wrong. I shouldn't and don't need L2 CMIP data to load L1 channel 10.

Results for ABI:

[DEBUG: 2021-12-01 14:08:30 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/abi_l1b.yaml',)
[DEBUG: 2021-12-01 14:08:30 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/media/nas/x21308/abi/G16-ABI-L1B-RadC/2021/OR_ABI-L1b-RadC-M6C10_G16_s20213332001177_e20213332003561_c20213332004003.nc']
[DEBUG: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/abi_l2_nc.yaml',)
[DEBUG: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Assigning to abi_l2_nc: ['/media/nas/x21308/abi/G16-L2-ACHAC/OR_ABI-L2-ACHAC-M6_G16_s20213332001177_e20213332003550_c20213332007025.nc']
[DEBUG: 2021-12-01 14:08:31 : satpy.composites.config_loader] Looking for composites config file abi.yaml
[DEBUG: 2021-12-01 14:08:31 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[WARNING: 2021-12-01 14:08:31 : satpy.readers.yaml_reader] Required file type '['abi_l2_cmip_c10', 'abi_l2_mcmip']' not found or loaded for 'C10'
[WARNING: 2021-12-01 14:08:31 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='C10', wavelength=WavelengthRange(min=7.24, central=7.34, max=7.44, unit='µm'), calibration=<calibration.brightness_temperature>, modifiers=())

Resampling does not generate C10 either.

Results for FCI:

Required file type 'nc_fci_crm' not found or loaded for 'crm_vis09'
The following datasets were not created and may require resampling to be generated: DataID(name='crm_vis09', wavelength=WavelengthRange(min=0.894, central=0.914, max=0.934, unit='µm'), resolution=1000, modifiers=())

Environment Info:

• OS: openSUSE Leap 15.3
• Satpy Version: v0.31.0-158-g5dae1e6c

[djhoese]

Oh very interesting. So the problem is that the ABI L2 reader has a C10 that it can load:

satpy/satpy/etc/readers/abi_l2_nc.yaml

Lines 80 to 85 in 4950ccd

	CMIP_C10: # Cloud Moisture Image Products Channel 10
	name: C10
	wavelength: [7.24, 7.34, 7.44]
	calibration: brightness_temperature
	file_key: CMI
	file_type: [abi_l2_cmip_c10, abi_l2_mcmip]

If you asked for the "calibration='radiance'" version of that channel I would suspect it to work. I'm not sure there is an easy way to fix this. It would require the portion of the Scene/DependencyTree that is asking the reader for a dataset to check all readers that know about a dataset and attempt to load the dataset from each one until it is successful (or fails).

[gerritholl]

I vote for renaming the ABI L2 C10 product...

[djhoese]

The only issue with that is then they can't be used when making composites as most of the composites use the specific product name.

[gerritholl]

Possibly related? #2331

[djhoese]

Wow, can't believe this issue already exists. @simonrp84 brought this up on slack which I'm guessing is why you commented here. In his case he was able to get a L1b + L2 composite to work when he updated the composites using L1b channels to specify a resolution: X since the L2 reader's DataID doesn't include resolution, but the L1b reader does. This is a workaround, but only lucky in this particular case since logically the L2 reader should be able (allowed) to be specify the resolution. Also important to note, he was not provided the L2 files that provided the equivalent of the L1b channels so the dependency tree wanted to use DataIDs that weren't available/loadable.

The main issue(s) I discovered after last diving into the dependency tree code is that the dependency tree looks at all known DataIDs and doesn't consider whether or not the DataIDs are available. Also the dep tree says "here are all the DataIDs from all the readers that match the request, which one matches most closely". In the L1b/L2 case, just by chance of the way it was coded, when a resolution isn't specified in the request, the resolution in the DataID is ignored and the L2 (no resolution) DataID is considered a "better match". If the L2 DataID did define a resolution I have a feeling we'd get a TooManyResults exception meaning that the dependency tree found two or more DataIDs that matched equally well to the requested DataQuery.

Regardless of the above, even if L2 specified resolution for its DataIDs and even if the user provided the necessary L2 data files, the dependency tree should probably be able to resolve/choose these conflicts based on:

1. What's available. For example, L2's version isn't available but L1b is, use L1b.
2. Some inherent order or user provided order. The user may initialize the Scene with a filenames={"reader": [...], "reader2": [...]} dictionary where the dict has some order associated with it depending on how it was created. Or the readers could have some relationship/understanding that the L2 reader is reading "descendant" datasets of the L1b data. I'm not sure if in that case the dependency tree should prefer L2 (the descendant/child) or L1b (the ancestor/parent).

[simonrp84]

On your second point, I think that if both L1 and L2 dataset files are provided by the user them the higher level dataset should be used by default.

[gerritholl]

This also affects FCI:

import hdf5plugin
from satpy import Scene
from glob import glob
sc = Scene(
        filenames={
            "fci_l1c_nc": glob("/media/nas/x23352/MTG/FCI/L1c/2024/12/17/12/*FDHSI*BODY*O_0073_*.nc"),
            "fci_l2_nc": ["/media/nas/x23352/MTG/FCI/L2-cases/202412171200/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-2-CLM--FD------NC4E_C_EUMT_20241217121531_L2PF_OPE_20241217120000_20241217121000_N__C_0073_0000.nc"]})
sc.load([0.91])

fails with

Required file type 'nc_fci_crm' not found or loaded for 'crm_vis09'
The following datasets were not created and may require resampling to be generated: DataID(name='crm_vis09', wavelength=WavelengthRange(min=0.894, central=0.914, max=0.934, unit='µm'), resolution=1000, modifiers=())

In this case, the level-2 product defines the wavelength, so there is no product we can rename.

satpy/satpy/etc/readers/fci_l2_nc.yaml

Lines 779 to 788 in 7a0dc1a

	crm_vis09:
	name: crm_vis09
	long_name: TOA Bidirectional Reflectance at 0.91um (temporal average)
	standard_name: toa_bidirectional_reflectance
	resolution: 1000
	wavelength: [0.894, 0.914, 0.934]
	file_type: nc_fci_crm
	nc_key: mean_clear_sky_reflectance
	units: '%'
	vis_channel_id: 4