Passing multiple readers fails if generic_image is among them

[gerritholl]

Describe the bug

If passing multiple readers to the Scene class constructor, and one of the readers is the generic_image readers, Satpy fails with TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime', because the Scene class attempts to find the time extrema among all readers, but the generic_image reader does not define any start_time attribute (which is reasonable, because the geotiff I'm reading is timeless).

To Reproduce

from satpy import Scene
from glob import glob
from satpy.utils import debug_on
debug_on()
Scene(reader=["seviri_l1b_hrit", "generic_image"], filenames=glob("/tmp/seviri/H-000*201312121000*")+["/media/nas/x21308/DEM/dem_eu_1km.tif"])

Expected behavior
I'm not sure what I expect yet because I'm not sure what passing multiple readers is supposed to do! But what I would like to happen (this may be a feature request rather than a bug) is to get a Scene in which sc.available_dataset_names() is a union of the ones corresponding to all the readers I have passed. Ultimately, I would like to create a compositor that uses fields from both seviri_l1b_hrit, nwcsaf-geo, as well as a static DEM from a geotiff read with generic_image.

Actual results

$ python mwe.py
/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/pyspectral/config.py:75: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = recursive_dict_update(config, yaml.load(fp_))
[DEBUG: 2019-04-12 15:25:14 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/geotools/etc/'
[DEBUG: 2019-04-12 15:25:14 : satpy.readers] Reading ['/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/satpy/etc/readers/seviri_l1b_hrit.yaml']
[DEBUG: 2019-04-12 15:25:14 : satpy.readers.yaml_reader] Assigning to seviri_l1b_hrit: ['/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000021___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000015___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000023___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000020___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000011___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000022___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000018___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000009___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000012___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000019___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000024___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000010___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000017___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000013___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000014___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000016___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-HRV______-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_016___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_039___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_087___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_097___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_108___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_120___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-IR_134___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS006___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-VIS008___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_062___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000007___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000004___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000005___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000001___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000002___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000006___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000003___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-WV_073___-000008___-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-_________-PRO______-201312121000-__', '/tmp/seviri/H-000-MSG3__-MSG3________-_________-EPI______-201312121000-__']
[INFO: 2019-04-12 15:25:14 : hrit_msg] No IMPF configuration field found in prologue.
[DEBUG: 2019-04-12 15:25:14 : hrit_msg] Computing actual satellite position
[DEBUG: 2019-04-12 15:25:14 : satpy.readers] Reading ['/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/satpy/etc/readers/generic_image.yaml']
[DEBUG: 2019-04-12 15:25:14 : satpy.readers.yaml_reader] Assigning to generic_image: ['/media/nas/x21308/DEM/dem_eu_1km.tif']
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f12ff17c0b8>
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Starting outermost env
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] No GDAL environment exists
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f12ff17c320> created
[DEBUG: 2019-04-12 15:25:14 : rasterio._env] GDAL_DATA found in environment: '/media/nas/x21324/miniconda3/envs/py37d/share/gdal'.
[DEBUG: 2019-04-12 15:25:14 : rasterio._env] PROJ_LIB found in environment: '/media/nas/x21324/miniconda3/envs/py37d/share/proj'.
[DEBUG: 2019-04-12 15:25:14 : rasterio._env] Started GDALEnv <rasterio._env.GDALEnv object at 0x7f12ff17c320>.
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Got a copy of environment <rasterio._env.GDALEnv object at 0x7f12ff17c320> options
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f12ff17c0b8>
[DEBUG: 2019-04-12 15:25:14 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2019-04-12 15:25:14 : rasterio._base] Dataset <open DatasetReader name='/media/nas/x21308/DEM/dem_eu_1km.tif' mode='r'> is started.
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f12ff17c0b8>
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f12ff17c320> options
[DEBUG: 2019-04-12 15:25:14 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f12ff17c320>.
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Exiting outermost env
[DEBUG: 2019-04-12 15:25:14 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f12ff17c0b8>
Traceback (most recent call last):
  File "mwe.py", line 6, in <module>
    filenames=glob("/tmp/seviri/H-000*201312121000*")+["/media/nas/x21308/DEM/dem_eu_1km.tif"])
  File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/satpy/scene.py", line 159, in __init__
    self.attrs.update(self._compute_metadata_from_readers())
  File "/media/nas/x21324/miniconda3/envs/py37d/lib/python3.7/site-packages/satpy/scene.py", line 177, in _compute_metadata_from_readers
    for x in self.readers.values())
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'

Environment Info:

• OS: Linux openSUSE 15.0
• Satpy Version: 0.14.0+5.ge385d17b
• PyResample Version: 1.11.2
• Readers and writers dependencies (when relevant): [run from satpy.config import check_satpy; check_satpy()]

Readers
=======
abi_l1b: ok
abi_l1b_scmi: ok
acspo: ok
ahi_hrit: ok
ahi_hsd: ok
amsr2_l1b: ok
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt: cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud: ok
clavrx: ok
electrol_hrit: ok
fci_l1c_fdhsi: ok
generic_image: ok
geocat: ok
ghrsst_l3c_sst: ok
goes-imager_hrit: ok
goes-imager_nc: ok
grib: cannot find module 'satpy.readers.grib' (No module named 'pygrib')
iasi_l2: ok
jami_hrit: ok
li_l2: ok
maia: ok
modis_l1b: ok
msi_safe: cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
mtsat2-imager_hrit: ok
nucaps: ok
nwcsaf-geo: ok
nwcsaf-pps_nc: ok
olci_l1b: ok
olci_l2: ok
omps_edr: ok
safe_sar_l2_ocn: ok
sar-c_safe: ok
scatsat1_l2b: ok
seviri_l1b_hrit: ok
seviri_l1b_native: ok
seviri_l1b_nc: ok
slstr_l1b: ok
viirs_compact: ok
viirs_edr_active_fires: ok
viirs_edr_flood: ok
viirs_l1b: ok
viirs_sdr: ok
virr_l1b: ok

Writers
=======
cf: ok
geotiff: ok
mitiff: ok
ninjotiff: cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi: ok
simple_image: ok

[gerritholl]

Oh, maybe it's because I passed a list rather than a dict for filenames...

[djhoese]

I mentioned this on slack, but I'll say it again here for the record: What you've done should technically work, but the generic_image reader is one of the more difficult readers to working this way. If your filenames had date/times in them then the reader might return a valid datetime object and may work as expected.

You could pass things as a dictionary to filenames {'generic_image': [list of tifs], 'seviri_l1b_hrit': [list of seviri files]} but that may still fail on this check. This method is slightly faster because satpy doesn't have to go through all of the files and figure out which one belongs to which reader. You've told it you know where they should go.

I'm not sure what the best way around this is for generic_image. It needs to provide a datetime to play nicely with other parts of Satpy.

[gerritholl]

Right, I confirm that I get the same failure when I pass a dictionary for the filenames (and leave out the reader keyword argument).

[gerritholl]

(I just ran into this again)

It's always possible to come up with some date/time in the generic_image reader:

• Search headers for creation dates; the specifics depend on the image file format, so this step can be involved as it would potentially cover a large number of cases. For example, there may be a creation time in a tiff header.
• If nothing found in the headers, use stat on the source file. If available, use creation time. If no creation time is available, use modification time.

But how important are the actual values? How are these times used by Satpy? Is Satpy going to behave any differently if times are found to not overlap?

[djhoese]

But how important are the actual values? How are these times used by Satpy? Is Satpy going to behave any differently if times are found to not overlap?

Good question. I think start/end times are used in a couple cases (some that don't apply for generic_image):

1. Used to sort file handlers for a single file handler (ex. "this granule comes before this granule").
2. Used to give the Scene its attributes (.attrs), but probably not used much after that besides us wanting to provide a standard set of metadata.
3. Used in the output filename (ex. {start_time:%Y%m%d_%H%M%S}).

There are fancier use cases that may use it like the MultiScene's timeseries blending function that joins DataArrays along a time dimension. Not likely with generic_image. So I'm not really sure what the best solution is here. It would be nice if there was a start_time in the generic_image DataArrays but that also makes it harder to detect when we should ignore that start_time (like in a composite where the background image start_time should be ignored but the foreground should be used).

[gerritholl]

Would having start_time = numpy.datetime64('NaT') be appropriate and desirable?

[djhoese]

Maybe...I don't have a lot of experience with NaT so I'm a little worried it will break things, but maybe that is OK for now since this is just the generic image reader.

[gerritholl]

We should check if #1560 accidentally fixes this or at least allows an easy workaround.

[djhoese]

No, #1560 shouldn't be fixing this in the default case since it shouldn't be touching the start/end time metadata.

[gerritholl]

Whatever has changed, this works with current Satpy.