empty ancestor list in recipe_esacci_lst

[remi-kazeroni]

Describe the bug
The recipe_esacci_lst.yml fails at the diagnostic level because an ancestor list seems empty. The end of the log.txt of the only diagnostic ends with:

Traceback (most recent call last):
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/lst/lst.py", line 232, in <module>
    _diagnostic(config)
  File "/home/b/b309192/ESMValTool/esmvaltool/diag_scripts/lst/lst.py", line 170, in _diagnostic
    ancestor_list.append(ancestors['ts'][0])
KeyError: 'ts'

We got the same type of errors when testing recipes for #1609. I thought we had covered all possible cases. But it seems the list of ancestors is empty for the OBS dataset, see main_log_debug.txt:

   'save': {'compress': False,
            'filename': PosixPath('/work/bd0854/b309192/recipe_testing/recipe_testing_v2p8/v08rc1/scripts/esmvaltool_output/recipe_esacci_lst_20230306_230232/preproc/timeseries/ts/OBS_ESACCI-LST_sat_1.00_Amon_ts_2004-2005.nc')}}
  ancestors:
  None

Any idea what's happening here @bouweandela and @schlunma?

Please attach

• Run: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0rc1/recipe_esacci_lst_20230306_230232/

[bouweandela]

It looks like this happens because dataset ESACCI-LST has the wrong variable name, lst where it should be ts:

ncdump -h /work/bd0854/DATA/ESMValTool2/OBS/Tier2/ESACCI-LST/OBS_ESACCI-LST_sat_1.00_Amon_ts_200401-200412.nc

netcdf OBS_ESACCI-LST_sat_1.00_Amon_ts_200401-200412 {
dimensions:
	time = 12 ;
	lat = 3600 ;
	lon = 7200 ;
	bnds = 2 ;
variables:
	float lst(time, lat, lon) ;
		lst:_FillValue = 1.e+20f ;
		lst:standard_name = "surface_temperature" ;
		lst:long_name = "Surface Temperature" ;
		lst:units = "kelvin" ;
		lst:cell_methods = "time: mean" ;
	double time(time) ;
		time:axis = "T" ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1950-1-1 00:00:00" ;
		time:standard_name = "time" ;
		time:long_name = "reference time of file" ;
		time:calendar = "gregorian" ;
	double time_bnds(time, bnds) ;
	double lat(lat) ;
		lat:axis = "Y" ;
		lat:bounds = "lat_bnds" ;
		lat:units = "degrees_north" ;
		lat:standard_name = "latitude" ;
		lat:long_name = "latitude coordinate" ;
	double lat_bnds(lat, bnds) ;
	double lon(lon) ;
		lon:axis = "X" ;
		lon:bounds = "lon_bnds" ;
		lon:units = "degrees_east" ;
		lon:standard_name = "longitude" ;
		lon:long_name = "longitude coordinate" ;
	double lon_bnds(lon, bnds) ;

// global attributes:
		:geospatial_lon_max = 360. ;
		:geospatial_lon_min = 0. ;
		:information = "Mean of Day and Night Aqua MODIS monthly LST" ;
		:Conventions = "CF-1.7" ;
}

Since #1837, this information is updated from the preprocessed data to allow changing it in preprocessor functions, but this means that the input data does have to have the right variable name.

I'm no expert on this topic, but I think the solution to this issue would be to update the CMORizer so it creates data with the correct variable name.

[schlunma]

Looks like we found an example where #1907 is relevant 😉

[remi-kazeroni]

Thanks for taking a look @bouweandela! Good to see this is not really an issue for the upcoming release of the Core 👍

I'm no expert on this topic, but I think the solution to this issue would be to update the CMORizer so it creates data with the correct variable name.

I also think this is the right approach here. I'll open a separate issue in the Tool. Also the metadata look wrong and would need fixing.

Looks like we found an example where #1907 is relevant 😉

Maybe but I think we could try to avoid such problems when it comes to data created (CMORized) with ESMValTool

[schlunma]

Yeah, I also think that this should be fixed by changing the CMORizer, but if there was a warning or error in the first place then we would have see this problem much earlier (while writing the CMORizer).

[remi-kazeroni]

Yeah, I also think that this should be fixed by changing the CMORizer, but if there was a warning or error in the first place then we would have see this problem much earlier (while writing the CMORizer).

I agree, it should throw an error if possible. It could be useful to have a test/check that checks that newly added CMORizers produce data with the right name. @schlunma would you mind opening an issue about that in the Tool so the idea is not forgotten? I'll take a look at the faulty CMORizer in the meantime.

[schlunma]

I think it would be better add this check to our CMOR checks in ESMValCore as suggested in #1907. Since users are free to do the CMORization as they like (Python vs. NCL, using the utility functions or not, etc.) I don't think there is a place in the code where we can check the var_name consistently and independently of the method.