Coregistration of ocean colour data to SST dataset failing

[kjpearson]

Expected behaviour

Ocean colour should be resampled onto the SST grid after resampling with coregister

Actual behaviour

"validate" option returns successfully "no problems encountered" but executing the coregister operation fails with:

Invalid Input

'Oops, Cate couldn't carry out the requested Executing operation "coregister" because:

Could not find a valid intersection to perform coregistration on'

There is then the traceback info but I can't find a way to copy that here. The last call is to coregistration.py, line340, in _find_intersection

Steps to reproduce the problem

For speed, I have created a subset of the SST CCI L4 v1.1 data for 2004-2005, lat=[-10,10], lon=[-175,-115] to act as the master dataset for coregister. This worked successfully with the cloud cci data as the slave dataset

I have then downloaded a 2004-01-01,2004-05-01 subset of the Ocean Colour CCI dataset "Global attenuation coefficient for downwelling irradiance (Kd490) gridded on a geographic projection V2

Attempting to coregister these two datasets immediately fails because there the ocean colour dataset has a variable "crs" with only a time dimension. I used select_var to extract "MERIS_nobs_sum" ad separately "Kd_940" but trying to coregister either of these caused the error above. I can't see what the problem is as both of these appear to be global datasets.

Specifications

MacOS, Cate 2.0.0-dev16

[kjpearson]

I get the same error when trying to coregister either the "active" or "passive" soil moisture datasets against a master global SST dataset. I also get it when trying to coregister the two soil moisture datasets against each other. I have taken 4 month temporal subsets in all three cases for 2004-01-01 to 2004-05-01. These are all globally gridded data sets.

[kjpearson]

I said above that the cloud cci variables successfully regrid but I'm having second thoughts. Although the coregistration operation completes, it looks like the data for the whole globe in the cloud dataset has been remapped down to the subregion I had in the SST dataset, rather than just those elements of the cloud dataset that were already within the boundary of the subregion. At least, that's how it looks to me from the patterns in the data. The "cee" and "cfc" variables seem the best to look at for this. Could you take a look at this too? If you agree, maybe it should be split to a different issue.

[forman]

Hi @JanisGailis could you please get back to @kjpearson

[JanisGailis]

@kjpearson I will investigate this. I have seen a similar effect before when doing spatial subset, but that turned out to be a visualization issue.

[JanisGailis]

@kjpearson The effect with global data being 'compressed' into the subset should be fixed in latest dev release. The problem with finding intersection still persists.

[HelenClifton]

Hi @kjpearson @forman @JanisGailis I'm retesting the coregistration with cate-2.0.0-dev.21 and still seeting several combinations of datasets where I'm getting the error

Example traceback pasted in below
Traceback (most recent call last):
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
result = future.result()
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures_base.py", line 425, in result
return self.__get_result()
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\concurrent\futures\thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
result = method(*method_params, monitor=monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\webapi\websocket.py", line 292, in set_workspace_resource
monitor=monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\wsmanag.py", line 320, in set_workspace_resource
workspace.execute_workflow(res_name=res_name, monitor=monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workspace.py", line 662, in execute_workflow
self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 632, in invoke_steps
step.invoke(context=context, monitor=monitor.child(work=1))
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 318, in invoke
self._invoke_impl(_new_context(context, step=self), monitor=monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\workflow.py", line 980, in _invoke_impl
return_value = self._op(monitor=monitor, **input_values)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\core\op.py", line 216, in call
return_value = self._wrapped_op(**input_values)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 140, in coregister
return _resample_dataset(ds_master, ds_slave, methods_us[method_us], methods_ds[method_ds], monitor)
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 296, in _resample_dataset
global_bounds=(-90, 90))
File "C:\Users\HClifton\cate-2.0.0.dev20\lib\site-packages\cate\ops\coregistration.py", line 341, in _find_intersection
raise ValidationError('Could not find a valid intersection to perform'
cate.core.types.ValidationError: Could not find a valid intersection to perform coregistration on

[HelenClifton]

@forman @JanisGailis Please could you re-open this issue and take a look. It's apparently an "Invalid Input" error so I don't know if it's related to the datasets or the code

[JanisGailis]

@HelenClifton Thanks for the report!

However, I won't reopen this issue, unless it's exactly the same datasets, as the error can be caused by many underlying data issues. Can you please open a new issue where you outline which datasets you're trying to coregister when you get the error?

[JanisGailis]

@HelenClifton One more thing. Norman did a release, but I'm not sure if the dev21 actually includes the latest changes from 'cate core', as he forgot to do something in that repository, as evidenced by C:\Users\HClifton\cate-2.0.0.dev20 in the stack trace. I don't know how the release process works in detail, but it is possible that the fix is not actually in the latest release yet.

@forman Correct me if I'm wrong.