Co-registering errors

[esacci]

dev10

Co-registering not working in at least several cases ; did not note these down nor try out combinations as so many permutations. In these cases the errors were 'caught' (no crashes) and messages provided, though the messages were not particularly explanatory. Nevertheless, I found these errors to have occurred very frequently ; I couldn't get co-registering to work in any of the cases I tried.

Co-registering is an extremely important feature as it is one of the reasons the toolbox exists - to mash together the CCI datasets.

[JanisGailis]

Nevertheless, only reproducible errors can be fixed. Is it possible to construct a minimal reproducible example?

[forman]

Just fixed a bug with the corregistration progress monitor introduced by an update of the latest dask-core library (a1e31c4)

[HelenClifton]

@esacci , @forman , @JanisGailis The coregistration error was seen during a demonstration using cate-2.0.0-dev.15

[JanisGailis]

@forman Can you copy the stack trace here along with which datasets you were using if you remember?

[JanisGailis]

Job: Executing operation "coregister"
Message: Failed at nopython (nopython frontend)
Failed at nopython (nopython frontend)
cannot unify bool and array(bool, 1d, C) for '$phi354.1', defined at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (375)
File "..\..\ideaprojects\ccitools\cate\cate\ops\resampling.py", line 375
[1] During: typing of assignment at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (375)
[2] During: resolving callee type: type(CPUDispatcher(<function _downsample_2d at 0x0000024A67CF59D8>))
[3] During: typing of call at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (220)
Code: -32003
Method: set_workspace_resource
Exception: numba.errors.TypingError

Traceback (most recent call last):
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result
    result = future.result()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\util\web\jsonrpchandler.py", line 306, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\webapi\websocket.py", line 291, in set_workspace_resource
    monitor=monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\wsmanag.py", line 317, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\workspace.py", line 612, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\workflow.py", line 632, in invoke_steps
    step.invoke(context=context, monitor=monitor.child(work=1))
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\workflow.py", line 318, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\workflow.py", line 980, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\core\op.py", line 216, in __call__
    return_value = self._wrapped_op(**input_values)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 139, in coregister
    return _resample_dataset(ds_master, ds_slave, methods_us[method_us], methods_ds[method_ds], monitor)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 311, in _resample_dataset
    retset = ds_slave.apply(_resample_array, keep_attrs=True, **kwargs)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\dataset.py", line 2772, in apply
    for k, v in iteritems(self.data_vars))
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\dataset.py", line 2772, in <genexpr>
    for k, v in iteritems(self.data_vars))
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 258, in _resample_array
    temp_array = _nested_groupby_apply(array, groupby_list, _resample_slice, kwargs)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 393, in _nested_groupby_apply
    kwargs=kwargs)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 516, in apply
    return self._combine(applied, shortcut=shortcut)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 520, in _combine
    applied_example, applied = peek_at(applied)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\utils.py", line 133, in peek_at
    peek = next(gen)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 515, in <genexpr>
    for arr in grouped)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 388, in _nested_groupby_apply
    return array.groupby(groupby[0]).apply(apply_fn, **kwargs)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 516, in apply
    return self._combine(applied, shortcut=shortcut)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 520, in _combine
    applied_example, applied = peek_at(applied)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\utils.py", line 133, in peek_at
    peek = next(gen)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\xarray\core\groupby.py", line 515, in <genexpr>
    for arr in grouped)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\coregistration.py", line 213, in _resample_slice
    us_method)
  File "c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py", line 100, in resample_2d
    return _mask_or_not(_resample_2d(src, mask, use_mask, ds_method, us_method, fill_value, mode_rank, out),
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 330, in _compile_for_args
    raise e
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 307, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 579, in compile
    cres = self._compiler.compile(args, return_type)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 80, in compile
    flags=flags, locals=self.locals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 763, in compile_extra
    return pipeline.compile_extra(func)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 360, in compile_extra
    return self._compile_bytecode()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 722, in _compile_bytecode
    return self._compile_core()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 709, in _compile_core
    res = pm.run(self.status)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 246, in run
    raise patched_exception
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 238, in run
    stage()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 452, in stage_nopython_frontend
    self.locals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 865, in type_inference_stage
    infer.propagate()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 844, in propagate
    raise errors[0]
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 137, in propagate
    constraint(typeinfer)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 415, in __call__
    self.resolve(typeinfer, typevars, fnty)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 441, in resolve
    sig = typeinfer.resolve_call(fnty, pos_args, kw_args, literals=literals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 1113, in resolve_call
    literals=literals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typing\context.py", line 202, in resolve_function_type
    return func.get_call_type_with_literals(self, args, kws, literals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\types\abstract.py", line 266, in get_call_type_with_literals
    return self.get_call_type(context, args, kws)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\types\functions.py", line 175, in get_call_type
    template, pysig, args, kws = self.dispatcher.get_call_template(args, kws)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 269, in get_call_template
    self.compile(tuple(args))
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 579, in compile
    cres = self._compiler.compile(args, return_type)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\dispatcher.py", line 80, in compile
    flags=flags, locals=self.locals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 763, in compile_extra
    return pipeline.compile_extra(func)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 360, in compile_extra
    return self._compile_bytecode()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 722, in _compile_bytecode
    return self._compile_core()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 709, in _compile_core
    res = pm.run(self.status)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 246, in run
    raise patched_exception
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 238, in run
    stage()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 452, in stage_nopython_frontend
    self.locals)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\compiler.py", line 865, in type_inference_stage
    infer.propagate()
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 844, in propagate
    raise errors[0]
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 137, in propagate
    constraint(typeinfer)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 162, in __call__
    typeinfer.copy_type(self.src, self.dst, loc=self.loc)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 862, in copy_type
    unified = self.typevars[dest_var].union(self.typevars[src_var], loc=loc)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 89, in union
    self.add_type(other.type, loc=loc)
  File "C:\Users\Norman\Miniconda3\envs\cate-env\lib\site-packages\numba\typeinfer.py", line 60, in add_type
    loc=loc)
numba.errors.TypingError: Failed at nopython (nopython frontend)
Failed at nopython (nopython frontend)
cannot unify bool and array(bool, 1d, C) for '$phi354.1', defined at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (375)
File "..\..\ideaprojects\ccitools\cate\cate\ops\resampling.py", line 375
[1] During: typing of assignment at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (375)
[2] During: resolving callee type: type(CPUDispatcher(<function _downsample_2d at 0x0000024A67CF59D8>))
[3] During: typing of call at c:\users\norman\ideaprojects\ccitools\cate\cate\ops\resampling.py (220)

[JanisGailis]

@forman Do you remember which dataset it was?

[forman]

Ozone and Cloud monthly, I used 2007, 12 time steps

[forman]

My "mistake" during demo was, to not filter out all but a single variable.

[JanisGailis]

I managed to reproduce the stack trace above. This should now be fixed. Please re-open if it persists in later releases.