Skip to content
This repository was archived by the owner on Aug 29, 2023. It is now read-only.

opening soil moisture dataset - new "integer division or modulo by zero" error #444

Closed
HelenClifton opened this issue Sep 29, 2017 · 19 comments
Closed

opening soil moisture dataset - new "integer division or modulo by zero" error #444

HelenClifton opened this issue Sep 29, 2017 · 19 comments
Assignees
@kbernat
Labels
bug ds regression
Milestone
1.0

Comments

@HelenClifton
Copy link

Expected behavior

Running either use case 6 happy path and use case 2 happy path
Opening dataset
esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.03-2.r1

Actual behavior

Dataset downloads then there is an error when it is opened
use case 6
sm
use case 6 Traceback:
sm traceback.txt

use case 2
sm 2
Traceback:
sm traceback 2.txt

Steps to reproduce the problem

use case 6
steps to reproduce uc6.docx

use case 2
steps to reproduce - uc 2.docx

Notes

Thought to be different to #326 because here only sm and sm_uncertainty varaiables are selected.
Champion users should decide whether this dataset should be in the whitelist

Specifications

cate-0.9.0-dev.7
Windows 7 Professional
@forman forman self-assigned this Oct 1, 2017
@forman forman added this to the 1.0 milestone Oct 1, 2017
@forman
Copy link
Member

forman commented Oct 2, 2017 •
edited
Loading

@HelenClifton, @kbernat if I use the same constraint parameters in 1.0.0.dev1, the error does not occur. Instead, nothing will be displayed. It seems that when the dataset is opened via OPeNDAP with region (72, 8, 85,17), the lat dimension disappears:

image

Latitude should not disappear because resolution is 1440/360 = 4 grid cells per degree, therefore new lat dim size should be (17 - 8) * 4 = 36 cells. Lon size is correct with (85 - 72) * 4 = 52 cells.

Chris, I guess the bug is somewhere downstream of open_dataset if it switches to OPeNDAP and spatial subset computation occurs. Please consider that

  • latitude axes can point up or down in NetCDF-CF, OPeNDAP datasets meaning that latitude values may be monotonically increasing or decreasing with the dimension index;
  • in a longitude range given by user, the first value may be larger than the second which means anti-meridian crossing, see Regional subset crossing anti-meridian #318.

@forman forman assigned kbernat and unassigned forman Oct 2, 2017
@forman forman removed the in_progress label Oct 2, 2017
@JanisGailis
Copy link
Member

JanisGailis commented Oct 3, 2017 •
edited
Loading

I can reproduce the bug on Cate 0.9.dev7 GUI:

Cate Desktop, version 0.9.0-dev.7

set_workspace_resource() call raised exception: "integer division or modulo by zero"

An error (code 20) occurred in Cate Core:

Traceback (most recent call last):
  File "C:\Users\janis\cate\lib\site-packages\cate\util\web\jsonrpchandler.py", line 190, in send_service_method_result
    result = future.result()
  File "C:\Users\janis\cate\lib\concurrent\futures\_base.py", line 398, in result
    return self.__get_result()
  File "C:\Users\janis\cate\lib\concurrent\futures\_base.py", line 357, in __get_result
    raise self._exception
  File "C:\Users\janis\cate\lib\concurrent\futures\thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\janis\cate\lib\site-packages\cate\util\web\jsonrpchandler.py", line 267, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "C:\Users\janis\cate\lib\site-packages\cate\webapi\websocket.py", line 277, in set_workspace_resource
    return workspace.to_json_dict()
  File "C:\Users\janis\cate\lib\site-packages\cate\core\workspace.py", line 269, in to_json_dict
    ('resources', self._resources_to_json_list())
  File "C:\Users\janis\cate\lib\site-packages\cate\core\workspace.py", line 281, in _resources_to_json_list
    resource_descriptor = self._get_resource_descriptor(res_id, res_update_count, res_name, resource)
  File "C:\Users\janis\cate\lib\site-packages\cate\core\workspace.py", line 303, in _get_resource_descriptor
    variable_descriptors.append(self._get_xarray_variable_descriptor(variable))
  File "C:\Users\janis\cate\lib\site-packages\cate\core\workspace.py", line 373, in _get_xarray_variable_descriptor
    image_layout = self._get_variable_image_layout(variable)
  File "C:\Users\janis\cate\lib\site-packages\cate\core\workspace.py", line 437, in _get_variable_image_layout
    max_size, tile_size, num_level_zero_tiles, num_levels = ImagePyramid.compute_layout(array=variable)
  File "C:\Users\janis\cate\lib\site-packages\cate\util\im\image.py", line 816, in compute_layout
    cardinal_div_round(max_height, tile_height * 2 ** (num_levels - 1))
  File "C:\Users\janis\cate\lib\site-packages\cate\util\im\utils.py", line 66, in cardinal_div_round
    return int(num + denom - 1) // int(denom)
ZeroDivisionError: integer division or modulo by zero

The CLI interface of the Cate 0.9.dev7 installation works as expected. The dataset can be downloaded, opened and looks fine when printed:
image

kbernat pushed a commit that referenced this issue Oct 3, 2017
#318, #444 fix for opendap data download with regional subset crossin…
a2be6c5 
…g anti-meridian
@kbernat kbernat closed this as completed Oct 3, 2017
@mzuehlke mzuehlke reopened this Oct 5, 2017
@mzuehlke
Copy link
Collaborator

mzuehlke commented Oct 5, 2017

set_workspace_resource() call raised exception: "The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load()."

Traceback (most recent call last):
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 60, in __getitem__
    data = getitem(self.get_array(), key)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 66, in robust_getitem
    return array[key]
  File "netCDF4/_netCDF4.pyx", line 3823, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:40266)
  File "netCDF4/_netCDF4.pyx", line 4647, in netCDF4._netCDF4.Variable._get (netCDF4/_netCDF4.c:52046)
IndexError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/util/web/jsonrpchandler.py", line 192, in send_service_method_result
    result = future.result()
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/util/web/jsonrpchandler.py", line 269, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/webapi/websocket.py", line 282, in set_workspace_resource
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/wsmanag.py", line 321, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workspace.py", line 646, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 624, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 315, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 977, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/op.py", line 211, in __call__
    return_value = self._wrapped_op(**input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ops/io.py", line 81, in open_dataset
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/ds.py", line 523, in open_dataset
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 911, in make_local
    raise e
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 904, in make_local
    self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 767, in _make_local
    local_netcdf.store_dataset(remote_dataset)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 209, in store_dataset
    self.store(dataset, dataset.attrs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 246, in store
    *args, **kwargs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 215, in store
    unlimited_dims=unlimited_dims)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 288, in set_variables
    super(NetCDF4DataStore, self).set_variables(*args, **kwargs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 227, in set_variables
    name, v, check, unlimited_dims=unlimited_dims)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 332, in prepare_variable
    return nc4_var, variable.data
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/dataarray.py", line 394, in data
    return self.variable.data
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 288, in data
    return self.values
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 329, in values
    return _as_array_or_item(self._data)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 205, in _as_array_or_item
    data = np.asarray(data)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/indexing.py", line 375, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 71, in __getitem__
    raise IndexError(msg)
IndexError: The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load().

Traceback (most recent call last):
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 60, in __getitem__
    data = getitem(self.get_array(), key)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 66, in robust_getitem
    return array[key]
  File "netCDF4/_netCDF4.pyx", line 3823, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4/_netCDF4.c:40266)
  File "netCDF4/_netCDF4.pyx", line 4647, in netCDF4._netCDF4.Variable._get (netCDF4/_netCDF4.c:52046)
IndexError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/util/web/jsonrpchandler.py", line 192, in send_service_method_result
    result = future.result()
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/util/web/jsonrpchandler.py", line 269, in call_service_method
    result = method(*method_params, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/webapi/websocket.py", line 282, in set_workspace_resource
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/wsmanag.py", line 321, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workspace.py", line 646, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 624, in invoke_steps
    steps[0].invoke(context=context, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 315, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/workflow.py", line 977, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/op.py", line 211, in __call__
    return_value = self._wrapped_op(**input_values)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ops/io.py", line 81, in open_dataset
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/core/ds.py", line 523, in open_dataset
    monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 911, in make_local
    raise e
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 904, in make_local
    self._make_local(local_ds, time_range, region, var_names, monitor=monitor)
  File "/home/marcoz/Projects/BC/PyBc/cci-tools/cate-core/cate/ds/esa_cci_odp.py", line 767, in _make_local
    local_netcdf.store_dataset(remote_dataset)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 209, in store_dataset
    self.store(dataset, dataset.attrs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 246, in store
    *args, **kwargs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 215, in store
    unlimited_dims=unlimited_dims)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 288, in set_variables
    super(NetCDF4DataStore, self).set_variables(*args, **kwargs)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/common.py", line 227, in set_variables
    name, v, check, unlimited_dims=unlimited_dims)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 332, in prepare_variable
    return nc4_var, variable.data
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/dataarray.py", line 394, in data
    return self.variable.data
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 288, in data
    return self.values
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 329, in values
    return _as_array_or_item(self._data)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/variable.py", line 205, in _as_array_or_item
    data = np.asarray(data)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/core/indexing.py", line 375, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/home/marcoz/Projects/Anaconda/miniconda3/envs/cate/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 71, in __getitem__
    raise IndexError(msg)
IndexError: The indexing operation you are attempting to perform is not valid on netCDF4.Variable object. Try loading your data into memory first by calling .load().

@mzuehlke mzuehlke added blocker and removed blocker labels Oct 5, 2017
@kbernat
Copy link
Collaborator

kbernat commented Oct 5, 2017

no some data it works, on some it does not :/ - working on it.

@kbernat
Copy link
Collaborator

kbernat commented Oct 5, 2017 •
edited
Loading

I compared SST (which works perfect) and SOILMOISTURE, the one obvious difference is that
SOILMOISTURE dataset defines 'lat' as a degrees_north but values starts from 90.. to -90
SST has the same units degrees_north but values goes from -90 to 90

netcdf ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-20070101000000-fv03.2 {
dimensions:
	lat = 720 ;
	lon = 1440 ;
	time = 1 ;
variables:
	float **lat(lat)** ;
		lat:_FillValue = NaNf ;
		lat:_CoordinateAxisType = "Lat" ;
		lat:standard_name = "latitude" ;
		lat:valid_range = -90., 90. ;
		lat:units = "**degrees_north**" ;
		lat:_ChunkSizes = 720 ;
	float lon(lon) ;
		lon:_FillValue = NaNf ;
		lon:units = "degrees_east" ;
		lon:_CoordinateAxisType = "Lon" ;
		lon:standard_name = "longitude" ;
		lon:valid_range = -180., 180. ;
		lon:_ChunkSizes = 1440 ;
	double time(time) ;
		time:_FillValue = NaN ;
		time:_CoordinateAxisType = "Time" ;
		time:standard_name = "time" ;
		time:calendar = "standard" ;
		time:units = "days since 1970-01-01 00:00:00 UTC" ;
		time:_ChunkSizes = 1 ;


...

data:

 **lat** = **89.875**, 89.625, 89.375, 89.125, 88.875, 88.625, 88.375, 88.125, 
    87.875, 87.625, 87.375, 87.125, 86.875, 86.625, 86.375, 86.125, 85.875, 

netcdf \20060101120000-ESACCI-L4_GHRSST-SSTdepth-OSTIA-GLOB_LT-v02.0-fv01.1 {
dimensions:
	time = 1 ;
	lat = 3600 ;
	lon = 7200 ;
variables:
	int time(time) ;
		time:units = "seconds since 1981-01-01 00:00:00" ;
		time:standard_name = "time" ;
		time:axis = "T" ;
		time:calendar = "gregorian" ;
		time:comment = "" ;
		time:bounds = "time_bnds" ;
		time:long_name = "reference time of sst file" ;
		time:_ChunkSizes = 1 ;
	float lat(lat) ;
		lat:_FillValue = NaNf ;
		lat:standard_name = "latitude" ;
		lat:units = "degrees_north" ;
		lat:valid_min = -90.f ;
		lat:valid_max = 90.f ;
		lat:axis = "Y" ;
		lat:reference_datum = "geographical coordinates, WGS84 projection" ;
		lat:bounds = "lat_bnds" ;
		lat:comment = "" ;
		lat:long_name = "Latitude" ;
		lat:_ChunkSizes = 3600 ;
	float lon(lon) ;
		lon:_FillValue = NaNf ;
		lon:standard_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:valid_min = -180.f ;
		lon:valid_max = 180.f ;
		lon:axis = "X" ;
		lon:reference_datum = "geographical coordinates, WGS84 projection" ;
		lon:bounds = "lon_bnds" ;
		lon:comment = "" ;
		lon:long_name = "Longitude" ;
		lon:_ChunkSizes = 7200 ;
...

data:

 **lat** = **-89.975**, -89.925, -89.875, -89.825, -89.775, -89.725, -89.675, 
    -89.625, -89.575, -89.525, -89.475, -89.425, -89.375, -89.325, -89.275,

@kbernat
Copy link
Collaborator

kbernat commented Oct 5, 2017

@forman @mzuehlke @JanisGailis imho the problem lies somewhere in subset operation or any other harmonization/normalization part.
After I reused subsetting operation impl it crashes in different way.. but..
Please, try to download whole dataset in CLI (no region constrains) and to perform subset operation..

$ cate res set subs subset_spatial ds=@soil_2007_cli_test region=1,2,3,4
Executing 2 workflow step(s): [##############################] 100%                                                                                                                     Traceback (most recent call last):
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/util/web/jsonrpchandler.py", line 192, in send_service_method_result
    result = future.result()
  File "/home/dev/cate/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/home/dev/cate/envs/cate/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/home/dev/cate/envs/cate/lib/python3.6/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/util/web/jsonrpchandler.py", line 271, in call_service_method
    result = method(**method_params, monitor=monitor)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/webapi/websocket.py", line 282, in set_workspace_resource
    monitor=monitor)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/wsmanag.py", line 321, in set_workspace_resource
    workspace.execute_workflow(res_name=res_name, monitor=monitor)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/workspace.py", line 646, in execute_workflow
    self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/workflow.py", line 629, in invoke_steps
    step.invoke(context=context, monitor=monitor.child(work=1))
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/workflow.py", line 315, in invoke
    self._invoke_impl(_new_context(context, step=self), monitor=monitor)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/workflow.py", line 977, in _invoke_impl
    return_value = self._op(monitor=monitor, **input_values)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/core/op.py", line 211, in __call__
    return_value = self._wrapped_op(**input_values)
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/ops/subset.py", line 58, in subset_spatial
    return adjust_spatial_attrs(subset_spatial_impl(ds, region, mask))
  File "/home/dev/Projects/CCI-Tools/cate-core/cate/util/opimpl.py", line 442, in subset_spatial_impl
    return ds.sel(**indexers)
  File "/home/dev/cate/envs/cate/lib/python3.6/site-packages/xarray/core/dataset.py", line 1208, in sel
    result = self.isel(drop=drop, **pos_indexers)
  File "/home/dev/cate/envs/cate/lib/python3.6/site-packages/xarray/core/dataset.py", line 1147, in isel
    return self._replace_vars_and_dims(variables, coord_names=coord_names)
  File "/home/dev/cate/envs/cate/lib/python3.6/site-packages/xarray/core/dataset.py", line 561, in _replace_vars_and_dims
    dims = calculate_dimensions(variables)
  File "/home/dev/cate/envs/cate/lib/python3.6/site-packages/xarray/core/dataset.py", line 110, in calculate_dimensions
    (dim, size, k, dims[dim], last_used[dim]))
ValueError: conflicting sizes for dimension 'lat': length -8 on 'sm' and length 0 on 'lat'

Executing 2 workflow step(s): done                                                                                                                                                      
cate res: error: set_workspace_resource() call raised exception: "conflicting sizes for dimension 'lat': length -8 on 'sm' and length 0 on 'lat'"

@forman
Copy link
Member

forman commented Oct 5, 2017

@JanisGailis @kbernat In my first comment above I wrote

latitude axes can point up or down in NetCDF-CF, OPeNDAP datasets meaning that latitude values may be monotonically increasing or decreasing with the dimension index;

Any subset operation should be aware of that. It is also fully in line with CF.

@mzuehlke
Copy link
Collaborator

mzuehlke commented Oct 5, 2017

Looking at the ds.sel invocations in opimpl.py this seems to be handled nowhere.

@JanisGailis
Copy link
Member

I'll look into it.

I remember that there was something with us using a polygon that made things tricky, but I don't remember what exactly it was.

@forman forman assigned JanisGailis and unassigned kbernat Oct 5, 2017
@JanisGailis
Copy link
Member

@forman Can you point me in the right direction regarding where it is in the CF conventions?

I can not find anything regarding directionality of the latitude vector, it just talks about endorsing the usage of 'degrees_north' as the unit, and it doesn't say anything regarding if the counting should start at the North or at the South pole. Maybe this is what you mean, that it is not specified => legal.

@JanisGailis
Copy link
Member

Heh. I had made a test for inverted dimensions, but the test was wrong (the same as nominal). When I fixed the test it now fails in spatial attribute adjustment thing.

On it.

@kbernat
Copy link
Collaborator

kbernat commented Oct 6, 2017

Both mentioned by me data sources uses units degrees_north, It means that one is wrong.

This was referenced Oct 6, 2017
Closed
Closed
@mzuehlke
Copy link
Collaborator

mzuehlke commented Oct 6, 2017

@kbernat as mentioned be @forman earlier the direction of the axis can be increasing or decreasing.
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#latitude-coordinate

@kbernat
Copy link
Collaborator

kbernat commented Oct 6, 2017 •
edited
Loading

@mzuehlke thanks! I really expected to see.. hmm.. degree_south :) in case of descending order. Now it's clear for me!

@JanisGailis
Copy link
Member

Can someone verify if it works as expected with the GUI now, after the spatial_subset fix?

@JanisGailis JanisGailis assigned mzuehlke and kbernat and unassigned JanisGailis Oct 6, 2017
@mzuehlke
Copy link
Collaborator

mzuehlke commented Oct 6, 2017

@kbernat was in the process of using the spatial_subset in the ds code when encountering this problem.

@mzuehlke mzuehlke removed their assignment Oct 6, 2017
@mzuehlke
Copy link
Collaborator

mzuehlke commented Oct 6, 2017

@kbernat any progress on this issue ?

@kbernat
Copy link
Collaborator

kbernat commented Oct 6, 2017

@mzuehlke need bit more time to verify it. At the moment my Cate Desktop from git does not recognize cate webservice anymore..

kbernat pushed a commit that referenced this issue Oct 9, 2017
#441, #443, #444 - missing coordinates for copied ODP datasets
b3bbacd
@kbernat kbernat closed this as completed Oct 9, 2017
@HelenClifton
Copy link
Author

Confirmed it is fixed in cate 1.0.0-dev.3

@JanisGailis JanisGailis mentioned this issue Mar 7, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kbernat kbernat

Labels
bug ds regression
Projects
None yet
Milestone
1.0
Development

No branches or pull requests
5 participants
@mzuehlke @forman @kbernat @JanisGailis @HelenClifton