Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Revision of Cate's Data Management #153

Closed
8 tasks done
forman opened this issue Mar 14, 2017 · 3 comments
Closed
8 tasks done

Revision of Cate's Data Management #153

forman opened this issue Mar 14, 2017 · 3 comments
Assignees
Milestone

Comments

@forman
Copy link
Member

forman commented Mar 14, 2017

We need a slight revision of Cate's Data Management, especially a revised cate.core.ds.DataSource interface. Details are here:
https://github.com/CCI-Tools/cate-core/wiki/Revision-of-Cate's-Data-Management

  • Introduce new DataSource.open_dataset API, add more constraint args, deprecate protocol arg, use new types, generalise towards any return type (forman)
  • Introduce new DataSource.make_local API (forman)
  • Rename LocalFilePatternDataStore and LocalFilePatternDataSource to LocalDataStore and LocalDataSource (forman)
  • Reimplement all open_dataset overrides wrt deprecate protocol arg and support new constraint params (kbernat)
  • Implement make_local overrides, reuse old syncmethod and Chris' new code (kbernat)
  • Use new open_dataset constraint args in CLI + GUI (forman)
  • Use new make_local in favor of sync where appropriate. We might keep private _sync method as an implementation detail (forman, kbernat)
  • Increase cate.core.ds and cate.ds code coverage by unit-tests (kbernat)
@JanisGailis
Copy link
Member

Looks good.

One note. When using a 'PolygonLike' from new types, one has to keep in mind that what one can get might be whatever valid polygon. Then, it depends on what OpenDAP can do, but probably a bounding box has to be extracted and used for selection.

One question though is what to do with values that are in the bounding box, but not part of the polygon. In the 'subset_spatial' operation I plan to mask these with NaN. I'm not sure what would be the best approach upon opening - mask with NaN, create a 'mask' variable, or just select the bounding box.

@forman
Copy link
Member Author

forman commented Mar 15, 2017

..., but probably a bounding box has to be extracted and used for selection.

Yepp!

In the 'subset_spatial' operation I plan to mask these with NaN.

Please make this an optional behavior. e.g. use new kwarg mask_polygon_exterior: bool = False.

forman added a commit that referenced this issue Mar 15, 2017
@kbernat
Copy link
Collaborator

kbernat commented Mar 15, 2017

Hi Norman,
in make_local method, do we need both local_name and local_id? Role of local name is obvious but how should we use second parameter, is it kind of suffix for general (ecv) name?

  1. Replace sync by make_local

For this, the DataSource method sync shall be replaced by make_local and shall receive a new parameter local_id. ...

def make_local(self,
               local_name: str,
               local_id: str = None,
               time_range: TimeRangeLike.TYPE = None,
               ...

forman added a commit that referenced this issue Mar 16, 2017
kbernat pushed a commit that referenced this issue Mar 16, 2017
kbernat pushed a commit that referenced this issue Mar 16, 2017
kbernat pushed a commit that referenced this issue Mar 17, 2017
kbernat pushed a commit that referenced this issue Mar 20, 2017
kbernat pushed a commit that referenced this issue Mar 20, 2017
kbernat pushed a commit that referenced this issue Mar 21, 2017
kbernat pushed a commit that referenced this issue Mar 21, 2017
kbernat pushed a commit that referenced this issue Mar 21, 2017
kbernat pushed a commit that referenced this issue Apr 6, 2017
kbernat pushed a commit that referenced this issue Apr 6, 2017
kbernat pushed a commit that referenced this issue Apr 6, 2017
kbernat pushed a commit that referenced this issue Apr 7, 2017
kbernat pushed a commit that referenced this issue Apr 7, 2017
… type instead of GeometryLike (too general for region selection), fixed minor issues in region calculation
kbernat pushed a commit that referenced this issue Apr 7, 2017
… instead of Tuple[datetime,datetime] (part 2)
kbernat pushed a commit that referenced this issue Apr 7, 2017
…RangeLike instead of Tuple[datetime,datetime] (ESA ODP DataSource)
kbernat pushed a commit that referenced this issue Apr 7, 2017
kbernat pushed a commit that referenced this issue Apr 7, 2017
kbernat added a commit that referenced this issue Apr 12, 2017
#153 AppVeyor build fix, mocked ODP datastore query which tries to access esfg service
kbernat pushed a commit that referenced this issue Apr 12, 2017
kbernat pushed a commit that referenced this issue Apr 12, 2017
…ed verification if all needed attributes exists
kbernat pushed a commit that referenced this issue Apr 12, 2017
…ed verification if all needed attributes exists
kbernat added a commit that referenced this issue Apr 12, 2017
#153 added make local and delete local to CLI
kbernat pushed a commit that referenced this issue Apr 13, 2017
kbernat pushed a commit that referenced this issue Apr 13, 2017
kbernat pushed a commit that referenced this issue Apr 13, 2017
kbernat pushed a commit that referenced this issue Apr 13, 2017
kbernat pushed a commit that referenced this issue Apr 13, 2017
@kbernat kbernat closed this as completed Apr 20, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants