Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Ingest SST and Soil Moisture Data from ODP #71

Closed
forman opened this issue Oct 13, 2016 · 7 comments
Closed

Ingest SST and Soil Moisture Data from ODP #71

forman opened this issue Oct 13, 2016 · 7 comments
Assignees
Milestone

Comments

@forman
Copy link
Member

forman commented Oct 13, 2016

Make sure SST and Soil Moisture Data from ODP is accessible and can be opened correctly.

@forman forman added this to the v2.0 milestone Oct 13, 2016
@forman forman added the feature label Oct 13, 2016
@JanisGailis JanisGailis self-assigned this Dec 9, 2016
@JanisGailis
Copy link
Member

JanisGailis commented Dec 9, 2016

from cate.core.ds import DATA_STORE_REGISTRY
from cate.core.monitor import ConsoleMonitor
import cate.ops as ops

monitor = ConsoleMonitor()
data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp')
sm = ops.open_dataset('esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.02-2.r1',
                       '2000-01-01',
                       '2003-12-31', sync=True, monitor=monitor)

Syncing doesn't work for soil moisture. No monitor output (neither using API nor CLI), files are not created in ~/.cate/data_stores/esa_cci_odp/xx. However, whem monitoring network traffic, incoming traffic jumps when trying to sync both on API and CLI, so it does seem to be downloading something.

SST seems to work as expected.

Both datasets are daily, we need monthly, so temporal aggregation (producing monthly mean) will have to be done.

kbernat pushed a commit that referenced this issue Dec 9, 2016
…ith offset and limit) from ESGF search service
kbernat pushed a commit that referenced this issue Dec 9, 2016
…ith offset and limit) from ESGF search service
@kbernat
Copy link
Collaborator

kbernat commented Dec 9, 2016

@JanisGailis, you should be able to open this data source now. There was a problem with fetching multiple responses from ESFG service, it's fixed now.

@kbernat
Copy link
Collaborator

kbernat commented Dec 9, 2016

cate ds sync esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.02-2.r1 2001-01-01 2001-02-01
Sync esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.02-2.r1: progress                              
32 of 32 file(s) synchronized.

kbernat added a commit that referenced this issue Dec 9, 2016
#71, esgf search additional fix, covered case where it's more than two responses
@JanisGailis
Copy link
Member

That was quick. I already fetched it manually, but I'll try if the syncing works as expected then!

@JanisGailis
Copy link
Member

OK, I still get the same behavior. I'll investigate it further on Monday, maybe I missed something.

In the mean time, while opening SST data, I get OSError 'Too many open files', people at xarray are discussing it:
pydata/xarray#463

It's three years of daily data. So 1000+ files

@JanisGailis
Copy link
Member

@kbernat I can confirm that synchronization of the soil moisture dataset now works. One just have to wait a bit before the process really starts and the output gets written to the monitor, I was apparently too impatient on Friday.

However, #102 is now a thing. This is not blocking for UC06 development as I will just use smaller datasets for that purpose.

@JanisGailis
Copy link
Member

Datasets as they are on ODP can be opened. SST has quite high compression resulting in lots of 'in-memory' data, which will be solved by #118

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants