-
Notifications
You must be signed in to change notification settings - Fork 15
Ingest SST and Soil Moisture Data from ODP #71
Comments
from cate.core.ds import DATA_STORE_REGISTRY
from cate.core.monitor import ConsoleMonitor
import cate.ops as ops
monitor = ConsoleMonitor()
data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp')
sm = ops.open_dataset('esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.02-2.r1',
'2000-01-01',
'2003-12-31', sync=True, monitor=monitor) Syncing doesn't work for soil moisture. No monitor output (neither using API nor CLI), files are not created in ~/.cate/data_stores/esa_cci_odp/xx. However, whem monitoring network traffic, incoming traffic jumps when trying to sync both on API and CLI, so it does seem to be downloading something. SST seems to work as expected. Both datasets are daily, we need monthly, so temporal aggregation (producing monthly mean) will have to be done. |
…ith offset and limit) from ESGF search service
…ith offset and limit) from ESGF search service
@JanisGailis, you should be able to open this data source now. There was a problem with fetching multiple responses from ESFG service, it's fixed now. |
|
#71, esgf search additional fix, covered case where it's more than two responses
That was quick. I already fetched it manually, but I'll try if the syncing works as expected then! |
OK, I still get the same behavior. I'll investigate it further on Monday, maybe I missed something. In the mean time, while opening SST data, I get OSError 'Too many open files', people at xarray are discussing it: It's three years of daily data. So 1000+ files |
@kbernat I can confirm that synchronization of the soil moisture dataset now works. One just have to wait a bit before the process really starts and the output gets written to the monitor, I was apparently too impatient on Friday. However, #102 is now a thing. This is not blocking for UC06 development as I will just use smaller datasets for that purpose. |
Datasets as they are on ODP can be opened. SST has quite high compression resulting in lots of 'in-memory' data, which will be solved by #118 |
Make sure SST and Soil Moisture Data from ODP is accessible and can be opened correctly.
The text was updated successfully, but these errors were encountered: