"Write to read-only" errors when downloading datasets

[HelenClifton]

Expected behavior

Attempted to download different datasets: soil moisture, SST, cloud

Actual behavior

In each case, the download starts but shortly afterwards there is an error of the following type :

Log file for soil moisture example here:
sm download trace.txt

Steps to reproduce the problem

1. Select esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.03-2.r1
2. Select Download Dataset to Local option with:
   Time Constraint: 2007-01-01 to 2007-12-31
   Region Constraint:
   o Lon. from: 72
   o Lon. to: 85
   o Lat. from: 8
   o Lat. to: 17
   Variables Constraint: sm, sm_uncertainty

Same type of error happens with the following datasets:

esacci.SST.day.L4.SSTdepth.multi-sensor.multi-platform.OSTIA.1-1.r1
Time Constraint: 2006-01-01 to 2007-12-31
Region Constraint:
o Lon. from: -175
o Lon. to: -115
o Lat. from: -10
o Lat. to: 10
Variables Constraint: analysed_sst, analysis_error

and

esacci.CLOUD.mon.L3C.CLD_PRODUCTS.multi-sensor.multi-platform.ATSR2-AATSR.2-0.r1
Time Constraint: 2006-01-01 to 2009-12-31
Region Constraint:
o Lon. from: 7
o Lon. to: 10
o Lat. from: 48
o Lat. to: 52
Variables Constraint: cfc

Note

@forman , @JanisGailis , @papesci
This problem is blocking happy path for use case 6 and use case 2
Whereas with the datasets in the use case 22 happy path, it happens at some times but not others.
It could be related to update which was made to fix #631

Specifications

cate-2.0.0-dev.11
Windows 7 Professional

[JanisGailis]

Likely a duplicate of #664

[HelenClifton]

@forman , @JanisGailis , @papesci Same happens with dataset below. This is blocking happy path for new use case 4.

esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1
Time Constraint: 1997-01-01 to 2008-12-31
Region Constraint:
o Lon. from: -160
o Lon. to: -69
o Lat. from: 23
o Lat. to: 50
Variables Constraint: O3_du_tot

[papesci]

Hi I have just tried to execute all the tests mentioned in this issue on Linux with version dev13 successfully.

[HelenClifton]

Hi @papesci, @forman I've repeated the tests on my Windows machine, which now has dev.13 installed, and the same problems are happening.
However, I saw that my CLI environment is still reported as dev.11:

[papesci]

@HelenClifton cate core should be dev12. Can you open a shell and type cate --version just to confirm which core is in use ?

[HelenClifton]

@papesci In the CLI, it says 2.0.0.dev.12

[papesci]

I have just tried on Linux with your configuration running the test without problem so i need to investigate what is wrong on Window but I cannot access ODP from window yet.

cate ds: error: Data store "esa_cci_odp": Cannot download CCI Open Data Portal ECV index:
Cannot fetch information from CCI Open Data Portal server.

[HelenClifton]

@papesci I tried downloading the above datasets using a mac with dev.13 installed and there were no problems. It just seems to be Windows.

[hans-permana]

I can reproduce this issue on my Windows 10 Home machine. This did not happen when the variable and spatial filter were not set.

[HelenClifton]

@papesci Confirmed it still has same behaviour with dev.15 using Windows
however downloading soil moisture dataset mentioned above using a mac machine results in a different error, see #687

[papesci]

A different behavior is confirmed when using different OS.
After several tests, on all the supported OS included Window, the operation open_dataset of XArray react differently when a large number of files is open. The error don't appear when adding the optional parameter autoclose=True to xarray open_dataset function. I think this is more a workaround then a fix which should be implemented inside xarray library i guess.

However a branch has been created 667

[JanisGailis]

The autoclose option was implemented in xarray after us and other people reporting memory issues when opening datasets consisting of many files. So, that already is the fix. In either case we should use autoclose, it was pretty much made for our use case.