Reworksavecube

[meggart]

This is a rewrite of the savecube functionality, which also has some effect on mapCube. The main change is some new keyword arguments to saveCube:

• overwrite to enable automatic overwriting of existing cubes
• chunksize to determine the chunksize of the saved cube
• compressor let the user specify compression

Note that this function should now work on any AbstractCubeData and therefore allows nice things like downloading a cube subset from S3 through:

c = S3Cube()
cgermany = c[region="Germany", time=2001:2010, var=myvars]
saveCube(cgermany,"Germanycube")

Under the hood this is possible because the OutDims constructor take the chunksize, compressor and path as arguments as well, so that basically saveCube can fall back to a mapCube over the identity function, simply specifying the new output location and passing them to OutDims.

Along the way this should fix #128 and #125

[meggart]

@felixcremer can you try this branch and report problems you might still have?

[codecov-io]

Codecov Report

Merging #129 into master will increase coverage by 9.79%.
The diff coverage is 84.5%.

@@            Coverage Diff             @@
##           master     #129      +/-   ##
==========================================
+ Coverage   46.56%   56.35%   +9.79%     
==========================================
  Files          24       23       -1     
  Lines        2182     1879     -303     
==========================================
+ Hits         1016     1059      +43     
+ Misses       1166      820     -346

Impacted Files	Coverage Δ
src/Cubes/TransformedCubes.jl	58.33% <100%> (+1.81%)	⬆️
src/ESDL.jl	83.33% <100%> (+3.33%)	⬆️
src/DAT/registration.jl	71.05% <100%> (+1.6%)	⬆️
src/Cubes/Cubes.jl	39.6% <40%> (-2.4%)	⬇️
src/DAT/DAT.jl	85.49% <72.72%> (-0.31%)	⬇️
src/Cubes/ZarrCubes.jl	66.06% <83.33%> (-1.26%)	⬇️
src/Proc/CubeIO.jl	94.36% <94.11%> (+13.33%)	⬆️
src/Cubes/Axes.jl	62.04% <0%> (+3.95%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 374e23f...87a32d0. Read the comment docs.

[felixcremer]

When I am trying to save a cube with this, I get a

ERROR: type OutputCube has no field allaxes

this is due to a typo in ESDL/src/DAT/DAT.jl:509
the oc.allaxes should be oc.allAxes but when I change that, I run into the following error:

saveCube(kiuic_emd, "kiuic/s1cube_kiuic_new_emd2")
ERROR: type OutputCube has no field compressor
Stacktrace:
 [1] getproperty(::Any, ::Symbol) at ./sysimg.jl:18
 [2] (::getfield(ESDL.DAT, Symbol("##83#87")){ESDL.DAT.DATConfig{1,1},Array{NamedTuple{(:iloopax, :cs, :iscompressed, :innerleap),Tuple{Int64,Int64,Bool,Int64}},1}})(::RangeAxis{Dates.DateTime,:Time,Array{Dates.DateTime,1}}, ::Int64) at /home/qe89hep/.julia/dev/ESDL/src/DAT/DAT.jl:512
 [3] foreach(::Function, ::Array{CubeAxis,1}, ::UnitRange{Int64}) at ./abstractarray.jl:1867
 [4] getCacheSizes(::ESDL.DAT.DATConfig{1,1}) at /home/qe89hep/.julia/dev/ESDL/src/DAT/DAT.jl:498
 [5] #mapCube#49(::Nothing, ::InDims, ::OutDims, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(mapCube), ::typeof(copyto!), ::Tuple{ESDL.Cubes.ConcatCube{Union{Missing, Float32},4}}) at /home/qe89hep/.julia/dev/ESDL/src/DAT/DAT.jl:265
 [6] #mapCube at ./none:0 [inlined]
 [7] #mapCube#34 at /home/qe89hep/.julia/dev/ESDL/src/DAT/DAT.jl:181 [inlined]
 [8] #mapCube at ./none:0 [inlined]
 [9] #saveCube#30(::Bool, ::NTuple{4,Int64}, ::Zarr.NoCompressor, ::Function, ::ESDL.Cubes.ConcatCube{Union{Missing, Float32},4}, ::String) at /home/qe89hep/.julia/dev/ESDL/src/Proc/CubeIO.jl:127
 [10] saveCube(::ESDL.Cubes.ConcatCube{Union{Missing, Float32},4}, ::String) at /home/qe89hep/.julia/dev/ESDL/src/Proc/CubeIO.jl:110
 [11] top-level scope at none:0

[meggart]

I think I am going to merge this one right now, so that we cann all work against master again.