Finalsplit

[meggart]

These are the latest changes to split out the whole machinery to YAXArrays and leave the Geo-specific parts inside ESDL.jl

[meggart]

@felixcremer

[felixcremer]

I tested it, and the computations are working, but it feels slower as before, but I am not sure, whether this is also because I am currently not using the distributed computations. And I have to set up proper tests to compare the timings of the different versions.

I still can't open NetCDF files while the julia process is running but I can copy them and then they can be opened.
But I now can't open my RQA results, but I am not sure, whether this is due to ESDL changes or my QGIS version, because the netcdf file looks ok, in ncdump.
Could you test, whether you can open the file at
https://upload.uni-jena.de/data/5f3400804a46b4.93034284/lakecuberqa2.nc

I would also suggest, to not set the version number of YAXArrays directly to 1.0 because we still have some deprecations in there and I would also like to talk again about reducing the number of submodules. We could also think about renaming functions with cube in the name, because at the moment, this is a mix of Cube naming and YAXArray naming.

I would suggest to first make the split and then we can tidy up YAXArrays further.

[meggart]

I just tested your file and could read it using NetCDF.jl without problems. I could not try ArchGDAL though, because I don't have the NetCDF driver installed. Otherwise I agree with your comments about version 1.0 and the removing of the remaining "cube" functions.

[felixcremer]

I am still testing things and it seems that the type inference for RangeAxis is not working, because when I get the eltype of a RangeAxis it is any.

julia> testax = RangeAxis(:Test, 1:10)
Test                Axis with 10 Elements from 1 to 10

julia> eltype(testax)
Any

julia> eltype(testax.values)
Int64

On ESDL master testax has an eltype of Int64.
This is a problem when I try to use the TimeAxis in a computation, because I can't set the CacheSizes and the computation fails with the following error:

ERROR: Abstract type Any does not have a definite size.
Stacktrace:
 [1] sizeof at ./essentials.jl:449 [inlined]
 [2] mysizeof(::Type{T} where T) at /home/crem_fe/.julia/dev/YAXArrays/src/DAT/DAT.jl:564
 [3] (::YAXArrays.DAT.var"#463#467")(::Array{Int64,1}, ::YAXArrays.DAT.InputCube{1}) at /home/crem_fe/.julia/dev/YAXArrays/src/DAT/DAT.jl:580
 [4] #3 at ./generator.jl:36 [inlined]
 [5] iterate at ./generator.jl:47 [inlined]
 [6] collect_to!(::Array{Int64,1}, ::Base.Generator{Base.Iterators.Zip{Tuple{Array{Array{Int64,1},1},Tuple{YAXArrays.DAT.InputCube{3},YAXArrays.DAT.InputCube{1}}}},Base.var"#3#4"{YAXArrays.DAT.var"#463#467"}}, ::Int64, ::Tuple{Int64,Int64}) at ./array.jl:732
 [7] collect_to_with_first!(::Array{Int64,1}, ::Int64, ::Base.Generator{Base.Iterators.Zip{Tuple{Array{Array{Int64,1},1},Tuple{YAXArrays.DAT.InputCube{3},YAXArrays.DAT.InputCube{1}}}},Base.var"#3#4"{YAXArrays.DAT.var"#463#467"}}, ::Tuple{Int64,Int64}) at ./array.jl:710
 [8] collect(::Base.Generator{Base.Iterators.Zip{Tuple{Array{Array{Int64,1},1},Tuple{YAXArrays.DAT.InputCube{3},YAXArrays.DAT.InputCube{1}}}},Base.var"#3#4"{YAXArrays.DAT.var"#463#467"}}) at ./array.jl:691
 [9] map(::Function, ::Array{Array{Int64,1},1}, ::Tuple{YAXArrays.DAT.InputCube{3},YAXArrays.DAT.InputCube{1}}) at ./abstractarray.jl:2248
 [10] getCacheSizes(::YAXArrays.DAT.DATConfig{2,1}, ::Dict{Any,Any}) at /home/crem_fe/.julia/dev/YAXArrays/src/DAT/DAT.jl:586
 [11] mapCube(::typeof(ESDLHelpers.clombscargle), ::Tuple{YAXArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.IDataset},Array{CubeAxis,1}},RangeAxis{Date,:Time,Array{Date,1}}}; max_cache::Float64, indims::Tuple{InDims,InDims}, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Array{Int64,1}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/YAXArrays/src/DAT/DAT.jl:273
 [12] lombscargle(::YAXArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.IDataset},Array{CubeAxis,1}}) at /home/crem_fe/.julia/dev/ESDLHelpers/src/tsanalysis.jl:78
 [13] top-level scope at REPL[38]:1
 [14] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088

[felixcremer]

Removing this typehint gives a clash with the inner gapfillpoly function a few lines below and Julia starts to complain about compilation problems. I circumvented the problem for now, by renaming the inner gapfillpoly function to gapfillpoly! but I am not sure, whether this is the way to go.

[meggart]

Yes, that is the way to go, alternatively, _gapfillpoly would be ok as well.

[meggart]

The reason for removing the AbstractCubeData type was that in the medium term I want to make the operations in YAXArrays work for any array that implements the interface in YAXArrayBase, which means you can directly plug in a DImensionalArray or AxisArray.

[felixcremer]

I had to put this include below the using YAXArrays lines, because otherwise some YAXArray type wasn't defined yet which is used in the cubeinfo function.

[felixcremer]

The type inference fails also for categorical axes.

[meggart]

I just fixed the eltype bug which was simply a missing method

[felixcremer]

What is the state of this?
Is there anything I could help with to push this forward?

[meggart]

I think we can merge now. I have registered YAXArrays, so that the unit tests pass again now, so I think I will just do it....