Indexing with end fails in simple cases

[ali-ramadhan]

I tend to index using end a lot, e.g. to get the output at the last time step. But it seems to fail in some simple cases.

A[X(1:3:end), Ti(1:2:end)] (taken from the docs: https://rafaqz.github.io/DimensionalData.jl/dev/course/#Indexing-the-array-by-name-and-index) works but not A[X(1:3:end)], A[Y(1:2:end)], A[X(1:end)], A[X=1:end], A[X=end], and A[X(end)], etc.

Minimal working examples

julia> using DimensionalData

julia> A = DimArray(randn(10, 10), (X, Y))
DimArray with dimensions:
 X (type X): Base.OneTo(10) (NoIndex)
 Y (type Y): Base.OneTo(10) (NoIndex)
and data: 10×10 Array{Float64,2}
  1.74281    1.91806    -0.412829    0.649741  -1.31292   -2.03106     0.541802    2.4614     2.65145   -0.739313
  2.40075   -0.601865    1.08449    -0.488991   0.12993    0.957945   -0.128312   -0.004756   0.796635   0.914179
  0.250125   0.233316   -0.403383   -0.562963   0.464434  -0.737124   -2.86539     2.3688     1.64044   -0.617462
 -0.144088  -1.27702    -0.918338   -0.408036   1.46215   -0.738956   -0.482546    1.85599    0.114205   0.966627
 -0.340203  -0.138912   -0.164005   -0.690616  -0.212402   2.68914     0.603413   -2.07776   -2.03773    0.694398
 -2.37627   -0.0352735   0.988174   -0.647816  -0.569869  -0.336676    0.346312    0.599265  -0.22692   -0.247725
 -1.44058   -0.111664   -0.247909    0.063913   0.10068    0.115982   -0.422838   -1.82162    0.815291  -0.299867
  0.142698  -1.16767     0.0278394  -0.111511   1.03004   -1.54426    -0.612883   -1.17298   -1.16226   -1.76471
  0.416233   1.65931     0.94561     0.152582  -0.609753   0.0193246  -0.442374   -0.237472  -0.022429   0.257218
  0.268961  -0.777021    0.377889    0.469105  -0.658532   0.325553   -0.0075319   0.95508   -2.27181    1.69276

julia> A[X(1:3:end), Y(1:2:end)]
DimArray with dimensions:
 X (type X): 1:3:10 (NoIndex)
 Y (type Y): 1:2:9 (NoIndex)
and data: 4×5 Array{Float64,2}
  1.74281   -0.412829  -1.31292    0.541802    2.65145
 -0.144088  -0.918338   1.46215   -0.482546    0.114205
 -1.44058   -0.247909   0.10068   -0.422838    0.815291
  0.268961   0.377889  -0.658532  -0.0075319  -2.27181

julia> A[X(1:3:end)]
ERROR: BoundsError: attempt to access 10×10 Array{Float64,2} at index [1:3:100, 1:10]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{StepRange{Int64,Int64},Base.Slice{Base.OneTo{Int64}}}) at ./abstractarray.jl:541
 [2] checkbounds at ./abstractarray.jl:506 [inlined]
 [3] _getindex at ./multidimensional.jl:742 [inlined]
 [4] getindex at ./abstractarray.jl:1060 [inlined]
 [5] getindex at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:98 [inlined]
 [6] getindex(::DimArray{Float64,2,Tuple{X{Base.OneTo{Int64},NoIndex,Nothing},Y{Base.OneTo{Int64},NoIndex,Nothing}},Tuple{},Array{Float64,2},Symbol,Nothing}, ::X{StepRange{Int64,Int64},AutoMode{AutoOrder},Nothing}) at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:127
 [7] top-level scope at REPL[16]:1

julia> A[Y(1:2:end)]
ERROR: BoundsError: attempt to access 10×10 Array{Float64,2} at index [1:10, 1:2:99]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{Base.Slice{Base.OneTo{Int64}},StepRange{Int64,Int64}}) at ./abstractarray.jl:541
 [2] checkbounds at ./abstractarray.jl:506 [inlined]
 [3] _getindex at ./multidimensional.jl:742 [inlined]
 [4] getindex at ./abstractarray.jl:1060 [inlined]
 [5] getindex at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:98 [inlined]
 [6] getindex(::DimArray{Float64,2,Tuple{X{Base.OneTo{Int64},NoIndex,Nothing},Y{Base.OneTo{Int64},NoIndex,Nothing}},Tuple{},Array{Float64,2},Symbol,Nothing}, ::Y{StepRange{Int64,Int64},AutoMode{AutoOrder},Nothing}) at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:127
 [7] top-level scope at REPL[17]:1

julia> A[X=end]
ERROR: BoundsError: attempt to access 10×10 Array{Float64,2} at index [100, 1:10]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{Int64,Base.Slice{Base.OneTo{Int64}}}) at ./abstractarray.jl:541
 [2] checkbounds at ./abstractarray.jl:506 [inlined]
 [3] _getindex at ./multidimensional.jl:742 [inlined]
 [4] getindex at ./abstractarray.jl:1060 [inlined]
 [5] getindex at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:98 [inlined]
 [6] getindex at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:127 [inlined]
 [7] #getindex#16 at /home/alir/.julia/packages/DimensionalData/4a7uE/src/array.jl:134 [inlined]
 [8] top-level scope at REPL[18]:1

Julia environment

julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, cascadelake)

(@v1.5) pkg> st
Status `~/.julia/environments/v1.5/Project.toml`
  [c7e460c6] ArgParse v1.1.0
  [fbb218c0] BSON v0.2.6
  [6e4b80f9] BenchmarkTools v0.5.0
  [052768ef] CUDA v1.3.3
  [8f4d0f93] Conda v1.4.1
  [d58978e5] Dagger v0.9.2
  [68e73e28] DaggerGPU v0.1.1 `https://github.com/jpsamaroo/DaggerGPU.jl.git#master`
  [933edf66] DarkMode v0.1.0 `https://github.com/Pocket-titan/DarkMode#master`
  [a93c6f00] DataFrames v0.21.8
  [0703355e] DimensionalData v0.13.3
  [9b6fcbb8] GeoData v0.2.1 `https://github.com/rafaqz/GeoData.jl.git#master`
  [7073ff75] IJulia v1.21.4
  [85f8d34a] NCDatasets v0.10.4
  [9e8cae18] Oceananigans v0.43.0
  [c3e4b0f8] Pluto v0.12.4
  [08abe8d2] PrettyTables v0.9.1 `https://github.com/ronisbr/PrettyTables.jl.git#master`
  [438e738f] PyCall v1.92.1
  [d330b81b] PyPlot v2.9.0
  [295af30f] Revise v3.1.3
  [a759f4b9] TimerOutputs v0.5.6

[rafaqz]

You can't use end! It's sugar for lastindex in array syntax - and this isn't regular indexing, as the dimensions can be in any order and you can skip some.

There is a good argument to be made to move away from using A[X(1), Y(2)]] syntax and to use A(X(1), Y(2))` where end isn't a thing at all.

You can use lastindex(A, Y) instead, or just not use the Dimension wrappers at all.

[ali-ramadhan]

Ah that's unfortunate. Being able to do things like A[Ti=end] would make for very user-friendly syntax.

I tried digging into the source code but couldn't figure out how to make dim2indices or sel2indices support end in the way I want. Not too familiar with how Julia handles arrays and end so not sure if it's possible. Might try again later though.

The A[X(1:3:end), Ti(1:2:end)] example from the crash course seemed to suggest that you could use end. If end is not supposed to be used, then maybe that example should be removed?

[rafaqz]

end is parser syntax, it's not an object we can actually code with.

That's probably because I didn't write the crash course, and I already had to edit it a lot ;) But yes it should be removed!

We could add an object End, but it still wouldn't work in a range like you want. It would have to be a macro that replaced
end with lastindex(A, Ti). But macros are kind of lame.

[rafaqz]

This package might help you:

https://github.com/JuliaArrays/EndpointRanges.jl

[rafaqz]

Also see: JuliaLang/julia#38296

[ali-ramadhan]

Ah neat, did not know about EndpointRanges.jl! Will look into it.

I guess Python/xarray gets away with it in this case because Python always assumes that arrays start at 0 so -1 can always be used for the last index, while in Julia arrays can use more flexible indexing.

Feel free to close this issue (or maybe you want to keep it open until end is removed from the crash course).

[rafaqz]

Its annoying me this doesn't work and we need to load another package to make it work.

We should repurpose End that we already have and add Begin (which can alias Start) and just make this standard DimensionalData.jl syntax.