Indexing SubArrays with Int32 returns zero dimensional array

[nalimilan]

I've seen this error for the first time today when building RPM nightlies. It happened twice, both on 64-bit. This is with system LLVM 3.7.1 + Julia patches and ORCJIT.

    From worker 3:       * linalg/cholesky      Error During Test
    From worker 3:    Test threw an exception of type MethodError
    From worker 3:    Expression: norm(apd * (cpapd \ b) - b) / norm(b) <= ε * κ * n
    From worker 3:    MethodError: Cannot `convert` an object of type Array{Float32,0} to an object of type Float32
    From worker 3:    This may have arisen from a call to the constructor Float32(...),
    From worker 3:    since type constructors fall back to convert methods.
    From worker 3:    Closest candidates are:
    From worker 3:      convert(::Type{Float32}, !Matched::Int8)
    From worker 3:      convert(::Type{Float32}, !Matched::Int16)
    From worker 3:      convert(::Type{Float32}, !Matched::Int32)
    From worker 3:      ...
    From worker 3:     [inlined code] from ./range.jl:419
    From worker 3:     in setindex!(::SubArray{Float32,1,Array{Float32,2},Tuple{UnitRange{Int64},Int64},true}, ::Array{Float32,0}, ::Int64) at ./subarray.jl:182
    From worker 3:     [inlined code] from ./abstractarray.jl:341
    From worker 3:     in permute!!(::SubArray{Float32,1,Array{Float32,2},Tuple{UnitRange{Int64},Int64},true}, ::Array{Int32,1}) at ./combinatorics.jl:82
    From worker 3:     [inlined code] from ./array.jl:64
    From worker 3:     in A_ldiv_B!(::Base.LinAlg.CholeskyPivoted{Float32,Array{Float32,2}}, ::Array{Float32,2}) at ./linalg/cholesky.jl:252
    From worker 3:     in \(::Base.LinAlg.CholeskyPivoted{Float32,Union{DenseArray{Float32,2},SubArray{Float32,2,A<:DenseArray{T,N},I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex{N},Base.NoSlice,Colon,Int64,Range{Int64}}}},L}}}, ::Array{Float32,2}) at ./linalg/factorization.jl:39
    From worker 3:     [inlined code] from /builddir/build/BUILD/julia/test/linalg/cholesky.jl:116
    From worker 3:     in anonymous at ./no file:4294967295
    From worker 3:     [inlined code] from ./essentials.jl:78
    From worker 3:     in include_string(::UTF8String, ::ASCIIString) at ./loading.jl:371
    From worker 3:     in include_from_node1(::ASCIIString) at ./loading.jl:420
    From worker 3:     [inlined code] from ./util.jl:179
    From worker 3:     in runtests(::ASCIIString) at /builddir/build/BUILD/julia/test/testdefs.jl:7
    From worker 3:     in (::Base.Serializer.__deserialized_types__.##9046)(::ASCIIString) at /builddir/build/BUILD/julia/test/runtests.jl:36
    From worker 3:     in run_work_thunk(::Base.##254#256{Base.CallMsg{:call_fetch}}, ::Bool) at ./multi.jl:714
    From worker 3:     [inlined code] from ./multi.jl:1010
    From worker 3:     in (::Base.##253#255{Base.CallMsg{:call_fetch},TCPSocket})() at ./task.jl:59
Exception running test linalg/cholesky :
On worker 3:
LoadError: There was an error during testing
 in record at ./test.jl:317
 in do_test at ./test.jl:215
 [inlined code] from ./test.jl:137
 in anonymous at ./no file:4294967295
 [inlined code] from ./essentials.jl:78
 in include_string at ./loading.jl:371
 in include_from_node1 at ./loading.jl:420
 [inlined code] from ./util.jl:179
 in runtests at /builddir/build/BUILD/julia/test/testdefs.jl:7
 in #16 at /builddir/build/BUILD/julia/test/runtests.jl:36
 in run_work_thunk at ./multi.jl:714
 [inlined code] from ./multi.jl:1010
 in #253 at ./task.jl:59
while loading /builddir/build/BUILD/julia/test/linalg/cholesky.jl, in expression starting on line 24ERROR: LoadError: Some tests exited with errors.
 in (::##11#19)() at /builddir/build/BUILD/julia/test/runtests.jl:64
 in cd(::##11#19, ::ASCIIString) at ./file.jl:47
 in include(::ASCIIString) at ./boot.jl:264
 in include_from_node1(::UTF8String) at ./loading.jl:417
 in process_options(::Base.JLOptions) at ./client.jl:262
 in _start() at ./client.jl:318
while loading /builddir/build/BUILD/julia/test/runtests.jl, in expression starting on line 13
WARNING: Forcibly interrupting busy workers
ArgumentError("stream is closed or unusable")

https://copr-be.cloud.fedoraproject.org/results/nalimilan/julia-nightlies/epel-7-x86_64/00161544-julia/build.log.gz
https://copr-be.cloud.fedoraproject.org/results/nalimilan/julia-nightlies/fedora-rawhide-x86_64/00161544-julia/build.log.gz

[nalimilan]

Note that it never happens on 32-bit.

[nalimilan]

I get this 100% of the time on 64-bit. What can I do to help debugging? Would a bisection be useful?

[yuyichao]

Can you reduce the test case? (e.g. reproduce it in a single process test/single test and try to remove unrelated code from the test)

[yuyichao]

Hmmm, seems that I can reproduce it locally too.

[yuyichao]

Reduced test case

n = 10
srand(1234321)
areal = randn(n, n) / 2
breal = randn(n, 2) / 2
a = convert(Matrix{Float32}, areal)
apd  = a'*a                  # symmetric positive-definite
cpapd = cholfact(apd, :U, Val{true})
b = convert(Matrix{Float32}, breal)
cpapd\b

@andreasnoack ?

[andreasnoack]

Anything special about your setup? I don't get the error.

[yuyichao]

Just to check if you know anything related changes recently.

[andreasnoack]

I don't think so. I wondering if other changes in base could have affected this. The code lowering changes are definitely changing behavior for some packages.

Long shot, but I noticed that the type for pivoted Cholesky is weird. Would you try to substitute the StridedMatrix{T} in

julia/base/linalg/cholesky.jl

Line 123 in 4ce06c2

return CholeskyPivoted{T,StridedMatrix{T}}(A, uplochar, piv, rank, tol, info)

with typeof(A) and see what happens.

[nalimilan]

I also see it locally when building the RPM, but not when building from the Julia tree (i.e. without all of the USE_SYSTEM_*=1). Maybe an issue with 32-bit BLAS integers (LP64)?

[andreasnoack]

Maybe an issue with 32-bit BLAS integers (LP64)?

Yes. I've just been able to reproduce this with MKL LP64.

[yuyichao]

I won't be able to check in the next few ours but for the record I'm using system-wise installed openblas. (configure)

[andreasnoack]

This is not an issue with CholeskyPivoted, but an issue with permute! for SubArrays + Int32 permutation vector.

julia> permute!(sub(rand(10), 1:10), map(Int32, randperm(10)))
ERROR: MethodError: Cannot `convert` an object of type Array{Float64,0} to an object of type Float64
This may have arisen from a call to the constructor Float64(...),
since type constructors fall back to convert methods.

LP64 + Pivoted Cholesky just happens to produce a Vector{Int32} permutation.

[andreasnoack]

Indexing of SubArrays is returning a zero dimensional array when the index is Int32. @mbauman This has probably changed lately. Do you have a good fix for this?

julia> sb = sub(rand(10), 1:10, 1)
10-element SubArray{Float64,1,Array{Float64,1},Tuple{UnitRange{Int64},Int64},true}:
 0.0707466
 0.646484 
 0.676914 
 0.30983  
 0.295709 
 0.645556 
 0.631727 
 0.724235 
 0.130693 
 0.684235 

julia> sb[1]
0.07074656544530544

julia> sb[Int32(1)]
0-dimensional Array{Float64,0}:
0.0707466

[mbauman]

Yup, this is certainly my doing. I thought that #15144 should have prevented this, but I'll take another look.