SharedArrays convert-to-array method broken

[fgerick]

I have run into some problems when using SharedArrays in functions. Simple example:

When starting julia with julia -p 4:

function foo(n = 5)
    Z = SharedArray(Float64,n)

    @sync @parallel for i = 1:length(Z)
            Z[i] = rand()
       end
    return Array(Z) #or Z.s or sdata(s) equivalently
end

calling

foo()

gives:

5-element Array{Float64,1}:

signal (11): Segmentation fault
alignment at show.jl:1012
print_matrix at show.jl:1087
print_matrix at show.jl:1079
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at replutil.jl:23
with_output_limit at ./show.jl:1276
writemime at replutil.jl:23
jlcall_writemime_21953 at  (unknown line)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
display at REPL.jl:114
jlcall_display_21952 at  (unknown line)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
display at REPL.jl:117
jlcall_display_21951 at  (unknown line)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
display at multimedia.jl:151
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
print_response at REPL.jl:134
jlcall_print_response_21747 at  (unknown line)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
print_response at REPL.jl:121
jlcall_print_response_21746 at  (unknown line)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
anonymous at REPL.jl:624
run_interface at ./LineEdit.jl:1610
unknown function (ip: 0x7f4e95f99580)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
run_frontend at ./REPL.jl:864
run_repl at ./REPL.jl:168
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
_start at ./client.jl:420
unknown function (ip: 0x7f4e95fb2ea9)
jl_apply_generic at /home/felix/julia/usr/bin/../lib/libjulia.so (unknown line)
unknown function (ip: 0x401ab7)
unknown function (ip: 0x4016bf)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401705)
unknown function (ip: (nil))
Segmentation fault

This does not occur when only one worker is active or if julia is started with julia and then addprocs(4) is called. However, a slightly alternated function:

function foo(n = 5)
    Z = SharedArray(Float64,n,n)
     for j = 1:size(Z,1)
    @sync @parallel for i = 1:size(Z,2)
            Z[i,j] = rand()
       end
    end
    return Array(Z) #or Z.s or sdata(s) equivalently
end

gives a segmentation fault, no matter if julia is started in parallel or addprocs(4) is called.

Also:

function foo(n = 5)
    Z = SharedArray(Float64,n)

    @sync @parallel for i = 1:length(Z)
            Z[i] = rand()
       end
    return Z
end

and then calling

x = foo()
x.s

does not give an error.

FYI:

julia> versioninfo()
Julia Version 0.4.6-pre+41
Commit bc363de* (2016-06-03 10:12 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i3-2348M CPU @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT NO_AFFINITY NEHALEM)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Best regards, Felix.

[vtjnash]

converting a SharedArray to an Array (per the fallback call-to-convert fallback constructor) simply discards the wrapper that owns the lifetime of the finalization of the inner Array object. The gc happily cleans up the garbage and deletes both.

(accessing a field directly, such as Z.s, is generally considered to be undefined behavior, so it can do whatever it wants, that's not a bug)

[yuyichao]

(accessing a field directly, such as Z.s, is generally considered to be undefined behavior, so it can do whatever it wants, that's not a bug)

More specifically, accessing a field that is not documented to be accessed directly is considered undefined behavior. This is especially true for types that has finalizers since the lifetime of the fields might be managed by the finalizer on the parent.

[vtjnash]

I found that the reshape method is also broken for a similar reason. Probably the underlying issue here is that the finalizer for SharedArray aggressively finalizes S.s (its underlying array), even though it would already be scheduled for finalization normally?