serializing closures is slow on master

[amitmurthy]

using Serialization

function foo(n)
    io=IOBuffer()
    f = x->x
    @time for i in 1:10^n
        serialize(io, f)
        seekstart(io)
        deserialize(io)
    end
end

function bar(n)
    io=IOBuffer()
    @time for i in 1:10^n
        serialize(io, "Hello World")
        seekstart(io)
        deserialize(io)
    end
end

foo(1)
bar(1)
foo(4)
bar(4)

On 0.4

julia> foo(4)
  0.208748 seconds (680.01 k allocations: 35.706 MB, 1.70% gc time)

julia> bar(4)
  0.010271 seconds (100.00 k allocations: 9.003 MB, 24.10% gc time)

On master

julia> foo(4)
  3.409323 seconds (9.96 M allocations: 541.994 MB, 1.60% gc time)

julia> bar(4)
  0.012568 seconds (80.06 k allocations: 9.616 MB, 19.17% gc time)

@andreasnoack, your hunch was correct, this looks like the cause of the slowdown in distributed arrays - https://travis-ci.org/JuliaParallel/DistributedArrays.jl/builds/131800408

[amitmurthy]

cc: @shashi could also explain the slowdown you are seeing in ComputeFramework.

[ViralBShah]

This is important for 0.5.

[StefanKarpinski]

#16695 helps but is kind of a workaround?

[amitmurthy]

I'll leave this issue open as #16774 optimized it only for parallel processing - and still not upto 0.4 levels. Will remove the 0.5 tag though, because, as discussed in #16774, this is the best for now.

[ExpandingMan]

Not completely sure it's related to this, but deserializing large objects (in this case DataFrames) from disk seems to take significantly longer on 0.5 (took me 9 seconds for a 38 MB dataframe). I'm afraid I can't give my 0.4 number because I replaced the 0.4 build, but I'm quite sure it was much faster (maybe 1 s at most?). Serialization, on the other hand, seems fine.

I did some further testing and it seems that this depends strongly on what the stored datatypes are. I had a 300MB dataframe of floats load in 1s, but 38MB of mixed strings, DateTime and Float takes forever (9s).

[oscardssmith]

Do we still care about this? It seems not to have significantly improved between 0.5 and master, but maybe we should close as wont-fix since no one has complained in 4 years?

[oxinabox]

Serializing closures is pretty cruicial for our distributed processing, so yes?

[timholy]

Julia 0.5 is when fast closures got added, so maybe it's just inevitable?

[ViralBShah]

Just tried the benchmark again, and still as slow as reported.

[vtjnash]

Yes, but we recommend not sending closures (rather load packages on all nodes), so this isn't really an issue

[andreasnoack]

If that is the case then it would be helpful with hints to how we can avoid closures when communicating between processes in e.g. DistributedArrays as mention above.