JLD2 vs Knet load behave differently

[denizyuret]

When loading this file we seem to have a problem:

using JLD2, FileIO
a1 = FileIO.load("mnist.jld2","dtst")
a2 = Knet.load("mnist.jld2","dtst")
first(a1) => ok
first(a2) => error

As far as I can tell, a1 and a2 are identical except for the xtype field.

for n in fieldnames(Knet.Data)
    println(n,"\t",getfield(a1,n)==getfield(a2,n))
end
x	true
y	true
batchsize	true
length	true
partial	true
indices	true
shuffle	true
xsize	true
ysize	true
xtype	false
ytype	true

Both a1.xtype and a2.xtype print the same but they seem to be not equal! If I replace a2.xtype = a1.xtype it starts working. You may want to check how types are saved...

[ekinakyurek]

I actualy do nothing for types, it is very strange. I could not see why this is happening. Do you have any idea?

[denizyuret]

When saving and loading dictionaries, the Dict type is not preserved.

[ekinakyurek]

this issue is still exist (Julia v1.1, Knet v1.2.0), and very interesting

julia> using Knet, JLD2, FileIO

julia> a2 = Knet.load("mnist.jld2","dtst")
┌ Warning: read type Knet.Data is not a leaf type in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/KjBIK/src/data.jl:477
getfield(JLD2.ReconstructedTypes, Symbol("##Knet.Data#363"))(Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], UInt8[0x07 0x02 … 0x05 0x06], 100, 10000, false, 1:10000, false, (28, 28, 1, 10000), (10000,), Array{Float32,N} where N, Array{UInt8,1})

julia> a1 = FileIO.load("mnist.jld2","dtst")
┌ Warning: read type Knet.Data is not a leaf type in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/KjBIK/src/data.jl:477
getfield(JLD2.ReconstructedTypes, Symbol("##Knet.Data#364"))(Float32[0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], UInt8[0x07 0x02 … 0x05 0x06], 100, 10000, false, 1:10000, false, (28, 28, 1, 10000), (10000,), Array{Float32,N} where N, Array{UInt8,1})

julia> a1.xtype == Array{Float32}
true

julia> a2.xtype == Array{Float32}
false

julia> a1.xtype == a2.xtype
false

julia> dump(a1.xtype)
UnionAll
  var: TypeVar
    name: Symbol N
    lb: Core.TypeofBottom Union{}
    ub: Any
  body: Array{Float32,N} <: DenseArray{Float32,N}

julia> dump(a2.xtype)
UnionAll
  var: TypeVar
    name: Symbol N
    lb: Core.TypeofBottom Union{}
    ub: Any
  body: Array{Float32,N} <: DenseArray{Float32,N}

[ekinakyurek]

I think @JeffBezanson may have a comment on that.

[JeffBezanson]

Is the TypeVar N === to the one inside the Array type in the body field?

[ekinakyurek]

julia> a1.xtype.var == a2.xtype.var
false

julia> a1.xtype.var === a2.xtype.var
false

julia> a1.xtype.var
N

julia> a2.xtype.var
N

julia> a1.xtype.body.parameters[2] == a1.xtype.var
true

julia> a2.xtype.body.parameters[2] == a2.xtype.var
false

[ekinakyurek]

We were replicating deepcopy in our saving code. So, I realized this JuliaLang/julia#30911