loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry #582

dehann · 2020-08-09T09:46:02Z

finalGraph.tar.gz

On Julia 1.4.2

julia> using Flux
julia> using RoME
julia> dfg = initfg()
julia> loadDFG!(dfg, "finalGraph")  # or loadDFG!(dfg, "finalGraph.tar.gz")

sfolder = split(dstname, '.') = SubString{String}["/media/dehann/logs3/caesar/2020-08-09T05:30:25", "189/finalGraph", "tar", "gz"]
[ Info: loadDFG detected a gzip /media/dehann/logs3/caesar/2020-08-09T05:30:25.189/finalGraph.tar.gz -- unpacking via /tmp/caesar/random/6db72930-da24-11ea-229f-95665d6ce73b now...
ERROR: ArgumentError: type does not have a definite number of fields
Stacktrace:
 [1] fieldcount(::Any) at ./reflection.jl:705
 [2] #s16#65 at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443 [inlined]
 [3] #s16#65(::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ./none:0
 [4] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:526
 [5] read at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443 [inlined]
 [6] macro expansion at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:259 [inlined]
 [7] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{TimeZones.ZonedDateTime}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443
 [8] read at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443 [inlined]
 [9] macro expansion at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:259 [inlined]
 [10] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{BlobStoreEntry}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443
 [11] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{BlobStoreEntry}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443
 [12] read(::String, ::Type{T} where T; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:36
 [13] read(::String, ::Type{T} where T) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:36
 [14] unpackVariable(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Dict{String,Any}; unpackPPEs::Bool, unpackSolverData::Bool, unpackBigData::Bool) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:76
 [15] unpackVariable at /home/dehann/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:31 [inlined]
 [16] (::DistributedFactorGraphs.var"#118#124"{LightDFG{SolverParams,DFGVariable,DFGFactor},Array{DFGVariable,1}})(::IOStream) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/FileDFG/services/FileDFG.jl:134
 [17] open(::DistributedFactorGraphs.var"#118#124"{LightDFG{SolverParams,DFGVariable,DFGFactor},Array{DFGVariable,1}}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:298
 [18] open at ./io.jl:296 [inlined]
 [19] loadDFG!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::String) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/FileDFG/services/FileDFG.jl:132
 [20] top-level scope at REPL[19]:1

Affie · 2020-08-09T10:02:38Z

captured here #557

dehann · 2020-08-09T10:03:28Z

right thanks, so the data Entry is the issue perhaps?

Affie · 2020-08-09T10:08:01Z

Yes, part of my ongoing pain with serialization. If I remember correctly it was the addition of ZonedTimeStamp that eventually broke it.

Affie · 2020-08-09T10:09:59Z

I'm tempted to change everything to use the pattern in https://github.com/JuliaData/StructTypes.jl + JSON3

dehann · 2020-08-09T10:27:09Z

I'm tempted to change everything

Think we have other bigger issues to resolve first? We keep fixing serialization stuff over and over, so lets talk next week on what the most important things are too fix with serialization and balance that with what progress is required on other fronts. I'd say hold off on JSON3 stuff for now, if it can.

part of my ongoing pain with serialization. If I remember correctly it was the addition of ZonedTimeStamp

Haha, jip I believe it. The repercussions of small changes in DFG are dramatic downstream. From that quick ZonedDateTime change in #517 :-) At least this is a good test case for current dev work.

Affie · 2020-08-09T10:37:30Z

Actually that one is broken in another way: #560

I'm speaking about this one:

DistributedFactorGraphs.jl/src/DataBlobs/entities/AbstractDataEntries.jl

Line 31 in d1f3c41

createdTimestamp::ZonedDateTime # of when the entry was created

dehann · 2020-08-09T10:46:01Z

Another thing, we actively decided after a 2 year process not to define the "schema" for serialization. We originally had a protobuf packing approach which was hard schema type dependent and which supported forward-backward multilanguage compatibility. After much pain, we decided that top level packing MUST be JSON, and that the JSON string must at top level include the format in which the data is serialized. This is to mimic how the Internet has standardized it's data exchange format. So StructTypes.jl might be a real problem given the history on serialization.

So to stress, its probably not a good idea going down a path to unilaterally "change everything" on serialization.

In this case, it will be better to understand what the problems are with JSON.jl JSON2.jl JSON3.jl, because clearly they haven't figured it all out either. But we likely won't go down writing JSON4.jl either.

Also remember that the whole point of JSON is that serialization should be marginally or completely available outside Julia. The ZMQ interface for example is about multilanguage support and it uses the same serialization. The DB storage is definitely not a Julia implementation, later we will add a FuseDFG driver, etc, etc. A little while back we came up with the requirement that ZMQ messages should be exactly the same as FileDFG content. The schema discussion becomes a whole issue there. There are 100 different schema's available... So serialization only has 2nd best answers for everything. Whenever you pick the "best" answer for one requirement, then all others get real upset. So we have to accept a 2nd best solution for everybody and JSON is the best bet at this time.

Again, we should discuss this in the upcoming call, but can guarantee a schema'ed serialization approach is a touchy subject.

cc @GearsAD

dehann · 2020-08-09T11:10:09Z

xref JuliaRobotics/Caesar.jl#311

GearsAD · 2020-08-11T05:15:12Z

Definitely worth discussing on Friday. Is this still an issue?

dehann · 2020-08-11T06:46:41Z

now on Julia 1.5.0

Jip, im trying to debug a bit, but not familiar enough with this part of the code yet. I modified Serialization.jl Line 75 to print:

        for (k,bdeInter) in dataIntermed
            @show bdeInter
            @show dataElemTypes[k], getfield(DistributedFactorGraphs, dataElemTypes[k])
            fullVal = JSON2.read(bdeInter, getfield(DistributedFactorGraphs, dataElemTypes[k]))
            variable.dataDict[k] = fullVal
        end

Gives this on the data uploaded in the first comment:

(dataElemTypes[k], getfield(DistributedFactorGraphs, dataElemTypes[k])) = (:BlobStoreEntry, DistributedFactorGraphs.BlobStoreEntry)

ERROR: ArgumentError: type does not have a definite number of fields
Stacktrace:
 [1] fieldcount(::Any) at ./reflection.jl:722
 [2] #s17#65 at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:443 [inlined]
 [3] #s17#65(::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ./none:0
 [4] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [5] read at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:442 [inlined]
 [6] macro expansion at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:259 [inlined]
 [7] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{TimeZones.ZonedDateTime}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:442
 [8] read at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:442 [inlined]
 [9] macro expansion at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:259 [inlined]
 [10] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{DistributedFactorGraphs.BlobStoreEntry}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:442
 [11] read(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{DistributedFactorGraphs.BlobStoreEntry}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:442
 [12] read(::String, ::Type{T} where T; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:36
 [13] read(::String, ::Type{T} where T) at /home/dehann/.julia/packages/JSON2/ld4Kq/src/read.jl:36
 [14] unpackVariable(::LightDFG{SolverParams,DistributedFactorGraphs.DFGVariable,DistributedFactorGraphs.DFGFactor}, ::Dict{String,Any}; unpackPPEs::Bool, unpackSolverData::Bool, unpackBigData::Bool) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:78
 [15] unpackVariable at /home/dehann/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:31 [inlined]
 [16] (::DistributedFactorGraphs.var"#118#124"{LightDFG{SolverParams,DistributedFactorGraphs.DFGVariable,DistributedFactorGraphs.DFGFactor},Array{DistributedFactorGraphs.DFGVariable,1}})(::IOStream) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/FileDFG/services/FileDFG.jl:134
 [17] open(::DistributedFactorGraphs.var"#118#124"{LightDFG{SolverParams,DistributedFactorGraphs.DFGVariable,DistributedFactorGraphs.DFGFactor},Array{DistributedFactorGraphs.DFGVariable,1}}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:325
 [18] open at ./io.jl:323 [inlined]
 [19] loadDFG!(::LightDFG{SolverParams,DistributedFactorGraphs.DFGVariable,DistributedFactorGraphs.DFGFactor}, ::String) at /home/dehann/.julia/dev/DistributedFactorGraphs/src/FileDFG/services/FileDFG.jl:132
 [20] top-level scope at none:1

Some example data:

 Debug: DECODING Softtype = Pose2
└ @ DistributedFactorGraphs ~/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:117
bdeInter = "{\"label\":\"JOYSTICK_CMD_VALS\",\"id\":{\"value\":56433955245256332240062641469242912422},\"blobstore\":\"default_folder_store\",\"hash\":\"fc2c9f7456ddd63e6b77b558381d72d82f0b839fb6c4cb6f5ea6aa44cb8fd783\",\"origin\":\"DefaultUser|DefaultRobot|Session_4071d1|x0\",\"description\":\"\",\"mimeType\":\"\",\"createdTimestamp\":{\"utc_datetime\":\"2020-08-11T05:37:37.241\",\"timezone\":{\"name\":\"America/New_York\",\"transitions\":[{\"utc_datetime\":\"8511-01-01T00:00:00.0\",\"zone\":{\"name\":\"LMT\",\"offset\":{\"std\":{\"value\":-17762},\"dst\":{\"value\":0}}}},{\"utc_datetime\":\"1883-11-18T17:00:00.0\",\"zone\":{\"name\":\"EST\",\"offset\":{\"std\":{\"value\":-18000},\"dst\":{\"value\":0}}}},{\"utc_datetime\":\"1918-03-31T07:00:00.0\",\"zone\":{\"name\":\"EDT\",\"offset\":{\"std\":{\"value\":-18000},

GearsAD · 2020-08-11T07:06:53Z

This is due to the ZonedTimestamp being serialized in as-is into the structure. If you check one of the files, the createdTimestamp of a dataEntry is basically a whole timezone dump. It should just be a string.

dehann · 2020-08-11T07:09:42Z

Okay, load did not work with #588 merged, but let me create a new tar.gz and see if that does the trick...

dehann · 2020-08-11T07:10:29Z

just pointing to the link again

GearsAD · 2020-08-11T07:14:07Z

All the x variables still have that issue. I'm looking for a quick fix.

dehann · 2020-08-11T07:24:03Z

newfinalGraph.tar.gz this was created with #588 included

dehann · 2020-08-11T07:27:17Z

nope unfortunately still the same bug -- can continue with this tomorrow or later in the week. Don't worry about pushing too late tonight. Thanks for looking!

GearsAD · 2020-08-11T07:43:06Z

Okay so there's two problems here:

The ID is a GUID and it's being serialized as a massive int. We need to parse that to a string before saving it.
Similarly for the createdTimestamp, needs to be parsed to a string.

Discussion

I'd recommend using JSON and Unmarshal here, rather than JSON2. JSON has a lot more flexibility with preparsing, and it does at least handle the createdTimestamp correctly. The UUID, we need a custom parser for it.

The start of saving it correctly:

using TimeZones
using UUIDs
bse = BlobStoreEntry(:a, uuid4(), :abc, "", "", "", "", now(tz"UTC"))
using JSON
JSON.json(bse)

Gives (note the UUID is still wrong):

"{\"label\":\"a\",\"id\":{\"value\":211578186885085887193359900485672913068},\"blobstore\":\"abc\",\"hash\":\"\",\"origin\":\"\",\"description\":\"\",\"mimeType\":\"\",\"createdTimestamp\":\"2020-08-11T07:29:57.423+00:00\"}"

To parse back:

using JSON
using Unmarshal
interm = JSON.parse("{\"label\":\"a\",\"id\":{\"value\":211578186885085887193359900485672913068},\"blobstore\":\"abc\",\"hash\":\"\",\"origin\":\"\",\"description\":\"\",\"mimeType\":\"\",\"createdTimestamp\":\"2020-08-11T07:29:57.423+00:00\"}"
Unmarshal.unmarshal(BlobStoreEntry, interm)

Proposed Fix

Okay, if it's not 100% urgent tonight I better head out, but will look tomorrow morning again.

Current things to do:

Get JSON to emit the UUID correctly. Here's a quick fix using JSON a custom formatter (will emit UUID's correctly):

using JSON
import JSON.show_json
import JSON.Writer: StructuralContext, JSONContext, show_json
import JSON.Serializations: CommonSerialization, StandardSerialization
JSON.show_json(io::JSONContext, serialization::CommonSerialization, uuid::UUID) = print(io.io, "\"$uuid\"")

json = JSON.json(bse)

Unmarshal this correctly. This should work:

interm =JSON.parse(json)
using Unmarshal
Unmarshal.unmarshal(BlobStoreEntry, interm)

This recovers the correct data:

BlobStoreEntry(:a, UUID("9f2c7a00-7fc0-4f82-ad7f-560846ca48ac"), :abc, "", "", "", "", ZonedDateTime(2020, 8, 11, 7, 29, 57, 423, tz"UTC"))

dehann · 2020-08-11T07:45:42Z

Cool thanks, this is plenty of info thanks -- head out and we'll catch up another time!

GearsAD · 2020-08-11T07:52:12Z

Can you try this? #589

dehann · 2020-08-11T08:00:33Z

running new solve... will take 5-10mins

dehann · 2020-08-11T08:04:00Z

loading the old file (expected to fail) does but with a different error -- think that has to do with one of the items you mentioned earlier. New solve is under way I'll try as fast as possible. It's about halfway now... Thanks for pushing a fix so quick!

dehann · 2020-08-11T09:53:38Z

MWE of the current issue:

using Unmarshal, JSON

struct MyType
  createdTimestamp::ZonedDateTime
end

str = "{\"createdTimestamp\":\"2020-08-11T04:05:59.82-04:00\"}"

interm = JSON.parse(str)
# Dict{String,Any} with 1 entry:
#   "createdTimestamp" => "2020-08-11T04:05:59.82-04:00"

Unmarshal.unmarshal(MyType, interm)

ERROR: ArgumentError: Unable to parse string "2020-08-11T04:05:59.82-04:00" using format dateformat"yyyy-mm-ddTHH:MM:SS.ssszzz". Unable to parse date time. Expected directive DatePart(sss) at char 21
Stacktrace:
 [1] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:104 [inlined]
 [2] tryparsenext_core(::String, ::Int64, ::Int64, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:38
 [3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:150 [inlined]
 [4] tryparsenext_internal at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:125 [inlined]
 [5] parse at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Dates/src/parse.jl:282 [inlined]
 [6] ZonedDateTime(::String, ::Dates.DateFormat{Symbol("yyyy-mm-ddTHH:MM:SS.ssszzz"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.DatePart{'z'}}}) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:88
 [7] ZonedDateTime(::String) at /home/dehann/.julia/packages/TimeZones/v0mfN/src/parse.jl:87
 [8] unmarshal(::Type{T} where T, ::String, ::Bool, ::Int64) at /home/dehann/.julia/packages/Unmarshal/QoOc6/src/Unmarshal.jl:38
 [9] unmarshal(::Type{T} where T, ::Dict{String,Any}, ::Bool, ::Int64) at /home/dehann/.julia/packages/Unmarshal/QoOc6/src/Unmarshal.jl:113
 [10] unmarshal(::Type{T} where T, ::Dict{String,Any}) at /home/dehann/.julia/packages/Unmarshal/QoOc6/src/Unmarshal.jl:85
 [11] top-level scope at REPL[10]:1

dehann · 2020-08-11T10:29:41Z

BUT, this works:

julia> str = "{\"createdTimestamp\":\"2020-08-11T04:05:59.820-04:00\"}"
"{\"createdTimestamp\":\"2020-08-11T04:05:59.820-04:00\"}"

julia> interm = JSON.parse(str)
Dict{String,Any} with 1 entry:
  "createdTimestamp" => "2020-08-11T04:05:59.820-04:00"

julia> Unmarshal.unmarshal(MyType, interm)
MyType(ZonedDateTime(2020, 8, 11, 4, 5, 59, 820, tz"UTC-04:00"))

So problem really is that when milliseconds are written with one or two characters rather than the expected 3: SS.ssszzz. Perhaps we can make sure there are always 3 characters for the millisecond portion.

dehann · 2020-08-11T10:43:57Z

Okay, lets try this, will add to PR #589 soon:

# Corrects any `::ZonedDateTime` fields of T in corresponding `interm::Dict` as `dateformat"yyyy-mm-ddTHH:MM:SS.ssszzz"`
function standardizeZDTString!(T, interm::Dict)
    @debug "About to look through types of" T
    for (name, typ) in zip(fieldnames(T), T.types)
        @debug "name=$name" 
        @debug "typ=$typ"
        if typ <: ZonedDateTime
            @debug "must ensure SS.ssszzz on $name :: $typ"
            namestr = string(name)
            supersec, subsec = split(interm[namestr], '.')
            sss, zzz = split(subsec, '-')
            @debug "split time elements are $sss-$zzz"
            # make sure milliseconds portion is precisely 3 characters long
            if length(sss) < 3
                # pad with zeros at the end
                while length(sss) < 3
                    sss *= "0"
                end
                newtimessszzz = supersec*"."*sss*"-"*zzz
                @debug "new time string: $newtimessszzz"
                # reassembled ZonedDateTime is put back in the dict
                interm[namestr] = newtimessszzz
            end
        end
    end
    nothing
end

dehann · 2020-08-11T11:09:41Z

Wait, Sam's is so much better:

    # Make sure that the timestamp is correctly formatted with subseconds
    packedProps["timestamp"] = replace(packedProps["timestamp"], r":(\d)(\d)(Z|z|\+|-)" => s":\1\2.000\3")

dehann · 2020-08-11T11:15:19Z

@debug "must ensure SS.ssszzz on $name :: $typ -- $(interm[namestr])"
# Make sure that the timestamp is correctly formatted with subseconds, copy #588
interm[namestr] = replace(interm[namestr], r":(\d)(\d)(Z|z|\+|-)" => s":\1\2.000\3")
@debug "after SS.ssszzz -- $(interm[namestr])"

That didnt work?

┌ Debug: must ensure SS.ssszzz on createdTimestamp :: TimeZones.ZonedDateTime -- 2020-08-11T04:05:59.82-04:00
└ @ DistributedFactorGraphs ~/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:44
┌ Debug: after SS.ssszzz -- 2020-08-11T04:05:59.82-04:00
└ @ DistributedFactorGraphs ~/.julia/dev/DistributedFactorGraphs/src/services/Serialization.jl:46
ERROR: ArgumentError: Unable to parse string "2020-08-11T04:05:59.82-04:00" using format dateformat"yyyy-mm-ddTHH:MM:SS.ssszzz". Unable to parse date time. Expected directive DatePart(sss) at char 21

dehann · 2020-08-11T11:29:58Z

Okay, I was able to load a dfg object again thanks. Numerics of the application are completely wrong here, but at least the graph is loading again -- thanks for the help!

Affie · 2020-08-11T14:02:43Z

I'm a bit confused about all the timestamp functions such as (#582 (comment)) was this not sufficient?

DistributedFactorGraphs.jl/src/services/Serialization.jl

Line 11 in 309932b

    
           props["timestamp"] = Dates.format(v.timestamp, "yyyy-mm-ddTHH:MM:SS.ssszzz")#string(v.timestamp)

However, I don't like the idea of manually having to functions to serialise simple types such as these. That's why the JSON3 comment. It's definitely worth a discussion to sort out properly.

dehann · 2020-08-11T14:19:23Z

Doesn't seem to work.

julia> str = "2020-08-11T04:05:59.82-04:00"
"2020-08-11T04:05:59.82-04:00"

julia> replace(str, r":(\d)(\d)(Z|z|\+|-)" => s":\1\2.000\3")
"2020-08-11T04:05:59.82-04:00"

dehann · 2020-08-11T14:20:49Z

we can use (refactor) the new standardizeZDTString(..) function to fill this purpose, and replace in #588

GearsAD · 2020-08-11T15:06:47Z

Didn’t see the functions to standardize the strings, I used a regex in #588 , if there’s a standard fix will move to that. Net takeaway is actually timezones.jl is way too strict on time stamp format

Affie · 2020-08-11T15:11:35Z

If it weren't for these time zones, we would have been in the same time zone and I could have told you about this time zone issue.

GearsAD · 2020-08-11T15:13:02Z

Hahaha!

dehann added bug Something isn't working FileDFG labels Aug 9, 2020

dehann added this to the v0.10.0 milestone Aug 9, 2020

dehann added the serialization label Aug 9, 2020

dehann mentioned this issue Aug 9, 2020

CloudGraphs DFG Updates #517

Merged

3 tasks

dehann changed the title ~~loadDFG! error, "type does not have a definite number of fields"~~ loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry Aug 11, 2020

dehann mentioned this issue Aug 11, 2020

Test fix for BlobStoreEntry #589

Merged

dehann closed this as completed Aug 11, 2020

dehann mentioned this issue Aug 11, 2020

Fix for non-millisecond timestamps #588

Merged

dehann mentioned this issue Aug 14, 2020

Compatibility of loadDFG! with new module naming? #618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry #582

loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry #582

dehann commented Aug 9, 2020 •

edited

Loading

Affie commented Aug 9, 2020 •

edited by dehann

Loading

dehann commented Aug 9, 2020 •

edited

Loading

Affie commented Aug 9, 2020

Affie commented Aug 9, 2020

dehann commented Aug 9, 2020

Affie commented Aug 9, 2020

dehann commented Aug 9, 2020 •

edited

Loading

dehann commented Aug 9, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

Affie commented Aug 11, 2020

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

Affie commented Aug 11, 2020

GearsAD commented Aug 11, 2020

loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry #582

loadDFG! error, "type does not have a definite number of fields", BlobStoreEntry #582

Comments

dehann commented Aug 9, 2020 • edited Loading

Affie commented Aug 9, 2020 • edited by dehann Loading

dehann commented Aug 9, 2020 • edited Loading

Affie commented Aug 9, 2020

Affie commented Aug 9, 2020

dehann commented Aug 9, 2020

Affie commented Aug 9, 2020

dehann commented Aug 9, 2020 • edited Loading

dehann commented Aug 9, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020 • edited Loading

Discussion

Proposed Fix

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

Affie commented Aug 11, 2020

dehann commented Aug 11, 2020 • edited Loading

dehann commented Aug 11, 2020

GearsAD commented Aug 11, 2020

Affie commented Aug 11, 2020

GearsAD commented Aug 11, 2020

dehann commented Aug 9, 2020 •

edited

Loading

Affie commented Aug 9, 2020 •

edited by dehann

Loading

dehann commented Aug 9, 2020 •

edited

Loading

dehann commented Aug 9, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

GearsAD commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading

dehann commented Aug 11, 2020 •

edited

Loading