FlatBuffers.jl is writing unreadable flatbuffers

[ExpandingMan]

Alright, no clue what's happening here:

I'm trying to write

table Field {
  /// Name is not required, in i.e. a List
  name: string;

  /// Whether or not this field can contain nulls. Should be true in general.
  nullable: bool;

  /// This is the type of the decoded value if the field is dictionary encoded.
  type: Type;

  /// Present only if the field is dictionary encoded.
  dictionary: DictionaryEncoding;

  /// children apply only to nested data types like Struct, List and Union. For
  /// primitive types children will have length 0.
  children: [ Field ];

  /// User-defined metadata
  custom_metadata: [ KeyValue ];
}

which I've written in Julia as

@with_kw mutable struct Field
    name::String = ""
    nullable::Bool = false
    dtype_type::UInt8 = 0
    dtype::DType = nothing
    dictionary::Union{DictionaryEncoding,Nothing} = nothing
    children::Vector{Field} = []
    custom_metadata::Vector{KeyValue} = []
end
@ALIGN(Field, 1)
FB.slot_offsets(::Type{Field}) = UInt32[4,6,8,10,12,14,16]

specifically, I'm writing

Arrow.Metadata.Field
  name: String "col1"
  nullable: Bool true
  dtype_type: UInt8 0x02
  dtype: Arrow.Metadata.Int_
  dictionary: Nothing nothing
  children: Array{Arrow.Metadata.Field}((0,))
  custom_metadata: Array{Arrow.Metadata.KeyValue}((0,))

Note that both dictionary and custom_metadata are empty, so the definitions of these shouldn't matter.

Note that I've just read this in form a C++ source that presumably wrote it correctly. When I write it back however, I get something different than what I originally read.

What FlatBuffers.jl wrote was

julia> foreach(x -> print(" $x"), wtf)
 16 0 0 0 12 0 16 0 12 0 11 0 10 0 4 0 12 0 0 0 28 0 0 0 0 0 2 1 32 0 0 0 0 0 0 0 0 0 0 0 8 0 12 0 8 0 7 0 8 0 0 0 0 0 0 1 64 0 0 0 4 0 0 0 99 111 108 49 0 0 0 0

The original data was

julia> foreach(x -> print(" $x"), orig)
 16 0 0 0 0 0 10 0 12 0 6 0 5 0 8 0 10 0 0 0 0 1 3 0 12 0 0 0 8 0 8 0 0 0 4 0 8 0 0 0 4 0 0 0 1 0 0 0 20 0 0 0 16 0 20 0 8 0 6 0 7 0 12 0 0 0 16 0 16 0 0 0

FlatBuffers.jl was apparently able to read the original data correctly.

Note that I wrote wtf using FlatBuffers.bytes(Flatbuffers.build!(obj)).

I'm going to start digging into the format to see if I can figure out what in the world is going on here. Anyone have any ideas?

[ExpandingMan]

Alright, well I'm pretty sure I've figured out what the hell is happening, but I still have no idea why.

When a vector field (in particular children) is empty, the FlatBuffers.jl package doesn't write it at all, and it does not even appear in the vtable.

FlatBuffers.FlatBuffers.Table{Arrow.Metadata.Field}:
root offset: 0x0048
vtable start pos: 0x003c
vtable size: 12
0x00040 00 0c  [name]
0x00042 00 00  [nullable]
0x00044 00 0b  [dtype_type]
0x00046 00 04  [dtype]
0x00048 00 0c  [dictionary]
payload:
0x0004a 00 00 00 1c 00 00 00 00
0x00052 00 00 02 20 00 00 00 00
0x0005a 00 00 00 00 00 00 00 08
0x00062 00 0c 00 08 00 07 00 08
0x0006a 00 00 00 00 00 00 01 40
0x00072 00 00 00 04 00 00 00 63
0x0007a 6f 6c 31 00 00 00 00 ff
0x00082 ff ff ff 80 00 00 00 14
0x0008a 00 00 00 00 00 00 00 0c
0x00092 00 18 00 17 00 16 00 10
0x0009a 00 04 00 0c 00 00 00 18
0x000a2 00 00 00 00 00 00 00 00
0x000aa 00 00 00 18 00 00 00 00
0x000b2 00 03 03 00 00 00 00 00
0x000ba 00 0a 00 18 00 0c 00 08
0x000c2 00 04 00 0a 00 00 00 14
0x000ca 00 00 00 28 00 00 00 03
0x000d2 00 00 00 00 00 00 00 00
0x000da 00 00 00 01 00 00 00 00
0x000e2 00 00 00 00 00 00 00 18
0x000ea 00 00 00 00 00 00 00 00
0x000f2 00 00 00 01 00 00 00 03
0x000fa 00 00 00 00 00 00 00 00
0x00102 00 00 00 00 00 00 00 09
0x0010a 00 00 00 00 00 00 00 03
0x00112 00 00 00 00 00 00 00 05
0x0011a 00 00 00 00 00 00 00 ff
0x00122 ff ff ff 00 00 00 00

However, the table written by the C++ arrow implementation (which gets read to exactly the same object by FlatBuffers.jl is

FlatBuffers.FlatBuffers.Table{Arrow.Metadata.Field}:
root offset: 0x0044
vtable start pos: 0x0034
vtable size: 16
0x00038 00 08  [name]
0x0003a 00 06  [nullable]
0x0003c 00 07  [dtype_type]
0x0003e 00 0c  [dtype]
0x00040 00 00  [dictionary]
0x00042 00 10  [children]
0x00044 00 10  [custom_metadata]
payload:
0x00046 00 00 00 00 00 01 02 24
0x0004e 00 00 00 14 00 00 00 04
0x00056 00 00 00 00 00 00 00 08
0x0005e 00 0c 00 08 00 07 00 08
0x00066 00 00 00 00 00 00 01 40
0x0006e 00 00 00 04 00 00 00 63
0x00076 6f 6c 31 00 00 00 00 00
0x0007e 00 00 00 ff ff ff ff 88
0x00086 00 00 00 14 00 00 00 00
0x0008e 00 00 00 0c 00 16 00 06
0x00096 00 05 00 08 00 0c 00 0c
0x0009e 00 00 00 00 03 03 00 18
0x000a6 00 00 00 18 00 00 00 00
0x000ae 00 00 00 00 00 0a 00 18
0x000b6 00 0c 00 04 00 08 00 0a
0x000be 00 00 00 3c 00 00 00 10
0x000c6 00 00 00 03 00 00 00 00
0x000ce 00 00 00 00 00 00 00 02
0x000d6 00 00 00 00 00 00 00 00
0x000de 00 00 00 00 00 00 00 00
0x000e6 00 00 00 00 00 00 00 00
0x000ee 00 00 00 18 00 00 00 00
0x000f6 00 00 00 00 00 00 00 01
0x000fe 00 00 00 03 00 00 00 00
0x00106 00 00 00 00 00 00 00 00
0x0010e 00 00 00 09 00 00 00 00
0x00116 00 00 00 03 00 00 00 00
0x0011e 00 00 00 05 00 00 00 00
0x00126 00 00 00 ff ff ff ff 00
0x0012e 00 00 00

I'm still holding out a little hope that this is just a matter of dumb me having defined the schema wrong... but so far I still can't find how that can be.

[ExpandingMan]

Ok, another update (as much for my own clarification as public record keeping), I'm now pretty sure that what's happening is that sometimes FlatBuffers.jl neglects to put entries in the vtable for empty vectors. I honestly can't even tell if this is permitted in the "standard" because the flatbuffers documentation is so poor, but this doesn't seem to pose any problems for FlatBuffers.jl, but sometimes other flatbuffers readers seem to be unable to deal with this (although in all the cases, errors seem to be explicitly thrown, so I'm not sure if it's a real incompatibility or a check).

I still don't understand under what circumstances FlatBuffers.jl does this. It seems it can only happen in cases where the vector is empty (else it would clearly be an error), but sometimes I see it write them in the vtable, sometimes not.

I believe what's happening here is a bug (or oversight) in this block, but I'm still digging through it trying to figure out where the vectors get written (or fail to get written).

[ExpandingMan]

Ok, I have a lot more detail on what's happening, but still don't know what's causing it. The issue is definitely that defaults for Vector fields are getting written strangely.

For example, if you have a

@with_kw struct A
    v::Union{Vector{A}, Nothing} = nothing
end

create an instance with empty (but non-nothing) v serialize it, then deserialize it, you will get back nothing instead. Other flatbuffers don't seem to like this but I'm still trying to figure it out.

[quinnj]

Catching up on this after a long delay.

dictionary::Union{DictionaryEncoding,Nothing} = nothing

I think this is the root of the problem, and not necessarily some bug in FlatBuffers.jl, though in a way it IS the problem because it's not requiring you to use a verified flatc compiler to emit the correct code.

The problem here is that you're saying this field will be either a DictionaryEncoding OR A Nothing, which in flatbuffers world, translates to TWO fields: a v_type field that stores a UInt8 for whether v is a DictionaryEncoding or a Nothing, then v is one of those two things. The other issue is that non-scalar fields are not allowed to have default values, but default to "NULL". Moreover, unions are supposed to be declared separately as standalone types and aren't allowed "inline" like this in proper structs. Again, none of this is really your fault, but part of the dangerous world we live in without relying on the flatc compiler to generate correct definitions for us.

Now, none of that necessarily explains the low-level issues you've been investigating, but in short, I would guess that FlatBuffers.jl is serializing it as a Union type which is confusing other languages. But when another language serializes it as "NULL", Julia is being clever and decoding it as nothing.