Audit use of _promote_op

[shashi]

This has served us well so far, but increasingly there is frustration when you end up with a table of a single Any column. We need to use the Base.map trick as much as possible.

[piever]

Two relevant links:

• #101 from IndexedTables and related discussion (using list comprehension is I think a common way to get the undesired column Any)
• For the future this discourse thread, where it seems that some care is required when using map with Missings (which we don't use in this package but we will IIUC). In particular nalimilan links to many improvements he's been doing to base Julia about this issues.

[shashi]

Related to your second point is a behavior in join where we speculatively create DataValue columns when doing an outerjoin lest the right table doesn't have some matches.

[shashi]

Related to your second point is a behavior in join where we speculatively create DataValue columns when doing an outerjoin lest the right table doesn't have some matches.

[piever]

There's one more remark. Issue #110 is also probably caused by this. In general I think a drawback of the current design is that it's difficult to use groupby in cases where the output of f has many fields (which could be generated programmatically). One side of the issue is that it's difficult to generate a NamedTuple programmatically (this problem will go away in Julia 0.7). The other side is that the type of these NamedTuples won't change from one row to the next but I don't think it will be inferrable.

For these cases (f returns many fields, generated programmatically / a bit like what summarize does as well as the issue linked above) I'm afraid the only solution for now is to create a one line table and flatten.

[piever]

This is basically fixed in IndexedTables (after JuliaData/IndexedTables.jl#226 is merged). There are still a couple of uses in JuliaDB though:

• https://github.com/JuliaComputing/JuliaDB.jl/blob/master/src/join.jl#L174
• https://github.com/JuliaComputing/JuliaDB.jl/blob/master/src/join.jl#L335

But I don't know the distributed code well enough to know if it's feasible to remove those.

[piever]

Closing as now _promote_op is not used in either JuliaDB or IndexedTables

[joshday]

Very nice work @piever!

[shashi]

Yeah, amazing amounts of deft work! 👏