make pick more useful

[shashi]

julia> c = Columns(x=[1,2],y=[2.,4])
2-element IndexedTables.Columns{NamedTuples._NT_x_y{Int64,Float64},NamedTuples._NT_x_y{Array{Int64,1},Array{Float64,1}}}:
 (x = 1, y = 2.0)
 (x = 2, y = 4.0)
julia> pick(:y)(c)
2-element Array{Float64,1}:
 2.0
 4.0

julia> pick(:y)(c)
2-element Array{Float64,1}:
 2.0
 4.0

julia> pick((:y,))(c)
2-element IndexedTables.Columns{NamedTuples._NT_y{Float64},NamedTuples._NT_y{Array{Float64,1}}}:
 (y = 2.0)
 (y = 4.0)

julia> pick((:y,:x))(c)
2-element IndexedTables.Columns{NamedTuples._NT_y_x{Float64,Int64},NamedTuples._NT_y_x{Array{Float64,1},Array{Int64,1}}}:
 (y = 2.0, x = 1)
 (y = 4.0, x = 2)

julia> pick((2,1))(c)
2-element IndexedTables.Columns{NamedTuples._NT_y_x{Float64,Int64},NamedTuples._NT_y_x{Array{Float64,1},Array{Int64,1}}}:
 (y = 2.0, x = 1)
 (y = 4.0, x = 2)

The multiple column version is not inferable in the case the columns are of different types. That can be fixed by using Base.@pure if required.

[JeffBezanson]

If the case of manually writing out the desired fields is important, we could do it very efficiently with a macro @pick(a,b) that expands to

x -> @NT(a=x.a, b=x.b)

To extend that to Columns, we could instead expand to

x -> wrap(x, @NT(a=x.a, b=x.b))

where

wrap(x, y) = y
wrap(x::Columns, y) = Columns(y)

or something along those lines.

[shashi]

@pick

That's a neat idea! Gives us a chance to generate the required output type during macro expansion time (esp. since it's not possible with @generated functions).

We specialize map(p::Proj, c::Columns) to call p only on c.columns, this is pretty fast at the moment for this reason.

[shashi]

I've updated the PR. Added @pick with exactly the syntax you suggested. I've retained the old pick which has a slightly different behavior. Specifically, pick(:x)(@NT(x=1, y=2)) == 1, but @pick(x)(@NT(x=1, y=2)) == @NT(x=1).

[shashi]

Tests now pass except on OSX nightly (same problem of missing native library or something).

[shashi]

This line fails in a multi-process situation because the symbol $T only gets defined on the master process... This can be fixed by adding @everywhere to the previous line, but do we want to do such things during macro expansion??

[JeffBezanson]

If the new type gets eval'd in Main then it will be sent to other processes, which would be good for now. Related to JuliaData/NamedTuples.jl#27 (comment) --- that change to NamedTuples might help fix things like this more comprehensively.

[shashi]

it will be sent to other processes

I'm trying to make this change to NamedTuples but it doesn't look like we can send new types on release-0.6 branch of Julia:

> jl -p 1

julia> struct Foo
           x
       end

julia> remotecall_fetch(identity, 2, Foo("x"))
ERROR: On worker 2:
UndefVarError: Foo not defined
deserialize_datatype at ./serialize.jl:861
handle_deserialize at ./serialize.jl:601
deserialize at ./serialize.jl:571
ntuple at ./tuple.jl:108
handle_deserialize at ./serialize.jl:592
deserialize_msg at ./distributed/messages.jl:100
deserialize_msg at ./distributed/messages.jl:110
message_handler_loop at ./distributed/process_messages.jl:161
process_tcp_streams at ./distributed/process_messages.jl:118

[amitmurthy]

Doesn't https://github.com/JuliaLang/julia/blob/50c2a378ba2171250b9bee04d4ea63eb1f348932/base/serialize.jl#L487-L504 only work for functions? I don't think we ever sent type definitions automatically.

[JeffBezanson]

Oops, you're right. We should actually try to rely on the serialize method we've added for NamedTuple types (which is in JuliaDB, but should be moved to NamedTuples).

It would also be nice if this macro could do less work. Ideally it should expand to a @NT(...) expression, i.e. expand to exactly what you'd write if the macro didn't exist, instead of using NamedTuple internals.

To make all that work, we could have NamedTuples put the type it creates directly into its macro output, instead of generating a GlobalRef.

[shashi]

I was using @NT but running into some problem I can't remember now, but it was about creating the correct expression. I guess I needed to use Expr(:macrocall,...) instead of :(@NT(...)), will try again, thanks.

type it creates directly into its macro output

Can we use @NT inside a function then?

[JeffBezanson]

Yes, it should fix that. I'll work on a NamedTuples PR.

[JeffBezanson]

I find this definition very confusing. A Proj{f}() is supposed to be a function that selects field f, period. Anything else should be a different type.

Also I realize the definition for (p::Proj)(c::Columns) isn't really right anymore, and should be a method of broadcast now.

[shashi]

Okay, I can separate this out into a ProjFunction type. Right now I use Val{f} for the case you mentioned above.

(p::Proj)(c::Columns) can be removed.

should be a method of broadcast now.

And map too right?

[JeffBezanson]

Yes.

[shashi]

Updated. Works in the multi-process case with JuliaData/NamedTuples.jl#34 , but safe to merge here as-is.

[codecov-io]

Codecov Report

Merging #44 into master will increase coverage by 0.78%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #44      +/-   ##
==========================================
+ Coverage   91.98%   92.77%   +0.78%     
==========================================
  Files           6        6              
  Lines         836      844       +8     
==========================================
+ Hits          769      783      +14     
+ Misses         67       61       -6

Impacted Files	Coverage Δ
src/utils.jl	98.19% <100%> (+4.02%)	⬆️
src/columns.jl	94.57% <100%> (+1.55%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 063d41f...25c1b62. Read the comment docs.