Introducing higher-order functions on columns

[kspangsege]

Note: This is a work in progress.

The ambition is to provide a set of higher-order functions that combines extensibility with efficiency.

For example, here is how you could compute the variance in an integer column using Knuth's online algorithm (if that is your favourite version):

size_t n = 0;
double mean = 0;
double m2 = 0;
my_table.foreach_int(column_ndx, [&](int v) {
    n = n + 1;
    double delta = v - mean;
    mean += delta/n;
    m2 += delta * (v-mean);
});
variance = m2 / (n-1);

Note: This already works. It would also have worked for a floating point column.

Note: This is actually a good algorithm, since it is both online (one pass) and numerically stable. It also produces the mean (or average) for free.

Note also that because the function is passed as argument, it does not suffer from the penalty of regular iteration. Also, the specified function can in general be inlined inside a low-level loop.

Another relished higher order function is 'fold'. Here is how you could compute the square sum of a double column using 'fold left':

double sq_sum = my_table.foldl_double(column_ndx, [](double a, double v) {
    return a + v*v;
}, 0.0);

Note: foldl_string() could be used, for example, to compute the size of the largest string in a string column.

I already use these functions in the tightdb_tools repository for the SQL-like prompt.

I plan to also provide:

my_table.map_int(column_ndx, [](int v) { return roof-v; });
my_table.find_double(column_ndx, [](double v) { return sin(v) < -0.5; });

I also plan to provide multi-column versions - somehow.

I realize that these functions have substantial overlap with already provided functionality, however, these higher-order functions will provide much more flexibility for the customer.

/cc @astigsen @kneth @rrrlasse @bjchrist

[kspangsege]

Comment from astigsen:

Very cool. A lot of programmers from a functional background will love this :-)

A few questions/thoughts:

• I assume you will make it available through the typed interface somewhat like this: my_table.column().age.map([](int v) { return roof-v; });
• Do they work with old style predicates as well?
• Have you thought about how to expose them to other language bindings?
• How about doing this lazily on top of a query?

[astigsen]

When it can be done on entire rows at a time, it obviously opens up wide possibilities. But have you thought of any special use cases for this, when single column only?

I will assume that we end up implementing most statistical operations on columns as intrinsics, so that they don't have to unpack values and can do SSE optimizations. What kind of operations (on columns), that we would not be likely to implement, do you imagine users could be interested in?

[jenkins-tightdb]

Test FAILed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/17/

[jenkins-tightdb]

Test FAILed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/93/

[jenkins-tightdb]

Test FAILed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/94/

[jenkins-tightdb]

Test PASSed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/100/

[jenkins-tightdb]

Test PASSed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/103/

[timanglade]

@kspangsege Is this still WIP? Did we abandon the idea?

[kspangsege]

It is still WIP. As far as I know, we have not abandoned it [scarry thought
:-)].

On Tue, Feb 4, 2014 at 6:55 AM, Tim Anglade [email protected]:

@kspangsege https://github.com/kspangsege Is this still WIP? Did we
abandon the idea?

Reply to this email directly or view it on GitHubhttps://github.com//pull/53#issuecomment-34033014
.

[jenkins-tightdb]

Test PASSed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pullreqs/193/

[jenkins-tightdb]

Test FAILed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pr/13/

[jenkins-tightdb]

Test FAILed.
Refer to this link for build results: https://ci.tightdb.com/job/core_pr/24/

[oleks]

I'm excited to see this merged into core!

I like the name "reduce" instead of "foldl" if we are not going to have "foldr" as well.

BTW, this is more formally called "second-order functions".

[oleks]

Uh! Filter would be nice as well.

[jedelbo]

Abandonned