Switching to closure tree #588

[mgharbik]

No description provided.

[cbrunsdon]

Interesting, less changes than I was expecting (though that one test failure seems legit).

It would be nice to get some benchmarks in some different situations before/after the change.

Something like $X inserts into the children of a taxon, maybe a few of the common queries with $Y taxons. I don't fully understand the plusses and minuses of the change here.

[mgharbik]

Hi @cbrunsdon, I am working on some benchmarks, I'll get back to you as soon as possible.

[mgharbik]

Hi again,

At some points, we were thinking maybe we would make tests for trees which are large (heigh number of nodes in each parent) as well as the profound ones (heigh number of levels). But I guess the stores would never have profound trees, so we skipped this case. Moreover, we tried also Adjacency List Structure that it would give good results since it does not have any extra table, neither there is no need to calculate lft and rgt.

The following results are all made under a 27_931 nodes tree, 4 levels and each parent produces 30 nodes.

1. Generating the tree:

######################## AWESOME NESTED SET ########################
#                              user     system      total        real
# Generating the tree   341.430000  12.840000 354.270000 (765.117659) 12.75196098333333 min

######################## CLOSURE TREE ###############################
#                              user     system      total        real
# Generating the tree   249.890000   9.390000 259.280000 (367.055578) 6.11759296666667 min

######################## Adjacency List #############################
#                              user     system      total        real
# Generating the tree   691.820000  10.810000 702.630000 (900.876022) 15.01460036666667 min

While generating the whole tree, we see that closure tree is a clear winner.

1. Displaying the whole tree:

##################### AWESOME NESTED SET #####################
#                       user     system      total        real
# Displaying tree  15.830000   0.620000  16.450000 ( 17.845013)

##################### CLOSURE TREE ###########################
#                       user     system      total        real
# Displaying tree  20.530000   0.640000  21.170000 ( 22.585553)

##################### Adjacency List #########################
#                       user     system      total        real
# Displaying tree  39.620000   0.830000  40.450000 ( 42.714692)

As jstree was removed and now it must render the whole tree at once, we find this case is very significant. The winner here is nested set but closure tree still not bad. adjacency list takes almost double time.

1. Inserting a new taxon:

##################### AWESOME NESTED SET #####################
#                       user     system      total        real
# Inserting taxon  0.070000   0.000000   0.070000 (  0.118796)

##################### CLOSURE TREE ###########################
#                       user     system      total        real
# Inserting taxon  0.090000   0.020000   0.110000 (  0.133249)

##################### Adjacency List #########################
#                       user     system      total        real
# Inserting taxon  0.090000   0.010000   0.100000 (  0.109642)

Here the insertion is made in root at left. We took the average of 10 insertions approximately.

1. Move a taxon

################### AWESOME NESTED SET #####################
#                      user     system      total        real
# Moving a taxon  0.080000   0.010000   0.090000 (  1.653846)

################### CLOSURE TREE ###########################
#                      user     system      total        real
# Moving a taxon  0.120000   0.020000   0.140000 (  0.199836)

################### Adjacency List #########################
#                      user     system      total        real
# Moving a taxon  0.100000   0.010000   0.110000 (  0.203947)

Here we move the root's last descendant to the first child of root in the left. Average of 10 tries approximately.

1. Remove a taxon

################### AWESOME NESTED SET #####################
#                      user     system      total        real
# Removing a taxon  1.020000   0.060000   1.080000 (  3.554831)

################### CLOSURE TREE ###########################
#                      user     system      total        real
# Removing a taxon  0.090000   0.020000   0.110000 (  0.125421)

################### Adjacency List #########################
#                      user     system      total        real
# Removing a taxon  0.080000   0.010000   0.090000 (  0.101363)

Here we remove the first root child. Average of 10 deletions approximately.

Comparing the last 3 operations Insert/Move/Remove, nested set is slow in moving nodes and even slower in deleting. adjacency list is more effective in these operations but it is very slow when displaying the tree. In general, it seems to be closure tree performs better in these kind of trees, but we would like to hear your feedbacks about this issue. If you have any doubts please let us know!

With kind regards,

[mamhoff]

Wait, "overrides"? Is awesome_nested_set still around to be overridden, or does the method just follow the signature from awesome_nested_set?

Also: We tend to have comments above method definitions, along with a little bit of YARD doc. If you could move that once we've come to a decision about this PR, that'd be great.

[mgharbik]

@mamhoff you're right, not really overriding, it just follows the same logic as awesome_nested_set, which is totally removed.

[mgharbik]

@mamhoff I fixed the corresponding tests. Here I ordered the taxons by permalink, avoiding a complex inner join into the hierarchy table, but if you have any other approach please let me know!

[mamhoff]

Can you rebase this PR on current master, as well as fix the failing test? Once that's done, this PR will be up for review again.

[tvdeyen]

Did you run your benchmarks against the whole stack, so that rendering is also measured?

As we investigated in AlchemyCMS we found that rendering the tree is the slowest part. After switching to handlebars templates and only returning JSON from the controller we gained a ten fold speed up!!

Again, just by rendering in the frontend instead of iterating over rails partials. Alchemy still uses awesome_nested_set, because the speed gain was significant enough.

Reference:

AlchemyCMS/alchemy_cms#923

AlchemyCMS/alchemy_cms#894

[mgharbik]

@tvdeyen I didn't measure the templates rendering because I assumed that for solidus, the whole tree is retrieved once. As far as I understand it must work similar in both cases since all is done by the api controller, is it correct?

[tvdeyen]

Closing as stale. Please reopen if you think this is still something we should to tackle.