[Dashboard] Improve ordering by size performance

[ethervoid]

Context

Ordering datasets by size take a huge amount of time to apply

Steps to Reproduce

1. You need an account with hundreds/thousands of datasets
2. Click in order by size
3. Wait until it finishes

Current Result

Now it takes, in my test account, either ~47seconds or a timeout error

Expected result

Reduce that amount of time considerably

// @alonsogarciapablo

[ethervoid]

Here are some examples of the time spent calling the endpoint with order by size:

Timeout

Finished but after ~47 seconds

[ethervoid]

Update

I've made a couple of PoC to check if there is any improvement:

• I've made a new function in our PostgreSQL extension that I'm going to use in order to gather all the tables information
• I've changed the orderer class to include a call to that new function instead of using the .size attribute which makes a call to the user database in order to retrieve the size of the table.

The results is the following:

Current performance

Improved performance

As you can see there is a huge improvement using that new function, a magnitude order with accounts having a great number of tables.

What's next?

Well, there is a lot to do:

Be sure if we want to go on with this improvement because we need to spend more time here and implies multiples parts (@alonsogarciapablo / @javitonino ). If we decide to go on we have to:

• Finish the PostgreSQL function, there are some problems yet, fix the tests, test it and put into production
• In the rails app part, we have to figure out how to include this new way of order some types (size, mapviews, etc) and possibly made a refactor in our query_orderer

[alonsogarciapablo]

WOW. That looks like a huge improvement @ethervoid! Does it only help with "ordering by size" or with retrieving maps / datasets in general?

[ethervoid]

@alonsogarciapablo this would help in the ordering part for calls to the api/viz endpoint. In our dashboard is done by the order part but if we ask for visualizations orderer by size this improvement applies.

What do you mean for retrieving maps/datasets?

[alonsogarciapablo]

I was basically asking if this will only help when ordering by size, or also in other calls to /viz (eg: when loading latests maps and datasets for the home page).

[ethervoid]

This will only help for the ordering part

[alonsogarciapablo]

Hey @ethervoid! How is this going? Are we still working on it? Thank you! 🙇

[javitonino]

This has been stopped. There were still a lot of work to do and there is no backenders working on the dashboard as of now.

[alonsogarciapablo]

Closing for now. Let's reopen is this becomes an issue.