Display a new row "Total" showing the sum of values for all DataTable columns

[mattab]

Since #57 we process in Metadata Reports, a new <reportTotal>

For example in this metadata api output

    <reportTotal>
        <nb_visits>3</nb_visits>
        <nb_uniq_visitors>2</nb_uniq_visitors>
        <nb_actions>13</nb_actions>
        <nb_conversions>3</nb_conversions>
        <bounce_count>0</bounce_count>
        <revenue>3121</revenue>
    </reportTotal>

For Actions.getPageUrl here is the total (source code):

    <reportTotal>
        <nb_hits>4</nb_hits>
        <entry_nb_visits>2</entry_nb_visits>
    </reportTotal>

The goal of this issue is to:

• Check/ensure that the <reportTotal> includes the total for all columns that can be summed
• Add a new Cog icon option to show the Report Total in a new row in the report
• By default we don't enable it currently, but we may enable it in the future

Feel free to add your feedback as a comment here!

[rivadav]

+1 for disabling it by default
+1 for putting it into settings under Cog icon

Perhaps we can also make it a global setting in Settings/Manage/User settings for users convenience.

Another idea - after turning totals on for the first time for specific widget, we can ask user if he wants to display it for all tables (and where to disable it if he changes his mind).

What do you think?

[mattab]

@rivadav Good ideas and suggestions! I moved ticket to Short term.

[tsteur]

fyi: looking into report totals calculator this currently only works for a few defined metrics that can be easily summed. It doesn't work for avg, min, max, or any processed metric etc. This won't be too easy to develop.

You'd think using sumRow might help like this (as it supports min, max, and custom aggregators): https://github.com/matomo-org/matomo/compare/5267?expand=1

but even this won't work in all cases. Eg many filters and processors wouldn't be applied, depending on the report you may even need to apply the queued filters first or later after summing all rows etc. Also it wouldn't process any processed metrics. for many reports it would likely work just like that but not for all and not in all cases depending on which features are used.

There may be a way by not using reportTotalCalculator but it means changing quite a lot of core code and not only introducing a summary row but also a total row. We would probably need to fetch this information directly when fetching the original archive when we build the data table basically eg in Archive::createDataTableFromArchive. It may require a lot of code change and even plugins to consider it. The "total" row would need to be part of the generic data table that reports fetch, otherwise they cannot modify it, or react to it etc. It would be many many days of work I'd say.

I can create a simple PR for it and maybe someone can have a look to see if the data is mostly correct? Note: Because it calculates totals also for averages, it will show on hover percentage values for pretty much all columns now. This would be fixed and removed so they only appear on those columns as defined before. The grey percentage values can only be calculated for values that can be summed.

Todo: It would also need lots of extra code in the visualization to not activate things like Row evolution etc. If the solution was built properly by doing this in archive creation as suggested, things like Row evolution would work for this likely. But it would also turn it into MANY days of work.

Note: Would need to be tested how this affects performance.

[mattab]

There may be a way by not using reportTotalCalculator but it means changing quite a lot of core code and not only introducing a summary row but also a total row. We would probably need to fetch this information directly when fetching the original archive when we build the data table basically eg in Archive::createDataTableFromArchive. It may require a lot of code change and even plugins to consider it. The "total" row would need to be part of the generic data table that reports fetch, otherwise they cannot modify it, or react to it etc. It would be many many days of work I'd say.

This solution sounds good. Also for tables that have sub-tables, then the only way to get totals across all sub-tables is to process them at archiving times as you explain here?

So moving this out of 3.7.0 for now as it's a bit too complicated maybe?

[tsteur]

Nope, I meant it needs to fetch the dataTable that is being fetched in ReportsTotalsCalculator directly during Archive:: createDataTableFromArchive maybe. There is a PR for the first solution and it might work #13555

[tsteur]

Implemented feature in #13555

FYI: I didn't add a global setting to enable/disable it and neither ask on activation whether to enable it for all reports. That would be a new generic feature. Also you would kind of need to ask when disabling it for one report, whether to disable it for all reports otherwise you need to manually disable it everywhere etc.

[diosmosis]

@tsteur this issue is done right? Or was there a requirement not fulfilled by #13555?

[tsteur]

Should be all done 👍