Row Evolution: historical view of one or many specific rows in all Piwik reports!

[mattab]

Show graph of visits/pages over the last 30 days, for a specific row (keyword, country, etc.)

The plugin would add, in all data tables, an icon next to the keyword/ country / page view (or on hover), that would ajax load an evolution graph of the number of visits for this keyword, country or page, over the last 30 days/weeks/month. The graph would have the ‘data export’ links.

UI

We should find out how to provide this UI feature without bloating the UI with icons and new functionnality. Note that some rows in Piwik are already clickable (eg. to get the list of keyword by search engine).

One way of doing it would be to show a semi transparent icon when hovering over a row. The icon would be shown at the end of the row.

The new UI would also be used to report the new reports from #756

Report
The report would contain, for a specific keyword or search engine or country or page, the following information:
- visits in date range + sparkline
- unique visitors in date range + sparkline
- bounce rate + sparkline
- (for pages) time per page, exit rate, etc. + sparkline

Also, we should report Goals conversions, maybe in a different report, available by a click on a tab, or some other mechanism.
This report would show, for this keyword/country/page, the following info:
- overall conversions / conversion rate / revenue per visit
- for each goal…
– conversions / conversion rate/ revenue per visit.

Performance note

Piwik stores data on a “per day” basis. When plotting number of page views for the url ‘http://example.org/category/photo’ for the last 30 days, it will require selecting data for the 30 days, and navigate through the tree to get the child node data (/photo) – this is all done automatically via filters. In this example, piwik would select 90 blobs of data, and unserialize, load in memory, grep, etc. This is a severe requirement and feasability should be studied before starting the implementation.

When plotting evolution of keyword “piwik” over the last 30 days, this would require 30 blobs of data to be selected/unserialize/loaded/etc.

Note

In the first version, only one data would be plottable. In future versions, we could think about plotting 3 or 4 keywords/countris on the same graph. How would this be available in the UI? Clicking on the icon could “enable” the line to be displayed on the graph, and a “reset” button, or a click on the icon, would disable this line. Requires #397.

Keywords: evolution, graph, multiple values, multi lines, plot, entity, UI pattern

[mattab]

No description provided.

[mattab]

This would make Piwik a much stronger Analytics tool.

[DaSchTour]

to stay on track from #508

[mattab]

For reports that have a limited number of rows, for example: #2031, #583, #536, #584

It would be very nice to easily be able to splot the trends of each of these values over time, without having to click on each value separately to see the graph.

The graph plotting history could be shown for all these values at once, maybe with multiple lines on a line plot?

[timo-bes]

Page URL, page title, country and keyword are the labels of the DataTables. We could add a parameter "label" to the API. If set, the API plugin filters the table by label. This way, the other plugins don't event need to know about it.

This filtering funcionality has to be available for other plugins as well, eg for Actions.getPageUrl. Maybe it can be implemented in the DataTable class itself. DataTable_Array has to implement a similar function which passes the request to the other tables.

The filtering should provide only one record. If a match is found, we can stop searching. This is especially important for nested DataTables.

If performance is poor (which I suspect), we could add some sort of index to the DataTables. It doesn't have to be super fast so we can pick a lightweight one.

[mattab]

The datatable related code must definitely be automatic for all reports, and work consistently across them without need for custom hacks for some reports.

The big challenge of this ticket is the UI of drawing a graph for 1 row, or several rows at once eg.

• compare performance of 2 key pages over the last 12 months
• compare performance of the current top 10 keywords over the last 12 months (10 lines on a line plot, confusing but can be useful!)

The UI should make it easy to plot one row, but also plot multiple. How could it look like? how would we "reset" to the previous table view? Where would the graph be displayed? What would be the possible interactions with the graph?

We could ask Marcin our designer to prepare pixel perfect mockup but before would need to come up with wireframe level ideas of such complicated-to-integrate-nicely UI element. Any idea please suggest :)

[timo-bes]

I've started implementing the filters for the API using the indexes that are already there on DataTables. The API gets a new parameter "label" and filters the result, if it's a DataTable or DataTable_Array. I also updated the recursive search code of the Actions plugin - works fine.

As for the UI, we could open a properties dialog (similar to right-click in windows) when a row is clicked. There, we can show the available actions like "open url", "show recent development" etc. This is more flexible than icons and clutters the UI less.

[mattab]

Proposal for below the graph mini dashboard

= Keyword "Analytics free" =

$SPARKLINE Visits: between 134 - 670 visits per day, +14% over the period
$SPARKLINE Ecommerce orders: between 13 - 44 Ecommerce orders per day, -22% over the period

etc. for all metrics, below, ordered by metrics usefulness

• Like visible on the screenshot, text for other metrics faded out for other metrics to highlight current one
• The first number shows the range of values over the period.
• The second number +14% shows the difference between the first and last day of period. It is green or red based on performance (reusable css styles from All websites dashboard)
• it will process the % evolution using last period, which can be partial (only first half of current month)... so % will go down and be RED often... maybe we could base it on the last finished period?
• % between -5 and +5 could be left black instead of colored red/green?
  • Not a big fan of right click for UI, since right click is very hard to access and most people don't expect to use it in web app.
  • Maybe we could display a graph icon on hover on each row, for example this one which is great. I suppose it is not that frequent that users hover on the rows for no reasons.

See attached proposal mockup, workflow could be:

• show icon + text "Plot" on hover on each row, should be big enough to be easily clickable, but should not take whole width since the row itself is sometimes clickable (eg. to show Custom Variables values for a given row of Custom Variable name)
• hover on the selector would make it light yellow or something,
• a click would open the new UI element with graphs etc.
• A click on the grey outside or pressing ESC closes the popover.
  Thoughts? very excited by this feature :)

[mattab]

Attachment: Mockup to show click to launch popover
mockup click on row proposal.png

[mattab]

In the popover, when showing

• stats for browsers, OS, countries, search engines, etc. the logo (found in the api result) could be displayed
  • in the title next to the $label
  • in the graph in the legend - but maybe icons are too wide for the graph legend?
• when showing search engines keywords, or page urls evolution, the $label could be linked to the actual search engine result page or page URL (as found in API response)

thoughts?

[timo-bes]

(In [5722]) refs #534, #2742 metadata api extensions: more/better translations, actionToLoadSubTables

[timo-bes]

(In [5723]) refs #534, #2742 integration tests for previous commit

[mattab]

(In [5737]) Remove exit_rate from common metrics Refs #534

[mattab]

(In [5739]) * Refs #534 Removing exit_rate from metadata API, it was added in the standard processed metrics causing other reports to display it with 0% values

• Refs Integration testing: list of ideas of improvement #1465 Leave underscores in test names just to keep tests more readable (even though it is not a best practise to have _ in class prefixes!!)

[timo-bes]

(In [5758]) refs #534 recursive label filter for the api

[timo-bes]

(In [5759]) refs #534 integration tests for label filter

[timo-bes]

The previous commits introduce a label filter. It works for all API methods and filters out a single row - even from a subtable.

This filter will be used to get statistics for a single row in row evolution. In JS, the label (or the parts of the label in a nested table) will be retrived from the data table dom-nodes and an appropriate API request will be made.

Please review this part.

[timo-bes]

(In [5760]) refs #534 label filter integration tests for range (=> DataTable_Array)

[timo-bes]

(In [5761]) refs #534 removed originalLabel (left over from an earlier version of row evolution, not needed anymore)

[timo-bes]

(In [5762]) refs #534 svn properties set on new file DataTableLabelFilter.php

[timo-bes]

(In [5767]) refs #534: introducing row actions for data tables

[timo-bes]

The previous commit does not change the UI yet becuase I didn't commit the row actions' HTML in the template. I'll do that as soon as the other row evolution code is ready (because the committed code triggers row evolution).

For now, we can discuss the way row actions work. I didn't add them to datatable.js because this code is already pretty long and complex. Instead, datatable.js contains code that loads row action objects based on DOM elements in the table. This way, we can add more actions to the HTML and the existing code will load new actions, which we then have to add to datatable_rowactions.js.

Sounds good?

[mattab]

@ezdesign Kuddos!! Review:

• Great that you put the datatable_rowactions.js in a separate file. As you said the datatable.js is already quite bloated, so that's very welcome..
• Do we really need to support the DataTable_Array use case? Because, if it is not going to be used by the UI, then there is no need to support the new DataTableLabelFilter (since no API consumer will need it?)
  If not needed by UI, we can remove the special Datatable_Array code and tests in [5760]
• in the following code

if (!plabel) {
    break;
}
labels.push(plabel);

I think the if() could actually maybe be triggered, because of this frequent bug: #1024
So in the case that a "row evolution" report cannot be drawn (data not found in parent table, etc.), it would be nice to test/check that the code fails gracefully (since this edge case will happen).

I will make a few minor modifications as well in the next few hours

[timo-bes]

The DataTable_Array case is used every time row evolution is displayed. We want to get statistics for the previous days at once, so that will be an array.

[mattab]

(In [5774]) Refs #534

• To ensure that the "label filtering" works on all types of labels including special characters "' <> etc. I refactored the SafeDecodeLabel filter function.
• Changed DataTableLabelFilter to use Piwik_API_ResponseBuilder directly rather than duplicating code
• Added few tests cases

[mattab]

One more code review note:

• would you mind adding a test for Referers Search -> Keywords which would test the case that different APIs are called for parents VS children.

Otherwise looks good, please go ahead with the rest of commit -- I'm excited! :)

[mattab]

(In [Truncation was not working anymore (the tooltip was showing the truncated text rather than full text), reverting part of 5767)

Refs #534

[timo-bes]

In ResponseBuilder, we get the label from $this->request. The problem is that the values are trimmed in Piwik_API_Request::getRequestArrayFromString().

The page titles reports has labels starting with a space and therefore the labels can't be found using the index of the DataTable. Can we remove the trimming in Piwik_API_Request?

[timo-bes]

In [5848] you reverted backing up the original label to the DOM node's data because it changed the truncation. I can't reproduce the problem and neither jquery.truncate nor jquery.tooltip read data('originalText'). Can you please investigate and tell me how to reproduce the problem?

We need to know the original text in order to send it to the label filter.

[timo-bes]

(In [5872]) refs #534 lib / graph updates

• external series toggles for jqplot
• row evolution toggle
• using preloaded DataTable for ViewDataTable
• new Date methods subWeek and subYear
• hiding all view icons of a graph
• template html fixes
• and some minor things

[mattab]

I did some debugging and noticed that it is not the fault of json_encode. When I echo the properly escaped JSON in Piwik_Visualization_Chart::render, it gets escaped a second time later. The problem is line 157 of Piwik_ViewDataTable_GenerateGraphHTML. It replaces " with ". Why?? This makes escaping quotes impossible.

Please try removing the escaping of single/double quotes at this line. If all graphs still work then it's good :) it probably should not be there anyway but worth confirming that nothing else breaks.

[timo-bes]

(In [6196]) refs #534

• Proper handling of quotes in labels
• Better popover title algorithm: don't show urls for search engines, remove protocol and www from URLs, show the entire path - not only the last component - in row evolution (like in multi row evolution)
• Graph padding: a bit more for small values, make sure %-axes don't go above 100%
• Series with only 0 are shown in row evolution now (they were excluded because the graph couldn't handle them - now it works and excluding them caused other problems)
• Fixing issue reported in the forum - row evolution now works on subtables with period=week
• +100% over the period is not shown anymore when the first value is 0 (i.e. there probably is no data for the first sub period)
• If a series starts with 0 values, the min/max algorithm ignores them
• JSON encoding fix for quotes

[mattab]

so good to see Row evolution commits :)

Feedback

• I don't know if it's a regression from last commit, but when I try to open row evolution, for many rows it does not show any data below the graph, but the graph show that there is data for the selected date.
  • it seems that maybe the problem is when there is only one data point (in the current selected date) and no data before hand. See screenshots http://issues.piwik.org/attachments/534/graph%20working%20but%20not%20sparklines.png and http://issues.piwik.org/attachments/534/graph%20working%20but%20not%20sparklines%20bis.png
• When looking at row evolution for a particular URL, it would be nice/consistent if the URL in the popover was linking to the actual URL (target=_blank) and showed the yellow icon
• Small detail: When report is Flattened, and label is long and truncated (eg. custom var name - value), hovering on the row to see the tooltip also displays a white spot where the Row evolution icon normally is, hidding some text

[mattab]

Attachment:
graph working but not sparklines.png

[mattab]

Attachment:
graph working but not sparklines bis.png

[mattab]

(In [Refs #534 Fixing popover when period=range. Removed the _GETperiod) since it is redundant

[mattab]

We are going to release Row Evolution Beta as part of the next Piwik release as it's incredibly useful and a must have!

I propose to move all suggestions, ideas, bug reports to another ticket and close this ticket as we reached the beta milestone.

OK to close this ticket, or is there some critical bug to be fixed before stable release?

[mattab]

Timo, thanks for all the bug fixes earlier... there are really not any bug I can find now :)

I created the follow up ticket for all other improvements for Row Evolution V2: #3158
Thank you for continuing all discussion over there :)

It would be good to fix the following 2 small bugs before stable release:

• graphs should not be clickable to change date
• Clicking on black background should close popover

[sgiehl]

(In [6383]) refs #534 close row evolution popover on click outside + cancel requested content if popover is closed before loading is finished

[sgiehl]

(In [6384]) refs #534 disable links in popover evolution charts to switch the date

[mattab]

Thanks so much Steve!

ROW EVOLUTION V1 done! :)

well done guys..

[mattab]

(In [6410]) Refs #534, Refs #3158

• Displaying logos when comparing multiple rows. It looks very nice for example comparing Firefox VS Chrome VS IE Vs Safari :)

[mattab]

in: [revert partial of 6114 Refs #534 - Fixes #3387 Double encoding issue. Thanks larsen for report!

[timo-bes]

(In [7179]) refs #534: bringing the new popover behavior to row evolution

• the popover parameter is used to remember the current state of row evolution
• Piwik_Popover is used for controlling the popover
• ajax helper methods are refactored from Transitions to piwikHelper and reused in row evolution
• simplified logic for remembering rows for multi row evolution
• applied code formatter (hence the whitespace changes)