Dashboard for multiple sites (potentially hundreds of websites)

[mattab]

Some Piwik users add a lot of websites in Piwik.
It is also common that users of a given Piwik installation can access more than 10 websites.
Currently they have to go to each of these websites to check their reports.

They need quick & easy way to get an overview of what’s happening in all their websites at once, and a quick access to more details.

Mockup

[[Image(http://issues.piwik.org/attachments/389/multi%20site%20report.PNG?format=raw)]]

Overview
- This page is space efficient but clear to display data about several websites
- gives an overview of what happened on all your websites, so you can choose which ones need investigation.
- display raw numbers
- and show the evolution of the metrics over the last days/weeks/month. via sparkline + red/green indicator
– this page links to each website report by a click on the website name
- possible to change the period of the dashboard: Day/Week/Month/Year
- each website is a simple row on the page
- for each website, print the
– website name
– nb of unique visitors
– nb of pages
– (keep space for nb of goal conversions – coming)
– a sparklines of the last 30 days/week/month/year for unique visitors
- when hovering the website name, the “drag icon” appears on the left of the website name. This icon makes draggable the list of websites so that each user can reorder their list of websites. By default website are ordered alphabetically. The website order in the list are saved for each user in the database (like the dashboard is), except for anonymous for whom the order is only stored in session (alternatively the feature can be disabled for anonymous user). The drag n drop feature would use Jquery UI draggable that we already use to reorder widgets in the dashboard.
- This dashboard can be accessed by a new entry “All websites” that always appears on the top of the “Website selector” on the top right of the page.

Speed
When loading 300 websites, we will lookup numerical values, for the period selected.

To show UU + visits + pages + conversions we need to load the 4 related numeric values for 300 websites. that’s 1 SELECT query to return 1200 values.

to show green/red evolution indicator we need to select the compared numeric value from yesterday/last week/last month/last year. that’s one SELECT to return 300 values.

loading 300 sparklines will take a loong time, as the browser will issue limited http request on the piwik domain. loading 300 sparklines will certainly take more than 20 seconds.
is that an issue?

Future releases
- adding sorting feature so that user can sort table by unique visitors, conversions, etc.
- adding search feature so that user can see only websites with names matching a string. the search box would have auto suggest feature.
- add possibility to choose the default page to arrive on – either the dashboard with all websites or the first available website in the list

Estimate

between 4 and 6 days of work
Keywords: multi-sites

[mattab]

Attachment:
[multi site report.PNG](http://issues.piwik.org/attachments/389/multi_site_report.PNG)

[zawadzinski]

Current version in the attachment.

Now working on core to optimize Archive/Array/*.php

[anonymous-matomo-user]

I’ve installed and activated this plugin but there is no new entry in the “Website selector” on the top right of the page. Only my sites are listed, but there still isn’t an extra one for All websites. I installed and activated it on two different servers, and neither shows the new selection in the drop-down.

[anonymous-matomo-user]

Aha! I figured this one out. The documentation is incorrect. The plugin does work, just not as it states above. Rather than placing a new entry for All sites in the website selector in the top right of the page, it instead places an extra menu item in the main blue horizontal menu at the end after Dashboard, Visitors, Actions, Referrers. The new item is called Sites Summary.

[anonymous-matomo-user]

I originally reported this problem on the forums because trac was down for me, but now I’ve come here. Here’s a summary of the bug below, but a larger version of the error is at the forum thread located at http://forum.piwik.org/index.php?showtopic=831. Neither this report nor the one at the forum includes the entire error because it is incredibly huge.

Since I upgraded to version 0.2.33, the multisites plugin has produced an error – the FIRST time I run it each day. After the first time, it’s fine (until the next day). Here’s an edited version of the error. The real version was ENORMOUS. I’ve also edited my domain info to keep it generic.

Notice: Undefined property: Piwik_DataTable_Simple::$rows in /home/myfolder/public_html/piwik/core/DataTable.php on line 457 Backtrace —>#0 Piwik_ErrorHandler(8, Undefined property: Piwik_DataTable_Simple::$rows, /home/myfolder/public_html/piwik/core/DataTable.php, 457, Array ()) called at [/home/myfolder/public_html/piwik/core/DataTable.php:457]…………

[mattab]

see also: add a line with total for all websites: http://www.karlkopp.com/blog/2009/4/30/piwik-updated-multisitesplugin

[mattab]

Please update to the new MultiSitesPlugin.tar for compatibility with 0.2.35.

[anonymous-matomo-user]

This pops up with piwik 0.2.35 (using the updated version above):

Fatal error: Unsupported operand types in /var/www/apps/content/piwik/plugins/MultiSitesPlugin/Controller.php on line 204

[zawadzinski]

Current version of MultiSitesPlugin. It still needs a lot of improvements and refactoring.

I am concerning rewriting the plugin to use directly piwik_archive_numeric_* tables in order to speed up the plugin (it is still very slow when even having "just" hundreds of sites).

If you got < 500 sites you can use this version (but remember to setup auto archiving http://piwik.org/docs/setup-auto-archiving/ - piwik won't be capable of archiving so many sites via web request!)

[zawadzinski]

The Problem.
There is no way to use SQL query result limiting (LIMIT) and sorting (ORDER BY) using Piwik API. This is very important for MultiSitesPlugin, because when having thousands of sites we do not want to fetch all results, and sort & limit them in PHP.

Propsed solution.
We can build MultiSitesPlugin_API that will query directly SQL fetching data from piwik_archive_numeric_* tables. We do not want to check if data for specific site is archived, so we assume that all data is archived using cron auto-archiving (users will be noticed about it when installing MultiSitesPlugin).

This solution will significantly increase MultiSitesPlugin performance.

[zawadzinski]

Attachment:
MultiSitesPlugin.zip

[mattab]

good stuff!
here is my feedback:

about the UI

• the sparklines html includes could be loaded directly, no need to show the rolling image (ie. we know how the call looks like); currently it does one http request to request each image HTML but jquery could write directly the right html.
• the numbers and text could be centered / aligned middle in the cell, and cells could be less high
• 1 - 12 z 12 is in polish?
• the CSS could reuse the existing data table CSS for UI consistency

thanks! it is getting real close to be included in core ;)

PS: we can include in core before the optimization happen on the php side, so please feel free to do the UI updates and then work on the php side if you wish to push this in core.

[anonymous-matomo-user]

Is this placement normal ?

[[Image(http://img229.imageshack.us/img229/4594/20062009083816.jpg)]]

If anyone know what to change in the main CSS file...

[zawadzinski]

Attachment:
MultiSitesPlugin v0.2.zip

[zawadzinski]

New version in the attachment.

We use now Piwik's CSS styles, and plugin Piwik coding standards are followed.

[anonymous-matomo-user]

I'm sorry to say, the "button" in the menu is still "misplaced" :

(Firefox 3.0.11)

[[Image(http://img526.imageshack.us/img526/3176/09072009100750.jpg)]]

[zawadzinski]

Its caused by translation (that is too long and does not fit in the button), it is not this plugin related issue.

[anonymous-matomo-user]

Switching to english i still get this :

[[Image(http://img200.imageshack.us/img200/1931/10072009151203.jpg)]]

The submenu is always present no matter what language.

[mattab]

the location in the menu is temporary and will be fixed before submitting the plugin to the core - most likely it will appear as "All Websites" next to Dashboard, top left of screen.

[robocoder]

• For consistency, I'd suggest dropping the "Plugin" suffix from the plugin name.
• Given the autoloader, the require_once's can be removed from Controller.php and MultiSitesPlugin.php.

[anonymous-matomo-user]

A few issues I have on this plugin. Firstly, it forces the php process to use 100% of the CPU for a long period of time (around 60 seconds - which would match the max process time in php.ini). It also throws a javascript error:

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 4.0.20506; Tablet PC 2.0;)
Timestamp: Wed, 15 Jul 2009 00:33:50 UTC

Message: Unknown runtime error
Line: 124348672
Char: 3
Code: 0
URI: http://example.com/piwik/index.php?module=CoreHome&action=index&idSite=2&period=day&date=2009-07-15

Message: Unknown runtime error
Line: 124348672
Char: 3
Code: 0
URI: http://example.com/piwik/index.php?module=CoreHome&action=index&idSite=2&period=day&date=2009-07-15

[anonymous-matomo-user]

this plugin can't work properly on my IE browser(can't list any website),on chrome and firefox it works well,is it a bug or I've done something wrong?

[mattab]

Once the few bugs listed here are fixed, we would like to commit it to trunk asap after Piwik 0.6. Maciej, would you mind taking a look at the reported issues? thanks!

[pebosi]

Attachment:
MultiSites.zip

[pebosi]

removed the suffix and the require_once calls. added a link to topbar (see included patch).

changed some stuff in index.tpl, but i was just testing, so there's a problem when changing date or period.

regards pebosi

[mattab]

Pebosi, can you please clarify what the issues are? What is missing for the plugin to be included in 0.5.1 in a stable state?
Thank you!

[pebosi]

When changing the date the parameter "date" in url is not replace but it's appended to the url without an ampersand. so there's an error...
i tried using no ajax so i think there's a problem with the broadcast.

regards

[pebosi]

Attachment:
MultiSites.2.zip

[robocoder]

In [1678], refs #389, commit MultiSites plugin

[pebosi]

Hi,

bugs in the last commit [1678]:

• Site selector is not hidden
• date change is not affected

regards pebosi

[mattab]

(In [1692]) refs #389 Fixing date selection, fixing column sorting, few styles modifications

[mattab]

(In [1696]) MultiSites plugin enabled by default refs #389

[mattab]

(In [1701]) refs #389 using the jquery magic so that IE works OK

[mattab]

Included and enabled by default in 0.5.3

Next steps are performance improvements, see #1077

[julienmoumne]

(In [6478]) fixes #2708

refs #5491

• PDFReports major refactoring. Any plugin can now add new kinds of reports. Required for Send reports via SMS #2708 and Report Publisher #3118.
• test report functionality ($idReport == 0) dropped in Piwik_PDFReports_API->generateReport()
• All Websites report shows 3 more metrics: Goal Conversions, eCommerce Conversions and eCommerce Revenue. Can be removed if asked to.
• Piwik_PDFReports_API->sendEmailReport renamed to sendReport
• All Piwik_PDFReports_API method signatures updated to support generic report parameters
  refs Dashboard for multiple sites (potentially hundreds of websites) #389
• new API method to retrieve only one Piwik site : Piwik_MultiSites_API->getOne()
• per Send reports via SMS #2708 description, Piwik_MultiSites_API methods now support a new parameter named enhanced. When activated, Goal Conversions, eCommerce Conversions and eCommerce Revenue along with their evolution will be included in the API output.
• API metrics refactored in (@ignored)Piwik_MultiSites_API->getApiMetrics()
• Metadata now returns 12 metrics : nb_visits, visits_evolution, nb_actions, actions_evolution, revenu, revenue_evolution, nb_conversions, nb_conversions_evolution, orders, orders_evolution, ecommerce_revenue, ecommerce_revenue_evolution
  refs Report Publisher #3118
• ReportPublisher plugin could now easily be implemented
  commits merged
• r6243
• r6422 (Scheduled report by email with custom date range triggers error #3012)
  TODO
• the MobileMessaging settings page may need some embellishment
• @review annotations need some attention
• test if the MultiSites API evolutions have some impact on Piwik Mobile and other client code