Send reports via SMS

[julienmoumne]

The idea is to send reports via SMS using online SMS providers. Users need to provide one or several phone numbers and a SMS authorization token.

Reports

So far, we have identified three different kinds of reports. Here are the candidate templates :

• Standard Report for 1 website:

from=Report Virtual-drums.com Daily
text=Yesterday, 150 visits (+15%), 555 pages (+22%), Goals: X conversions (-XX%), Revenue: xx (-xx%).

• Report "All websites"

from=Report Virtual-drums.com Daily
text=Yesterday (Jan 21st): virtual-drums.com: 150 visits (+15%), 555 pages (+22%), Goals: X conversions (-XX%), Revenue: xx (-xx%). forum.piwik.org: 600 visits (-12%), 32 pages (+2%), mamamia.fr: 330 visits (+1%), 555 pages (+22%), Ecommerce Revenue: 34$ (-60%) from 45 orders (+111%)

• Information displayed for all websites: Visits, Pages
• Information displayed for all websites that have at least one conversion: Conversions, Revenue (if Revenue > 0)
• Information displayed for all website with Ecommerce enabled: Orders, Revenue
• API to use:
  • When "All Websites" report is selected: The API MultiSites.getAll can be used for these reports
  • It needs to be enhanced to also return the number of Conversions, and Ecommerce orders.
  • To not add overhead for the "All websites" dashboard, maybe we could add a new parameter to MultiSites.getAll called "enhanced", 0 by default. When enhanced==1, we would enrich the returning data with Conversions/Ecommerce orders?
  • When only "Standard summary report" is selected: Could we also use the MultiSites API, but implement it so it works for one website only?
  • The advantage of using this API is that it already returns the % evolution so that the SMS reports would not have to process any data
  • the implementation code for a new MultiSites.get could be shared with the existing MultiSites.getAll

Desirable features

• The periodicity of SMS reports is configurable the same way as for HTML and PDF reports.
• SMS reports can be downloaded for preview the same way as for HTML and PDF reports.

SMS Authorization Token and Account Management

Online SMS providers with open APIs provide a SMS authorization token to registered users.

Within Piwik, this token will be manageable one of two ways :

• One token by Piwik instance

In this case, it is the responsibility of the super user to provide the token in the SMS management user interface.

It is the only user who has the right to consult the number of remaining credits.

• One token by user

In this case, it is the responsibility of each user to provide the token.

Each user has access to the number of remaining credits.

Deciding between the two modes of management is done by the super user with the following options in the SMS management UI :

- (x) Allow other users to create and manage their own SMS account.
- ( ) Allow users with view or admin access to send SMS reports using your SMS credit.

The second option is the default.

When a token is typed-in by the user, a call to the SMS API will be performed to validate the token and display the amount of remaining credit. The token will not be stored if it can't be validated.

The user will have the ability to display the number of remaining credits in the SMS management pages. A message will be displayed in the report and SMS management pages when no credit are left in the account.

The token can not be modified. To change an already validated token the user will have to remove it and add a new one.

If the token is not provided

• In the SMS management page

When the token is managed by the super user and a standard user accesses the SMS management page, display "SMS reports are not properly configured, please contact your administrator".

Phone Number

Within the SMS management UI, each user can manage one or several phone numbers. A phone number will be selectable within the report creation and edition forms only if it has been validated. To validate a phone number, a token is sent via SMS after requesting it in the SMS management page. This token is then typed-in by the user.

Here is the candidate SMS validation template :

Code is 54321. To validate your phone number and receive Piwik SMS reports please copy this code in the form accessible via Piwik > Settings > SMS.

The validation token does not expire.

Phone numbers can not be modified. To change an already validated number the user will have to remove it and add a new one.

The UI will provide a list of countries to automatically assign the international extension number. The country corresponding to the current language will be selected as the default value.

The phone number input field accepts only numbers.

A report can be created and edited without assigning phone numbers. The report list will display a warning message and the user won't be able to send the report.

In the report page, when the user has not validated at least one number :

SMS reports are still editable, the form can be saved without a selected phone number but the following message will be displayed in the edit form: "The list of phone numbers is empty because you did not activate a phone number". A link will be provided for quick access.

SMS Management Page

The SMS management page will be added in "Settings".

Technical aspects

• This SMS report feature will be developed as a new plugin called MobileMessaging. It will be included in core but disabled by default.
• The code managing the SMS authorization token and phone numbers should be reusable by other features such as SMS alerts. Therefore, the MobileMessaging plugin should work without the PDFReports plugin.
• PDFReports plugin templates will have to implement a hook mechanism.
• Phone numbers and auth tokens will be stored in the option table one line per user with the following structure :

USERNAME__MobileMessagingSettings = 
  array(
    'PhoneNumbers' => array(
                         '003354827548' => NULL, // NULL : phone number is activated
                         '00542857842754' => 52123, // !NULL : validation token value
                       ),
    'ProviderAuthToken' => 'f4e8za4al', // Only if Super User or if MobileMessagingSettings.DelegatedManagement is set to TRUE
  )

• Whether the auth token is managed by the super admin or not will be defined in the option table with the following structure :

MobileMessagingSettings.DelegatedManagement = TRUE // True if users can manage their own SMS account 

• The code managing the SMS provider will follow the Service Gateway design pattern to keep an open door for multiple SMS providers.

[mattab]

See the following for phone number formatting: http://www.mediaburst.co.uk/blog/convert-a-mobile-number-into-international-format/

[peterbo]

Does this feature really make sense in times of smartphones / email / analytic apps / nearby constant wifi / EDGE / 3g / 4g connectivity?

IMO, sending a SMS would absolutely make sense in context of #1486 (any alert is triggered for a website), because an alert often indicates that urgent action is required.

[robocoder]

I'd rather see Alerts and push notifications, but we don't dictate what others work on here. The only thing we should decide is whether this is included in the core distribution or not.

[mattab]

I think it makes sense because not all the world is connected like EU/North america is (3g, constant wifi do not exist in most places in the world). SMS is huge is Asia, Africa, south america, etc.

Also, this ticket will build all the SMS logic handling code which will then be reused for Custom Alerts #1486 where, as mentionned, it will be even more interesting and useful.

It will be included in the core but disabled by default.

[mattab]

"DelegatedManagement" => TRUE // True if users can manage their own SMS account

I don;t think this is stored per user. Instead it is stored as a general Option value.

Otherwise, proposal looks great, looking forward to seeing it!

[mattab]

Maybe you want to use something fancy for the country selector,see this cool jquery plugin: http://baymard.com/labs/country-selector

Btw for the country selector, please use existing country translations. Try to default to the country of the selected language. Later, we might reuse this website selector in other places :)

[julienmoumne]

Now that plugins can have their own config files (#2871), I was wondering what is the best solution to store the phone numbers and SMS API tokens for the SMSManager plugin.

Do they belong in the database in the option table or in the new local config file ?

Is it only a question of being able to replicate this data easily when using multiple Piwik instances or is there other factors to weight-in ?

[robocoder]

Yes, it depends. In this case, I would say database because:

• you indicated either per site or per user settings
• this isn't information that needs to be loaded on every tracking / dashboard / api request
• it needs to scale as the number of users / sites increases

[mattab]

+1 with database storage of phone numbers and tokens, in the "piwik_option" table.

[julienmoumne]

There's a very high probability this plugin will be used to send alerts.

Could you please confirm 'SMSManager' is the name that makes the most sense?

[mattab]

Updated report description for more clarity + Proposal to use MultiSites.getAll and a new MultiSites.get to fetch the data to use in the SMS reports. The advantage of doing that would be to not have to process the % evolution in the SMS manager implementation. it would be better to leave all data processing to the MultiSites.get* API. Thoughts?

[julienmoumne]

We currently have one type of report : e-mail.

We now would like to implement a new type : sms.

While doing so, we would like to refactor the existing PDFReport plugin so it can manage both types, and possibly more in the future.

The idea is to use the hooking/event mechanism to allow other plugins to add new types of reports.

We already know that e-mail and sms reports share some properties but they do also have specifics ones.

shared properties :

• report description,
• site id,
• owner,
• periodicity,
• format
• report content
• recipients

e-mail specific properties :

• display format (with or without graphics)
• email me (email the owner of the report)

The first question we need to answer is how do we manage persistence.

There are many known academic anwsers on how to store polymorphic objects in a relational database.

• one table with nullable fields

  Have one table, piwik_report, with all the fields required by all the report types. Fields that are not shared by all reports types are nullable (e.g: email me, display format).

• one table per report type

  Have one table per report type, piwik_report_email, piwik_report_sms. Shared properties are duplicated, specifics properties are isolated in their own table.

• one table for shared properties, one smaller table for each report types

  Have one table, piwik_report, with all the shared properties. Have one smaller table, piwik_report_email, will the email type specific properties (email me, display format). This table has a foreign key linking to piwik_report. Same goes for piwik_report_sms.

Here are two less academic ways to solve this issue :

• one table with a JSON encoded parameter field

  Have one table, piwik_report, with all the shared properties and a field called 'parameters'. This field would contain the JSON encoded specific properties.

• two tables

  Have one table, piwik_report, with all the shared properties. Have another table, piwik_report_parameters. This table would be as follow : report id, parameter name, parameter value.

[mattab]

The JSON parameters array seems a good idea because it doesn't add a new table.

• The only downside would be that it would make it harder (yet not impossible) to select reports with a specific parameter value (eg. select all reports with 'email_me'), because it isn't a column.

It's not obvious what the best solution is... Because it's quite easy to add new fields on Piwik update, the "one table with nullable fields " is also do-able...

I think I would implement it with the JSON solution still, thoughts?

[julienmoumne]

(In [6478]) fixes #2708

refs #5491

• PDFReports major refactoring. Any plugin can now add new kinds of reports. Required for Send reports via SMS #2708 and Report Publisher #3118.
• test report functionality ($idReport == 0) dropped in Piwik_PDFReports_API->generateReport()
• All Websites report shows 3 more metrics: Goal Conversions, eCommerce Conversions and eCommerce Revenue. Can be removed if asked to.
• Piwik_PDFReports_API->sendEmailReport renamed to sendReport
• All Piwik_PDFReports_API method signatures updated to support generic report parameters
  refs Dashboard for multiple sites (potentially hundreds of websites) #389
• new API method to retrieve only one Piwik site : Piwik_MultiSites_API->getOne()
• per Send reports via SMS #2708 description, Piwik_MultiSites_API methods now support a new parameter named enhanced. When activated, Goal Conversions, eCommerce Conversions and eCommerce Revenue along with their evolution will be included in the API output.
• API metrics refactored in (@ignored)Piwik_MultiSites_API->getApiMetrics()
• Metadata now returns 12 metrics : nb_visits, visits_evolution, nb_actions, actions_evolution, revenu, revenue_evolution, nb_conversions, nb_conversions_evolution, orders, orders_evolution, ecommerce_revenue, ecommerce_revenue_evolution
  refs Report Publisher #3118
• ReportPublisher plugin could now easily be implemented
  commits merged
• r6243
• r6422 (Scheduled report by email with custom date range triggers error #3012)
  TODO
• the MobileMessaging settings page may need some embellishment
• @review annotations need some attention
• test if the MultiSites API evolutions have some impact on Piwik Mobile and other client code

[mattab]

Julien, Wonderful patch! Thank you for this work.

Really nice refactoring and well written code.

Code review

• Well done on renaming all pdf functions to have a more generic name. Unfortunately we can't rename the plugins yet, but one day we will do something clean for sure (Piwik 2.0 or 3.0? ;)

•             // @review this is not optimal in terms of performance: those metrics should not be retrieved in the first place 
  

  Agreed.. but that is technical debt I suppose.

  • in the test output I see: <revenue_evolution>$ 100%</revenue_evolution> [...] &lt;ecommerce_revenue_evolution>$ 0&lt;/ecommerce_revenue_evolution> -

I think this would be caused by the revenue beautifying code in Piwik::getPrettyValue - it shouldn't be called on money when metric contains 'evolution' ;)

• MobileMessaging_MobileReport_NoPhoneNumbers and MobileMessaging_MobileReport_MobileMessagingSettingsLink could be in the same translation with %s in the middle - makes it clearer for translators that it belongs to same sentence.
• Very nicely done on your preg_replace_callback filter beast! ;)
• When I click on "Download" a SMS report on a day with no visit, it outputs error

/index.php?module=API&action=index&idSite=1&period=day&date=yesterday&updated=1&token_auth=0b809661490d605bfd77f57ed11f0b14&method=PDFReports.generateReport&idReport=2&outputType=1&language=en

Output= Fatal error: Call to a member function getColumns() on a non-object in piwik\svn\trunk\core\API\ResponseBuilder.php on line 176

• The output .sms file looks like:

Tuesday 3 July 2012. 2 Visits (+100%), 3 Actions (+100%)<?xml version="1.0" encoding="utf-8" ?>
<result>
    <success message="ok" />
</result>

The <?XML response shouldn't be in the .sms output I believe this is a bug?

• The link "Download" is to make previewing easier. the goal really is to enable quick preview of the SMS content. I propose that:

  • the SMS text is displayed in the browser (in a new window) rather than triggering a file download

• //@review the length of the SMS text is limited by Mediaburst. Here, we truncate the SMS content because MultiSites.getAll can return a lot of sites. Is this the proper way to handle this case?

  Looks good!

  • However I propose that we clearly say that it is truncated. For example prepending the text [long](too) - it will take 10 letters but worth to convey the message clearly that is cut and we know it. Otherwise users might think "A bug" rather than what it is "A missing feature" :)

• // @review is this the correct way to store calling codes or should we use GLOBALS like in core/DataFiles/Countries.php?

It looks good - we used GLOBALS but could have used the static class solution instead.

• //@review sms can only contain one report, we currently silently discard all reports except the first one, is this ok or should we raise an exception?

  It sounds fine to me

• //@review when a wrong phone number is supplied we silently discard it, should an exception be raised?

  I think that if they submit 1 phone number and that it is wrong then it should be throwing an exception. If they submnit many and only 1 is wrong then it's OK not to say anything?

• @review should we also delete the plugin settings (API credentials, phone numbers) located in table piwik_option?

Api credentials and phone numbers should be removed when the plugin is deleted, not really when it's just deactivated. There is a method uninstall() on the Plugin class, but currently the UI does not allow to delete a plugin. So I think it's better to not delete API credentials on deactivate.

• When the "MultiSites" plugin is disabled:
  • there are some NOTICE errors displayed on the Messaging report listing reset() expects parameter 1 to be array, boolean given in piwik\svn\trunk\plugins\MobileMessaging\MobileMessaging.php on line 163
  • The generated SMS are empty, instead they should say a message "To generate SMS texts of your website stats, please enable the plugin MultiSites in Piwik."
• {* TODO this UI probably needs some embellishment by working out a better HTML structure (table ? list ?), specifying better locations for errors and so on *}

It would be great to improve the UI. I have some ideas see below. Also will have more once the UI works for me (see below for the bug i experience preventing signing in mediaburst)

• @review should this go in the LEGALNOTICE file ?

Ideally yes it should be in that file, but we haven't been enforcing this greatly.

• I like how you factorise every string as a private member, it helps with code clarity for sure.
• API: addPhoneNumber() I can see as a potential sec issue as it is could be misused by a Piwik "view" user who could request dozens of SMS and use all credits.
  • could we add a limit "per user" of 3 maximum phone numbers pending validation? With automatic reset after 48 hours or so?
    What do you think?

Testing

• I take note that SMS output is not tested because we do not currently test PDF outputs and Email reports text. Hopefully we will never break the display code ;)

Add new DIV to describe the SMS provider'

I propose that we add a text such as the following:

You can use Mediaburst.co.uk to send SMS Reports from Piwik.
 * First, [https://secure.mediaburst.co.uk/signup/piwik signup to Mediaburst]
 * Pay for some SMS to your account
 * Fill in your Mediaburst account details in Piwik.

About Mediaburst
 * Mediabursts text message API gives you fast, reliable high quality worldwide SMS delivery, over 450 networks in every corner of the globe. 
 * Cost per SMS message is ~0.08USD (0.06EUR).
 * Most countries and networks are supported and will receive the text messages, but a few networks are currently not supported in Nigeria, Colombia (Comcel), Bangladesh, Belgium (Movistar), Gibraltar, Greece(Cosmoat), Hungry (Panafon).

This text could be returned as a getter on the provider class.

Looking forward to seeing further small changes suggested in this ticket!

As a conclusion, I will say that this is a great new feature(s), and again Big Kuddos to you for the vision and the very nice implementation!!

[mattab]

(In [6563]) Refs #2708 Small changes

[mattab]

(In [6614]) Refs #2708 Using Piwik_Common::json* instead of core PHP to avoid annoying PHP bugs with old php builds

[mattab]

UI Review

• Move: "Allow users to manage their own SMS API credential" to the bottom of the page
  • Use same presentation as setting "Allow Piwik archiving to trigger when reports are viewed from the browser" ie. Yes and No above each other
  • Below "Yes" write "default" in grey (as in "Allow Piwik archiving to trigger when reports are viewed from the browser")
  • Below "No" write in grey "All users are able to receive SMS Reports and will use your account's credits."
• error message could be displayed between "SMS API Account" and "Phone numbers" to make it more visible
• Replace "SMS API Account" by "Manage SMS API Account" ?
• Default state, No SMS API configured:
  • below write "To create SMS reports and receive short text messages with your websites' stats on your mobile phone, please sign up with the SMS API and enter your information below."
  • Display the form below the text
  • Replace "You can not configure phone numbers. The API username and password have not been correctly configured." by "Before you can create and manage phone numbers, please connect Piwik to your SMS Account above" or similar?
• When an account is configured:
  • Replace "SMS API Account" by "Configure SMS Account"
  • "The Mediaburst SMS API account associated with username X is correctly configured. Current Credit: 100 Delete Account. Update Account : ". Replace with "Your Mediaburst SMS API account is correctly configured! You have X credits left. <br>You can also update or delete this account."
  • On click or update, the fields below would appear (hidden by default)
  • On click on delete, please display the confirmation modal popover window with "Are you sure you want to delete the SMS account? "

Phone numbers admin

• Below title write "Before receiving SMS (text messages) reports on a phone, the phone number must be entered below. When you click "Save", a SMS containing a code will be sent to the phone. The User will then be able to validate the phone number if he has access to Piwik. "
• add subtitle "Add a new Phone Number" above the form
• Below first input write in grey "Country code"
  • When user enters manually an international code, it would be nice if the first input form was set to the country that matches the code entered. This would confirm to the user that he's correct.
  • Instead of displaying the country input field before international code, could we maybe display on the right, in a yellow box inline help "If you do not know the phone country code, type your country name here: [....]" -- what do you think? Or maybe we could display it below the 2 input fields?
  • The internatioanl code INPUT field should be 4 characters wide
• Below second input write "Phone number"
• add subtitle "Manage Phone Numbers" above the listing of phone numbers

After adding a phone number, write "We just sent a SMS to this number with a code: please enter this code below and click "Validate"."

• when the code is incorrect write "Code entered was not valid, please try again."
• when the code is correct write "Phone number validated! You can now receive SMS with your stats."

Edit Report UI

• when "Mobile" is selected, the next row "SMS" --> could you just write SMS rather put a SELECT box since there's only one option?
• Clicking on "All Websites dashboard" does not select the radio
• Small Bug: Create SMS report. Then edit, click on the link " the Mobile Messaging settings page", &updated=1 is set in the SMS Settings page which displays the yellow box "Changes saved". To prevent, set "updated=null" in the link?

[mattab]

More suggestions/report:

• When "All websites" is selected and there is no data for all websites the text contains the date only. Could we prepend also "There is no data for this report." CoreHome_ThereIsNoDataForThisReport
• When 1 website has data, the data is displayed, but the website name is not, so we don't know which website it refers to
• When "All websites" is selected the From field should be General_Reports -- currently it is the website name which SMS was created under.
• When I select "Single website" and click or downlaod there is the error "API returned an error: The parameter 'idSite=' is missing from the request."
• in the SMS validation message "Piwik > Settings > SMS" should be "Piwik > Settings > Mobile Messaging". PS: can you please use the actual translations for "settings" and "mobile messaging" so that translators dont duplicate and use different wordings, since it's quite important to get this wording right :)

[julienmoumne]

(In [6727]) refs #2708

• fixing a bug in Piwik::getPrettyValue() where a revenue evolution would be prettified like so: $ 100%
• fixing a bug happening when no data is available for the selected day
• when MultiSites plugin is deactivated, invite user via SMS to activate it back
• SMS API accounts are now managed using an API key instead of a username and a password
• adding long at the end of the SMS content when it reaches the maximum length allowed by the SMS API
• support for UCS-2 characters
• Mediaburst rebranded to Clockwork
• various UI improvements based on comment:31:ticket:2708
• PDFReport.generateReport now supports $outputType=3 : output report in browser
• removing non-libre select-to-autocomplete jQuery plugin
• tracking count of phone number validation requests and SMS sent
• SMS content now contains CoreHome_ThereIsNoDataForThisReport when applicable
• setting SMS From with General_Reports when configured report is MultiSites_getAll
• adding Clockwork description

TODO

• using POST instead of GET to send SMS to go around a Clockwork limitation

[julienmoumne]

(In [6728]) refs #2708 fixing integration tests failing after r6727

[mattab]

Very cool, thanks for doing all changes! :)

Feedback

• Pending: the urls to clockworksms don't work yet, waiting for feedback from them
• The clockwork help should not be translated at this stage, since it contains URL, price etc. I suggest we simply hardcode MobileMessaging_SMSProvider_Description_Clockwork in the template for now.
  • the logo is not displayed on the page
• is it normal that a user can delete other users' phone numbers?
• when "All websites" is selected, but that only 1 website out of all has some data, the SMS report looks like a single site report, but it does not give the site name, so we don't know which site it refers to.
• All Websites "from" is "Reports@" with an @ sign at the end - is it normal?
• Tests: now that there's a simple way to get SMS text output, it would be awesome to have an integration test the SMS output in the 2 use cases (1 website, all websites) as it would make sure the plugin never breaks, eg. if we change the PDFReports code..

I will also commit a few small changes

[mattab]

(In [6738]) Refs #2708

• Small HTML tweaks
• Fix bug when anonymous user opens Edit reports page

[diosmosis]

(In [6741]) Refs #2708, tweak english translations.

[julienmoumne]

(In [6758]) refs #2708

• removing providers' descriptions from language file

• adding providers' logo

• fixing "API returned an error: The parameter 'idSite=' is missing from the request." when a normal user queries MultiSites.getOne

• displaying website name in SMS content even when MultiSites.getAll returns one website

  TODO

  • integration tests for SMS content
  • using POST instead of GET to send SMS to go around a Clockwork limitation

[julienmoumne]

(In [6761]) refs #2708 fixing regression introduced in r6758

[mattab]

I noticed an inefficiency in the Report generation.

I notice that when I run the following SMS "Single site" report with debug enabled:

http://localhost/trunk/index.php?module=API&action=index&idSite=1&period=month&date=2012-06-11&token_auth=0b809661490d605bfd77f&method=PDFReports.generateReport&idReport=4&outputType=3&language=en&debug

with, in the config file:

[log]
log_only_when_debug_parameter = 1
logger_message[] = "screen"

It outputs on screen: http://pastebin.com/8TyxhpAr

As you can see when generating one SMS reports it first loads all archives from the DB for ALL websites. The SMS was a "single website" so it should only query this site data. I havent looked at the code please let me know if you need help or more info! :)

NB: I tried generating a PDF that didn't contain the "All websites dashboard" and it does not do the extra requests for other websites, so maybe it's a problem with SMS only?

[julienmoumne]

(In [6764]) refs #2708 blacklisting MobileMessaging plugin from API integration tests

[mattab]

(In [6772]) Refs #2708

• Beautify the SMS Provider DIV

[julienmoumne]

integration tests task has been logged in #3323

[julienmoumne]

using POST instead of GET to send SMS logged in #3325

[mattab]

In the future, I think it would be cool to ask during the installation process if users want to send SMS reports, with a checkbox, and if so enable the MobileMessaging plugin, to maximise number of users who know about this feature.

Kuddos Julien for this great new feature!

[julienmoumne]

list of improvements logged in #3337

[julienmoumne]

(In [6849]) refs #3323 #3088 #2708 #71 #2318

• generate and compare HTML, PDF & SMS reports in Test_Piwik_Integration_EcommerceOrderWithItems & Test_Piwik_Integration_TwoVisitors_TwoWebsites_DifferentDays
• report content as return value of PDFReports->generateReport() with new output type OUTPUT_RETURN