Enable fingers crossed handler via INI config and show backtrace in logs/archive api output #13923

diosmosis · 2018-12-30T23:02:02Z

Fixes #8698
Fixes #8616

Tested in the following contexts:

web request (w/ warning/error all logs written to file log, only warning/error shows up in screen)
normal console command (fingers crossed handler not used)
core:archive top level command (fingers crossed handler not used)
core:archive climulti processes (w/ warning/error all logs written to file log
when a fatal error occurs

Also made following changes:

log more exceptions
make backtrace in logs include previous exceptions' backtraces
don't use screen writer if in CLI mode (so notifications don't end up on the screen when we don't want them to). could also add a check for trigger=archivephp.
always print backtrace if in CLI mode & trigger=archivephp

CC @tsteur not sure which milestone to put this in

tsteur · 2018-12-30T23:43:05Z

config/global.ini.php

@@ -104,6 +104,14 @@
 ; if configured to log in a file, log entries will be made to this file
 logger_file_path = tmp/logs/piwik.log

+; if set to 1, will enable monolog's FingersCrossedHandler, which buffers all logs and flushes them if a warning or error


I wonder if it actually needed to have an option for it or always enable it?

ATM I'm not sure what effect this will have in production, thought it could use some more testing before it's enabled by default. I suppose it wouldn't be useful for most users anyway? Unless we create some sort of bug summary feature that grabbed these logs so users could upload these w/ error reports.

I reckon it would always be useful to have it enabled so we get detailed logs when we ask users to enable it etc. Could also leave the option, test it for now on production, and follow up to remove the setting in 2 or 3 months and have it always enabled.

tsteur · 2018-12-30T23:43:24Z

config/global.ini.php

+
+; if enable_fingers_crossed_handler is set to 1, and this is set to 1, then the FingersCrossedHandler will stop buffering
+; on the first warning/error. if another one occurs later in the request, there will be no extra debug logs for them.
+fingers_crossed_stop_buffering_on_activation = 0


what's the use case for this one?

FingersCrossedHandler has an option where it stops buffering after the first error. I assume it's to avoid buffering too many logs in case there are lots of errors? I'm not exactly sure, but figured someone might want to set it true (by default I've left it disabled so every error triggers debug logs to be flushed).

Same here I reckon we could keep the setting and see how it behaves in production and then possibly remove the setting and always have it disabled but to be seen after some more testing

tsteur · 2018-12-30T23:44:31Z

I'll put it into 3.9 for now. Can other plugins get all the logged messages during runtime? To send them eg by email

diosmosis · 2018-12-30T23:54:04Z

@tsteur added some more changes just now, might want to have another look.

Can other plugins get all the logged messages during runtime?

Not at the moment. We could create a custom FingersCrossedHandler that posts an event when triggered. Or the plugin could grep the logs using the current request ID (that's what I was thinking at first).

tsteur · 2018-12-31T00:02:47Z

when eg a fatal error occurs, we're sending an email during that request and looking through logs is not really an option as the log files could be many GB big. It would be great if there was some static method or some way to get the logger instance with all the log messages without using events (re performance).

diosmosis · 2018-12-31T00:53:41Z

I'm not sure there is a way to do that currently. I think FingersCrossedHandler will abandon the saved logs once they are flushed to the internal handler, so they're not all retained. I guess we can try to extend/modify the handler.

Will we ever look into something like loggly/cloudwatch/etc.? If the logs were more easily accessible to us, it might not be needed to have them attached to an email (just a request ID would suffice).

tsteur · 2018-12-31T01:14:59Z

We will likely look into CloudWatch eventually but we wouldn't send any data to a 3rd party.

I thought there would be maybe some logging handler or writer that could be put in between maybe?

tsteur · 2018-12-31T01:15:52Z

Also the goal is really to have all information directly in the fatal error email to have immediately all information.

diosmosis · 2019-01-03T03:06:07Z

@tsteur Added the ability to get all logs in a request using a new handler. Also removed the INI config from global.ini.php and just check for them in code, so it's easier to remove if we find we don't need to include them later.

tsteur · 2019-01-03T21:40:56Z

core/API/Request.php

@@ -269,7 +271,9 @@ public function process()
                return $response->getResponse($returnedValue, $module, $method);
            });
        } catch (Exception $e) {
-            Log::debug($e);
+            StaticContainer::get(LoggerInterface::class)->error('Uncaught exception in API: {exception}', [


I wonder if we should keep debug level when it is fetched through UI to not expose the specific error here? just not sure if there could be any problem such as sensitive information or something. Likely not a problem though.

If it's not debug, it won't trigger the fingers crossed handler, BUT I made sure the screen handler doesn't get wrapped in the fingers crossed handler. So I think this will result in an exception displaying on the screen for UI requests, but it should just print the exception message, which is the current behavior right? I'll make sure to see exactly what happens.

Was just not sure whether it could have any side effects re the exception message re security or whether it could be any annoying or cause any other problems. Wonder if it would even show during an API request etc or what would happen then? I wonder if we need to add tests for all those cases where this error is triggered.

Would be nice to have tests, though I'm not sure how we'd verify. Eg, if we trigger an error in a PHPUnit test, how do we test if a message showed up on the screen? Could use a UI test, but that might get complicated to write and may not be worth it...

Would be easier w/ headless chrome, then we'd be able to pull in npm packages to make it easier to verify things like "did this get logged to the file log".

I meant mainly to make sure that the API output is still correct when an error occurs during an API request. Might be even good for one UI test to see what it shows there. Wouldn't need to check re what is logged to the file.

Created a UI test and noticed that the error will also appear as a notification. Not sure what the best way to handle this is, since w/o the log statement, there ends up being no record of the error in the file.

My personal opinion is that there really shouldn't be a screen logger, and everything should go into a log that only admins/devs can see. But I'm not sure if that change can be made.

That's what I meant with changing the log level from debug to error. The screen logger is definitely useful in that it helps us finding bugs/issues etc. where otherwise 98% of users would never look in the logs and not even know they exist etc.

The only thing I can think of be to change the log level depending on whether we are CLI or not. Or also enable the fingercrossedhandler on cli based on some other flags maybe. Can we maybe log it as debug and somehow pass some option to also attach all previous logs when this happens? Defeats the idea of the fingercrossedhandler maybe a little but could work maybe?

We could also go in similar, but opposite route, add a property to the ::error() calls to ignore by the screen handler, eg:

->error("...", [ 'exception' => $ex, 'handledInUI' => true, // or maybe 'ignoreInScreenWriter' to be more generic ]);

So anywhere we print the exception or don't want it to be displayed in the UI, we can add that to the log. What do you think? This way it keeps the meaning of FingersCrossedHandler.

Sure it should be fine this way as well 👍

Made the change + added some more logs. Also modified ExceptionToTextProcessor to replace {exception} in the message instead of the whole message.

tsteur · 2019-01-03T21:48:05Z

core/ExceptionHandler.php

@@ -41,6 +44,8 @@ public static function handleException($exception)
     */
    public static function dieWithCliError($exception)
    {
+        self::logException($exception);


Likely not too important but just realising that console is doing eg Piwik\ExceptionHandler::setUp(); (and also Piwik\ErrorHandler::registerErrorHandler();) before the environment is loaded and container is built. So there could be under circumstances a problem where a message is maybe logged but the container is not ready yet. Not sure what would happen and likely the chances are maybe low there is an error between registering error handler and loading environment...

actually stupid of me... I'm just seeing you catch an exception there anyway. Forget the comment :)

tsteur · 2019-01-03T21:49:19Z

core/ExceptionHandler.php

+                'exception' => $exception,
+            ]);
+        } catch (ContainerException $ex) {
+            // ignore (occurs if exception is thrown when resolving DI entries)


lets maybe also catch ContainerDoesNotExistException in case the container is not set up yet?

tsteur · 2019-01-03T21:50:30Z

core/FrontController.php

@@ -107,6 +108,10 @@ private static function generateSafeModeOutputFromError($lastError)
     */
    private static function generateSafeModeOutputFromException($e)
    {
+        StaticContainer::get(LoggerInterface::class)->error('Uncaught exception: {exception}', [


wouldn't that make the error visible in the UI?

Wouldn't it always be visible in the UI? If it's not logged, there won't be information about it in the logs (and it won't trigger fingers crossed).

Not quite. Eg on Cloud we do not want to show the actual exception and we overwrite the safemode controller action for this. I think under circumstances we would also show the error even if not wanted. Eg in safemode we replace the actual error message for anonymous with A fatal error occurred. . Haven't tested though whether it would actually show this message in the UI. Also not sure what happens for API calls etc.

tsteur · 2019-01-03T21:54:05Z

Also removed the INI config from global.ini.php and just check for them in code, so it's easier to remove if we find we don't need to include them later.

should we maybe create a follow up issue for 3.9 or 3.10 to enable it by default if our tests go well?

diosmosis · 2019-01-03T21:56:51Z

should we maybe create a follow up issue for 3.9 or 3.10 to enable it by default if our tests go well?

Makes sense, will do so once this PR is approved.

Findus23 · 2019-01-03T22:15:47Z

Will we ever look into something like loggly/cloudwatch/etc.?

It would be amazing if there was an easy way to get all Matomo errors/warnings into sentry.

I originally wanted to integrate Sentry into the Matomo monolog logger, but failed, so at the moment I am just using the default Sentry PHP error logger, which kind of works, but is incredibly hacky and probably doesn't show Matomo errors, just PHP errors and uncaught exceptions.
https://plugins.matomo.org/SentryLogger

Still, using (a self-hosted) Sentry to keep track of Matomo errors has been incredibly helpful as I can keep track of all errors I come across on the server and during development, can see if they occur regularly or just once, have them automatically grouped and see stack traces including all variables that were set during the exceptions.

Screenshot

So if there is any way (for a plugin) to get all errors and warnings including stack traces in Matomo to send it to Sentry, it would be really helpful for me (and guessing by the fact that nearly 500 people have downloaded the really specific plugin, probably others too).

Having better error backtraces sounds also really useful for debugging odd issues on the forum.

diosmosis · 2019-01-03T22:26:22Z

Didn't know about sentry, if there are any issues w/ integrating w/ Matomo that I can help w/, let me know, will probably want to use that on my personal matomo instance ;)

diosmosis · 2019-01-04T01:48:54Z

So if there is any way (for a plugin) to get all errors and warnings including stack traces in Matomo to send it to Sentry, it would be really helpful for me (and guessing by the fact that nearly 500 people have downloaded the really specific plugin, probably others too).

@Findus23 Just read this, you could try creating a custom log handler. Exceptions will be in the 'exception' property in the context (see eg how ExceptionToTextProcessor uses the property). I think you could use \DI\add or \DI\decorate w/ 'log.handlers' in your plugin to add it. You won't be able to catch exceptions that aren't logged, but we should probably be logging everything.

tsteur · 2019-01-07T20:00:41Z

core/API/Proxy.php

 {
    // array of already registered plugins names
    protected $alreadyRegistered = array();

-    private $metadataArray = array();
+    protected $metadataArray = array();


not 100% sure if this was meant to be in the PR? Seeing there were couple tests for this but not sure if it was meant for different PR maybe?

It's used by the new UI test, see CustomApiProxy in UITestFixture.php.

diosmosis · 2019-02-15T02:20:08Z

@tsteur any issue w/ merging this PR? could help on cloud

… previous exceptions, do not use screen writer if in cli mode, always print backtrace if in CLI mode and archivephp triggered.

…art of message in ExceptionToTextProcessor instead of whole message.

tsteur · 2019-02-15T02:47:06Z

Nope

@mattab

commit 517e800 Merge: 6179c1b 4bf164f Author: Fabian Dellwing <[email protected]> Date: Tue Mar 19 16:37:58 2019 +0100 Merge https://github.com/matomo-org/matomo into stylelint commit 4bf164f Author: Matthieu Aubry <[email protected]> Date: Tue Mar 19 17:50:18 2019 +1300 3.9.0 commit 7d0c3c6 Author: diosmosis <[email protected]> Date: Mon Mar 18 19:38:49 2019 -0700 Fixing build (matomo-org#14222) * Fix test failure. * Update expected test file.s * Fix action iconSVG regression. * Update submodule. * Update screenshots. * Update more UI test files. commit 73ef47a Author: Matthieu Aubry <[email protected]> Date: Tue Mar 19 15:30:38 2019 +1300 3.9.0-rc2 commit f5d425c Author: diosmosis <[email protected]> Date: Mon Mar 18 18:04:22 2019 -0700 in segmented visitor log segment is already encoded, decode it again when adding as new (matomo-org#14221) * segment is already double encoded, decode it again * Remove addSegmentAsNew after apply and save. commit 94fa2b3 Author: Thomas Steur <[email protected]> Date: Mon Mar 18 14:16:49 2019 +1300 update tag manager submodule (matomo-org#14215) commit c3a9183 Author: Thomas Steur <[email protected]> Date: Mon Mar 18 12:30:03 2019 +1300 Make sure to anonymize token only when needed (matomo-org#14216) commit cdbf954 Author: Kate Butler <[email protected]> Date: Mon Mar 18 08:08:14 2019 +1300 Add line break to email footer when whitelabel plugin active (matomo-org#14200) commit 9aabd38 Author: Stefan Giehl <[email protected]> Date: Sun Mar 17 13:53:26 2019 +0100 language update (matomo-org#14214) commit cd9f950 Author: sgiehl <[email protected]> Date: Sun Mar 17 13:23:32 2019 +0100 submodule updates commit 6179c1b Merge: 7ddf7c4 da72724 Author: Fabian Dellwing <[email protected]> Date: Fri Mar 15 16:43:41 2019 +0100 Merge https://github.com/matomo-org/matomo into stylelint commit da72724 Author: diosmosis <[email protected]> Date: Thu Mar 14 21:18:45 2019 -0700 Do not match by config ID if trust visitor cookies is used and a user ID is supplied. (matomo-org#14196) commit 8edc76e Author: Matthieu Aubry <[email protected]> Date: Fri Mar 15 16:43:53 2019 +1300 3.9.0-rc1 commit 3474bf2 Author: Stefan Giehl <[email protected]> Date: Fri Mar 15 01:35:51 2019 +0100 Allow customization of GeoIP2 database location using DI (matomo-org#13913) commit a99baa9 Author: Kate Butler <[email protected]> Date: Fri Mar 15 11:42:04 2019 +1300 Improve wording on anonymous user settings page (matomo-org#14175) commit 6395855 Author: Thomas Steur <[email protected]> Date: Fri Mar 15 11:24:36 2019 +1300 Support multiple plugin paths (matomo-org#14051) * do not hard code plugins directory * remove method that is not needed for now * use plugins directory in more places * some work on supporting multiple plugin directories * use more unique name * couple fixes * and another fix * sort plugins * adjust languagesmanager * adjust more usages * Update Manager.php * adding a plugin to test * more tests * make sure plugin resources can be located in custom directory * adding more tests * rewrite image paths * handle more cases * add tests * make sure to load plugin * trying to fix test * trying it this way * load plugin * fix ui test? * testing if tests succeed this way * another test * load custom dir plugin * load plugin in ui fixture * change the update statement * remove update script * delete column * fix ui test * make it work for tests * fix some tests * Fix merge. commit d7c9327 Author: Matthieu Aubry <[email protected]> Date: Fri Mar 15 10:24:13 2019 +1300 Rename piwik to matomo in a few INI settings for consistency (matomo-org#14187) * Rename piwik to matomo in a few INI settings for consistency Just updated the website FAQs to mention matomo.log instead of piwik.log so thought we should do the same in Matomo * add to changelog as it may be breaking * Update expected screenshot. * Update screenshot. commit 6d39aaa Author: Stefan Giehl <[email protected]> Date: Thu Mar 14 20:30:25 2019 +0100 Revert icon changes and add new property iconSVG instead (matomo-org#14199) * Revert icon changes and add new property iconSVG instead * update test files commit 00d15de Author: Thomas Steur <[email protected]> Date: Fri Mar 15 08:29:01 2019 +1300 Ignore config files placed in misc folder in fileintegrity check (matomo-org#14203) Follow up from matomo-org#13999 where we allowed to place config files in misc/user directory. These need to be ignored in the file check commit 8f77ece Author: Stefan Giehl <[email protected]> Date: Thu Mar 14 20:26:26 2019 +0100 Show device type and model in realtime map tooltip (matomo-org#14204) * Realtime map improvements * updates ui file commit 285f284 Author: diosmosis <[email protected]> Date: Wed Mar 13 20:51:38 2019 -0700 Do not run cron archiving last run check if matomo is not installed. (matomo-org#14202) commit 7393e13 Author: Thomas Steur <[email protected]> Date: Wed Mar 13 14:16:30 2019 +1300 When dispatch is disabled via a constant, it should not be dispatched (matomo-org#14034) * When dispatch is disabled via a constant, it should not be dispatched Eg when Matomo is not installed, it would still dispatch the request in https://github.com/matomo-org/matomo/blob/3.8.1-b1/plugins/Installation/Installation.php#L111 even when `PIWIK_ENABLE_DISPATCH` is disabled. Will set it to WIP for now as I'm not sure if we want to have this actually merged or not. * Update Installation.php * Use correct exception class * throw exception if one is given * adding a test * fix tests * trying to fix test commit def7436 Author: diosmosis <[email protected]> Date: Tue Mar 12 17:13:57 2019 -0700 Couple assorted changes (matomo-org#13935) * Allow annotations API to accept multiple periods, so evolution graphs that use multiple periods work. * Remove warning when rows_to_display viewdatatable config property is left at its default value. * Allow individual cells in an html visualization to be styled (if the visualization is extended). * Remove unneeded TODO. * In series picker encode picked rows in case the labels have commas. * Must decode the rows value as well (as it is not handled by API, must be done in plugin). * Allow joins to specified through LogAggregator::queryConversionsByDimension(). * Add safety check to _idts processing: if visitor is unknown, ignore _idts value, since it is their first visit. * In the tracker when searching by visitor ID, search through entire log_visit table instead of just in the last 30 mins. * When tracking visitor days since first, do not round since this can result in inaccurate data when rounding up. Which can cause trouble when finding the start visit for a log. * Allow HtmlTable descendants to add any html attributes to cells. * Allow derived Visualizations to add custom parameters to API requests via a new RequestConfig method. * Tweak to TODO. * Add test for annotations API change & get to pass. * Apply more review feedback * Update INI config docs for window_look_back_for_visitor. * Only copy visitor properties if action is part of an existing visit. * Some more properties that should be copied over from known visitor even if new visit. * Fixing some tests. * update test * Fix CustomEventsTest test failures. * Fixing more tests. * update rest of tests * Fixing tests. * Update some test files. * Fix log statements. * To better handle out of order actions, add part of last_action_time check to visitor ID search. * Update tests. * Updating expected screenshots. * Fix ArchiveCronTest. * Throw exception if idorder not unique. * Only throw exception if idorder specified. * Fixing a couple tests. * Fix another test. commit e26f5e7 Author: diosmosis <[email protected]> Date: Tue Mar 12 16:16:41 2019 -0700 Add diagnostic to check last time archiving was run successfully and … (matomo-org#13972) * Add diagnostic to check last time archiving was run successfully and display notification if empty report and archiving has not been run recently. * Move cron archiving diagnostics next to each other. * Add new test for archiving not done in time check. * Remove TODO comment. commit 549578d Author: Thomas Steur <[email protected]> Date: Wed Mar 13 09:32:28 2019 +1300 New setting to disable Db version check, and do not show update scree…n when auto update disabled (matomo-org#14058) * New setting to disable Db version check, and do not show update screen when auto update disabled * move it to DI commit f613456 Author: Christian Schmidt <[email protected]> Date: Tue Mar 12 21:17:30 2019 +0100 Pagination localisation clean-up (matomo-org#14157) * Range start should be one-based, like the range end * Clean up pagination translation * Restore ScheduledReports translations * Update screenshots * Revert changes to non-English translations * Update screenshots * Update screenshot * Replace UsersManager_XtoYofN with General_Pagination * Only remove translation from en.json. * Off by one * Update screenshots * Fix off-by-one in range end * Update screenshots * Updated screenshot commit 7ddf7c4 Merge: 5f5bf01 7edf461 Author: Fabian Dellwing <[email protected]> Date: Tue Mar 12 11:26:24 2019 +0100 Merge https://github.com/matomo-org/matomo into stylelint commit 7edf461 Author: diosmosis <[email protected]> Date: Mon Mar 11 22:04:41 2019 -0700 Sparklines do not show correct values if values are close to 1.0. (matomo-org#14006) * Scale sparkline values if they have floats. * Coerce value to int/float. * Check if numeric before coercing. * In RowEvoluton format fractional values correctly. * updates screenshots commit 88aa939 Author: diosmosis <[email protected]> Date: Mon Mar 11 18:22:38 2019 -0700 fix build (matomo-org#14183) * Fixing action segment regression + update UI test files + expected test files. * Update more test files. * update submodule * Fix integration test. * Update expected screenshot. * More test fixes. * Update expected files. * Update more expected test files. * Update submodule. * Update more tests. * Update some screenshots. commit e1c1f12 Author: Stefan Giehl <[email protected]> Date: Mon Mar 11 14:26:37 2019 +0100 Make it possible to define joins for log tables using `getWaysToJoinToOtherLogTables` (matomo-org#14062) * Make it possible to define joins for log tables using getWaysToJoinToOtherLogTables * Adds some tests for custom log table joins * add missing log tables joined using getWaysToJoinToOtherLogTables * automatically add log tables up the hierarchy * code improvements * Adds new ExampleLogTables plugin giving a showcase for custom log tables * specifiy table name in userid archiver to fix query if custom log table joins on user_id column * fix tests * Adds log table that does only indirectly join with log_visit * Allow defining joins on visit and action * update ui files commit 5f5bf01 Author: Fabian Dellwing <[email protected]> Date: Mon Mar 11 13:14:03 2019 +0100 use stylelint after merge commit 542957a Merge: 9e2feb7 ef48b1b Author: Fabian Dellwing <[email protected]> Date: Mon Mar 11 13:12:32 2019 +0100 Merge https://github.com/matomo-org/matomo into stylelint commit ef48b1b Author: Stefan Giehl <[email protected]> Date: Mon Mar 11 12:49:47 2019 +0100 Fix some tests (matomo-org#14179) * update exptected system test files * update ui files * updates integration test * restore widget order in UI tests * updates submodule * updates ui files commit 2974fce Author: Fabian Dellwing <[email protected]> Date: Mon Mar 11 11:49:34 2019 +0100 handle background color of opt out if font is almost white (matomo-org#14019) * handle background color of opt out if font is almost white If no background color is set, the background color is transparent. But the whole site background color is white, so if you choose white or a color that is nearly white as font color the iframe appears so be empty. This commit implements a new function that tests if a hex color is nearly white and if yes the code will change the background color to a grey color. * use CSS instead of angular commit 5f1d88e Author: Thomas Steur <[email protected]> Date: Mon Mar 11 18:06:06 2019 +1300 Allow config files to be placed per hostname in misc/user directory (matomo-org#13999) commit a345c2f Author: Stefan Giehl <[email protected]> Date: Mon Mar 11 05:47:27 2019 +0100 Run AllTests against PHP 7.3 on travis (matomo-org#14148) * Run AllTests against PHP 7.3 on travis * use INTL_IDNA_VARIANT_UTS46 for idn_to_utf8 PHP 7.2 deprecated INTL_IDNA_VARIANT_2003 but still uses it as default until 7.4 * Fix test as var_export signature for stdClasses changed in PHP 7.3 see php/php-src@e4e9cd8 commit f7b442a Author: Matthieu Aubry <[email protected]> Date: Mon Mar 11 15:31:57 2019 +1300 3.9.0-b3 commit b68d673 Author: Stefan Giehl <[email protected]> Date: Mon Mar 11 02:38:17 2019 +0100 updates icons submodule (matomo-org#14173) commit 7148c64 Author: diosmosis <[email protected]> Date: Sun Mar 10 09:07:39 2019 -0700 Propagate token auth in ajax requests for widgetized reports AND enable_framed_pages = 1. (matomo-org#14133) * Propagate token auth in ajax requests if something is widgetized OR if enable_framed_pages is set to 1. * Add UI tests for embedding entire app. * Forgot to add file. * Adds missing UI file commit c9bc3e3 Author: Stefan Giehl <[email protected]> Date: Sat Mar 9 09:41:03 2019 +0100 language update (matomo-org#14170) commit 0648513 Author: diosmosis <[email protected]> Date: Thu Mar 7 14:42:03 2019 -0800 Send email notification when user email changes. (matomo-org#14136) * Send email notification when user email changes. * Forgot to add file. * Apply pr fixes + send email for password changes too. * Add quick test for new emails. * Translate text * Refactor according to review. * ucfirst device name * Fixing integration test commit 7153700 Author: diosmosis <[email protected]> Date: Thu Mar 7 13:50:56 2019 -0800 Add goal report metadata for overview. (matomo-org#14164) * Add goal report metadata for overview. * Fix some tests + correct goal name. * another tweak to the name * Update expected files. commit e6a9dfa Author: diosmosis <[email protected]> Date: Thu Mar 7 12:31:29 2019 -0800 Do not calculate totals if totals row is found OR if totals metadata is found. (matomo-org#14165) commit e2b9414 Author: diosmosis <[email protected]> Date: Wed Mar 6 18:24:03 2019 -0800 Allow updating multiple plugins at once. (matomo-org#14052) * Allow updating multiple plugins at once. * Disable checkbox if plugin is not downloadable. * Fix translation typo. * really fix typo commit 0d5d0df Author: diosmosis <[email protected]> Date: Wed Mar 6 17:29:09 2019 -0800 Set segments for folder branches in page urls/page titles report (matomo-org#14076) * unfinished commit * Set matadata on Action folder rows in order to correctly set subfolder segments. * Get rid of warning * Apply review feedback. * Remove unneeded metadata. * Update expected test files. * Update expected screenshot. commit d94ab38 Author: sgiehl <[email protected]> Date: Mon Mar 4 23:06:18 2019 +0100 updates submodule commit 3ba0e38 Author: Stefan Giehl <[email protected]> Date: Mon Mar 4 22:10:06 2019 +0100 Improve total report values calculation (matomo-org#14158) * Ensure summed up values are numeric to prevent a `A non well formed numeric value encountered` on PHP 7.x * Do not calculate total values for average, rate or nested metrics * Fix calculation for minimal or maximal metrics * updates expected test files * Also skip evolution metrics * skip all non numeric values * update test file * calculate totals for nested metrics recursivly * update test files commit 283ec8d Author: Stefan Giehl <[email protected]> Date: Mon Mar 4 09:46:05 2019 +0100 Improve GeoIP2 re-scheduling (matomo-org#14161) commit 9241966 Author: Thomas Steur <[email protected]> Date: Mon Mar 4 17:26:14 2019 +1300 fix queued requests in JS tracker and add possibility to disable it (matomo-org#14146) * fix queued requests in JS tracker and add possibility to disable it * Regenerate core tracker JS. * Fix jslint error. * trying to fix and debug failing js test * fix js test * trying to fix test * debugging failing test * fix JS tests * fix jslint commit 637871b Author: TheYOSH <[email protected]> Date: Mon Mar 4 02:33:27 2019 +0100 Improved archive cleanup with 60% in time! (matomo-org#11988) * Improved archive cleanup with 60% in time! Changed only the function insertActionsToKeep Speed improvements: - Get all the needed ID field values in a single run per table in stead of amount of ID field times per table (3-5 times faster) - Start with the lowest ID instead of zero. With auto numbering and cleanup, the first valid ID will be higher and higher. So why start at zero if the first ID is somewhat about 106644353. This will safe a lot of empty selects - Use extra temporary tables for data pivoting * Couple code tweaks. commit eb76c69 Author: Diego Baños Fariñas <[email protected]> Date: Mon Mar 4 01:07:02 2019 +0100 GeoIP first will update the next day, then weekly/monthly after that. Fixes matomo-org#11006 (matomo-org#11659) * Feature: GeoIP first update will be the next day. Then Monthly or Weekly * Get unit test to pass. commit fd2a9c1 Author: Matthieu Aubry <[email protected]> Date: Mon Mar 4 11:46:31 2019 +1300 3.9.0-b2 commit 6aeea10 Author: Thomas Steur <[email protected]> Date: Mon Mar 4 11:33:08 2019 +1300 Add new parameter to Live. getLastVisitsDetails API to request more detailed information (matomo-org#14147) commit 8d52548 Author: Christian Schmidt <[email protected]> Date: Sun Mar 3 23:29:51 2019 +0100 Typo: LanguageManager => LanguagesManager (matomo-org#14154) commit 71c0755 Author: Fabian Dellwing <[email protected]> Date: Sun Mar 3 23:09:13 2019 +0100 fixes matomo-org#14150 (matomo-org#14151) commit dca17aa Author: Christian Schmidt <[email protected]> Date: Sun Mar 3 22:42:52 2019 +0100 Support changing translation after instantiation (matomo-org#14155) commit 17f95c2 Author: Christian Schmidt <[email protected]> Date: Sat Mar 2 17:23:28 2019 +0100 Preserve case for CLDR month/day names (matomo-org#14015) Do ucfirst() on formatted date instead. commit 0e3c30b Author: Kate Butler <[email protected]> Date: Sat Mar 2 16:48:13 2019 +1300 Report an error when the bzopen fails (matomo-org#14145) commit 8ac338d Author: Lukas Winkler <[email protected]> Date: Thu Feb 28 21:54:15 2019 +0100 remove getKeywordsForPageUrl widget (matomo-org#14093) * remove getKeywordsForPageUrl widget * fix tests * deprecate getKeywords... API * add to developer changelog * remove empty lines * improve deprecation message * fix test * fix test * fix tests * Update UI files commit a9e301f Author: diosmosis <[email protected]> Date: Tue Feb 26 21:44:22 2019 -0800 Fix build (matomo-org#14144) * Updating some expected files. * Fix more tests. * Try to fix travis test failure. * Try to fix test on travis. * Fix test (angular directive children not ready in time in phantomjs) * Add another screenshot. commit ba10680 Author: Kate Butler <[email protected]> Date: Wed Feb 27 10:33:42 2019 +1300 Fix undefined error in segmentation.js (matomo-org#14142) commit 16edd06 Author: Stefan Giehl <[email protected]> Date: Tue Feb 26 05:36:52 2019 +0100 Show next link in visitor log only if more records are available (matomo-org#13961) commit 4193c2e Author: diosmosis <[email protected]> Date: Mon Feb 25 19:25:12 2019 -0800 If more than one yaxis in a jqplot graph, make sure there is space on right for it. (matomo-org#14111) * If more than one yaxis in a jqplot graph, make sure there is space on right for it. * Remove debugging statement. * Fixing test. * Try to fix test. * fix tests * Update screenshot. commit 58b158e Author: Stefan Giehl <[email protected]> Date: Mon Feb 25 15:29:44 2019 +0100 language update (matomo-org#14134) commit a7510a3 Author: Christian Schmidt <[email protected]> Date: Mon Feb 25 14:20:29 2019 +0100 Do not render until form is initialized (matomo-org#14016) * Do not render until form is initialized * Update screenshot commit 4f7150f Author: diosmosis <[email protected]> Date: Mon Feb 25 04:49:23 2019 -0800 Hide View Detailed Visitor Log link when realtime visits widget is embedded. (matomo-org#14077) * Hide View Detailed Visitor Log link when realtime visits widget is embedded. * Use disableLink parameter for all widgets in standalone dashboard. * Fix disableLink check commit f32a681 Author: diosmosis <[email protected]> Date: Mon Feb 25 02:07:15 2019 -0800 Make access level filter in user permissions edit default to showing all websites. (matomo-org#14078) * Make access level filter in user permissions edit default to showing all websites. * update UI tests commit a0b260c Author: Thomas Steur <[email protected]> Date: Mon Feb 25 22:46:59 2019 +1300 Noindex, nofollow for login page and tracker default output (matomo-org#14121) * Noindex, nofollow for login page * Update Response.php * update tests * update UI files commit ddd5ef0 Author: Thomas Steur <[email protected]> Date: Mon Feb 25 21:45:14 2019 +1300 Tweak default tracker text that explains Matomo (matomo-org#14107) * Tweak default tracker text that explains Matomo Shown when no tracking request is sent to Matomo. cc @mattab * update tests commit 2b3bb2c Author: diosmosis <[email protected]> Date: Sat Feb 23 18:40:21 2019 -0800 In segmented visitor log, allow opening segment editor in new tab w/ segment used. (matomo-org#14092) * In segmented visitor log, allow opening segment editor in new tab w/ segment used. * Remove empty title. commit c02e6ce Author: Stefan Giehl <[email protected]> Date: Sun Feb 24 03:39:26 2019 +0100 Improve site search detection in fragment query (matomo-org#13978) * Improve site search detection in fragment query * adds some tests * improve fragment handling & add more tests commit 80cb39a Author: Stefan Giehl <[email protected]> Date: Sat Feb 23 00:46:33 2019 +0100 Realtime Log UI improvements/fixes (matomo-org#14115) * fix initial tooltips in realtime log * fix icon positions in realtime log commit 72126ea Author: diosmosis <[email protected]> Date: Fri Feb 22 15:44:56 2019 -0800 Add better error message for unsupported operand error. (matomo-org#14116) commit a4bbb14 Author: Stefan Giehl <[email protected]> Date: Sat Feb 23 00:05:10 2019 +0100 Updates plugin submodules (matomo-org#14131) * updates plugin submodules * update UI tests * update submodule commit 1101fd0 Author: Lukas Winkler <[email protected]> Date: Fri Feb 22 21:05:16 2019 +0100 update cacert.pem (matomo-org#14114) commit eff1079 Author: Matthieu Aubry <[email protected]> Date: Thu Feb 21 13:18:46 2019 +1300 3.9.0-b1 commit 8d3fe62 Author: diosmosis <[email protected]> Date: Mon Feb 18 06:02:29 2019 -0800 Enable fingers crossed handler via INI config and show backtrace in logs/archive api output (matomo-org#13923) * Add config to use FringersCrossedHandler (untested) * Get to work in different contexts. * Add changelog note. * Make sure more exceptions make it to the logs, make backtrace include previous exceptions, do not use screen writer if in cli mode, always print backtrace if in CLI mode and archivephp triggered. * Add log capturing handler. * Remove options from global.ini.php sibnce they may be temporary. * Add UI test triggering an error w/ the screen handler. * Add some more log statements, ignore logs in screen writer, replace part of message in ExceptionToTextProcessor instead of whole message. * Add missing license. * Update changelog, move new item to 3.9 * Fixing some integration tests. * Fix another unit test. * One more test fix. * Try to get rid of xss testing warning. * Try again to get rid of warning. * Try again to get rid of warning. * Try again to get rid of warning. commit 99d42f5 Author: diosmosis <[email protected]> Date: Mon Feb 18 03:16:34 2019 -0800 Add spans to realtime visitor log tooltips. (matomo-org#14050) * Add spans to visitor log tooltips. * Extra newline for tooltip. commit 7cf47ff Author: Stefan Giehl <[email protected]> Date: Fri Feb 15 10:12:10 2019 +0100 language update (matomo-org#14103) commit 232f37b Author: Matthieu Aubry <[email protected]> Date: Thu Feb 14 16:56:25 2019 +1300 Remove mention to InterSites plugin which doesn't exist anymore commit 056027d Author: Thomas Steur <[email protected]> Date: Thu Feb 14 10:14:33 2019 +1300 Let plugins change the report type in sendReport event (matomo-org#14098) This way I can set the `reportType` to eg `foobar` so it won't be handled and won't be sent. Needing this feature... Was going to add otherwise another parameter or another event but having two events with similar `sendReport` name may be confusing and another parameter could cause issues. No need to document this any further I would say. @diosmosis can you have a quick look? commit 06c334b Author: diosmosis <[email protected]> Date: Tue Feb 12 16:47:41 2019 -0800 Update submodules. commit c6cc4cd Author: Stefan Giehl <[email protected]> Date: Tue Feb 12 23:31:08 2019 +0100 Show actions in visitorlog as a (unnumerated) timeline (matomo-org#13916) * Show actions in visitorlog as a (unnumerated) timeline * use new svg icons in visitor log * use new icons also for visitor type * ligthen icons a bit * improve icon alignment * lighten icons bit more * adjust line color * Lighten svg images even further. * Updating screenshots. * show conversion count as green circle * Fixing tests. * Update expected screenshots. commit 6c4bfb8 Author: diosmosis <[email protected]> Date: Mon Feb 11 19:53:33 2019 -0800 Make segmented visitor log in sales page only display visits with orders. (matomo-org#14079) * Make segmented visitor log in sales page only display visits with orders. * Remove some uses. * Leave out invalid characters in widget unique ID. * Change widget ID as well as report ID. commit 8d59c5a Author: Lukas Winkler <[email protected]> Date: Tue Feb 12 03:04:08 2019 +0100 rename composer dependency to avoid deprecation warning (matomo-org#14089) commit 92fa86c Author: diosmosis <[email protected]> Date: Mon Feb 11 15:56:31 2019 -0800 POST to login plugin in login form (matomo-org#14081) * Instead of using referrer URL, use redirect post param so we can post to Login module. * Use actual login plugin name. * Remove sanitization for form_redirect POST value. * Couple more checks for a safer redirect. * Do not include port in host check. * Make sure hosts are not empty for more security. commit 8e9942f Author: Lukas Winkler <[email protected]> Date: Mon Feb 11 01:00:33 2019 +0100 Remove old unused files (matomo-org#14084) * remove java files * remove (probably ) unused files commit c673177 Author: diosmosis <[email protected]> Date: Sun Feb 10 15:12:20 2019 -0800 Include non-sql migrations in update dry run output. (matomo-org#14003) * Include non-sql migrations in update dry run output. * More translation updates. * Make check for by domain Matomo more robust. * Show migrations in separate boxes based on whether they are SQL or console commands. * Update two screenshots and fix test. commit f768274 Author: sgiehl <[email protected]> Date: Sat Feb 9 10:27:51 2019 +0100 improve test error message commit 433afea Author: Thomas Steur <[email protected]> Date: Tue Feb 5 18:46:38 2019 +1300 Don't access config setting directly (matomo-org#14073) Likely shouldn't cause any issue the way it is but changing it just to be safe. commit c73613e Author: Thomas Steur <[email protected]> Date: Mon Feb 4 16:03:34 2019 +1300 Optimize all archive tables only monthly (matomo-org#14068) There is no need to run this every day over all tables. commit ca6012c Author: Thomas Steur <[email protected]> Date: Thu Jan 31 16:08:01 2019 +1300 Visitor Map tooltip shows %s for "No Visit" (matomo-org#14057) fix matomo-org#14054 fixed the issue for me locally. commit 403c112 Author: Thomas Steur <[email protected]> Date: Thu Jan 31 16:00:58 2019 +1300 Use absolute path for alternative tracker file (matomo-org#14059) Currently it resulted in something like `./matomo.js` for example (which causes problems with our file sync). commit faa8c38 Author: Thomas Steur <[email protected]> Date: Wed Jan 30 13:28:01 2019 +1300 Don't hardcode plugins directory (matomo-org#14043) * do not hard code plugins directory * remove method that is not needed for now * use plugins directory in more places commit cec026c Author: Thomas Steur <[email protected]> Date: Tue Jan 29 18:07:21 2019 +1300 Add possibility to change mail transport through DI (matomo-org#14041) * Add possibility to change mail transport through DI * Fix test. commit c826e18 Author: diosmosis <[email protected]> Date: Mon Jan 28 21:05:26 2019 -0800 Set isHtmlMessage for more exceptions. (matomo-org#14024) * Set isHtmlMessage for more exceptions. * Escaping file names. commit 75cf2d2 Author: Stefan Giehl <[email protected]> Date: Mon Jan 28 10:05:31 2019 +0100 fixes typo in translation commit 02247d0 Author: Matthieu Aubry <[email protected]> Date: Mon Jan 28 20:02:19 2019 +1300 3.8.1 ## [Matomo 3.8.1 Changelog](https://matomo.org/changelog/matomo-3-8-1/) This Github download (below) is only meant for developers and it will require extra work to install it. * Latest stable production release can be found at https://matomo.org/download/ ([learn more](https://matomo.org/docs/installation/)) (recommended) * Beta and Release Candidates releases can be found at https://builds.matomo.org/ ([learn more](http://matomo.org/faq/how-to-update/faq_159/)) commit b85cadc Author: Matthieu Aubry <[email protected]> Date: Mon Jan 28 19:49:11 2019 +1300 3.8.1-rc1 commit ae066f8 Author: diosmosis <[email protected]> Date: Sun Jan 27 20:26:34 2019 -0800 Update submodule for test commit 315117f Author: diosmosis <[email protected]> Date: Sun Jan 27 19:28:47 2019 -0800 Make sure any precreated session cookies on the wrong path are deleted (matomo-org#14026) * Make sure if session cookie path in INI config is different from path in php.ini that any precreated session cookie on the wrong path is deleted. * Only do cookie change if not posting. * Check against final value * Switch to simpler approach of changed session name, since it needs to be changed anyway. commit 0501b17 Author: Lukas Winkler <[email protected]> Date: Mon Jan 28 03:58:46 2019 +0100 update JShrink (followup) (matomo-org#14011) * update JShrink * remove old library * Update submodule. commit 0cdfff7 Author: Thomas Steur <[email protected]> Date: Mon Jan 28 13:24:02 2019 +1300 Make sure to compare password with unsanitized password (matomo-org#14033) commit 60adbbc Author: diosmosis <[email protected]> Date: Sun Jan 27 16:08:41 2019 -0800 Show url and page title on different lines in realtime visits widget & show tooltip more quickly (matomo-org#14005) * Display url and page title on different lines and truncate url. * Reduce tooltip delay so it shows up quicker. * Remove truncate. * Apply tooltip change to realtime visits widget only. * Detect empty actions. commit 0f5b231 Author: diosmosis <[email protected]> Date: Sun Jan 27 15:50:30 2019 -0800 Allow DataTables to disable generic filters gthrough a datatable metadata. (matomo-org#14004) * Allow DataTables to disable generic filters gthrough a datatable metadata. * Add unit test for metadata. commit c4caca4 Author: Thomas Steur <[email protected]> Date: Mon Jan 28 10:34:10 2019 +1300 Update Tag Manager plugin (matomo-org#14029) commit 1c175e8 Author: Stefan Giehl <[email protected]> Date: Sun Jan 27 16:58:45 2019 +0100 language update (matomo-org#14030) commit 1419621 Author: diosmosis <[email protected]> Date: Sat Jan 26 19:23:03 2019 -0800 Add method to get ecommerce items to tracker (matomo-org#14028) * Add getEcommerceItems() method to tracker so users can see what was added/removed. * doc twea * Update minified JS. * Fix test. commit 7e89540 Author: Stefan Giehl <[email protected]> Date: Sat Jan 26 17:24:25 2019 +0100 updates url in translation commit 6f9188e Author: Stefan Giehl <[email protected]> Date: Fri Jan 25 19:14:13 2019 +0100 Disable Transitions feature for totals row (matomo-org#13958) commit bd22c3e Author: diosmosis <[email protected]> Date: Thu Jan 24 16:46:58 2019 -0800 Do not enable brute force detection during update process. (matomo-org#14001) * Do not enable brute force detection during update process. * Try detection through checking for updates. * Do not enable brute force detection until version is successfully updated to 3.8.0. * $dbSchemaVersion may be false commit 4936fa2 Author: Thomas Steur <[email protected]> Date: Fri Jan 25 13:40:03 2019 +1300 When running cron jobs or console commands, and 2FA is forced for everyone, do not fail (matomo-org#14013) * When running cron jobs or console commands, and 2FA is forced for everyone, do not fail For tests we still enable 2FA as otherwise things wouldn't really be testable. * Add missung use statemtn. commit d17e701 Author: Thomas Steur <[email protected]> Date: Fri Jan 25 12:47:53 2019 +1300 Make config for sessions more clear it is still possible to use PHP configured sessions (matomo-org#14014) commit fe615e3 Author: diosmosis <[email protected]> Date: Thu Jan 24 15:30:33 2019 -0800 Fixing build (matomo-org#14007) * Fix format of scheduled report tests. * Fixing goal name regression. * Fix change unneeded for evolution graph. * decode in segment generator to make up for value encode * Update two expected screenshots. * Update two more screenshots. commit d95e2d0 Author: Thomas Steur <[email protected]> Date: Fri Jan 25 10:09:05 2019 +1300 Only migrate GoogleAuthentication if plugin is activated (matomo-org#14022) fix matomo-org#14020 see https://forum.matomo.org/t/post-upgrade-superuser-login/31401

mattab · 2019-12-02T03:26:12Z

should we maybe create a follow up issue for 3.9 or 3.10 to enable it by default if our tests go well?

Makes sense, will do so once this PR is approved.

@diosmosis have you created issue or could you do it? Be great to enable it eventually, once we know it works well..

diosmosis · 2019-12-02T14:14:37Z

@mattab created an issue in 4.0: #15220

diosmosis added the Needs Review PRs that need a code review label Dec 30, 2018

tsteur reviewed Dec 30, 2018

View reviewed changes

tsteur added this to the 3.9.0 milestone Dec 30, 2018

diosmosis changed the title ~~Enable fingers crossed handler via INI config~~ Enable fingers crossed handler via INI config and show backtrace in logs/archive api output Dec 30, 2018

tsteur reviewed Jan 3, 2019

View reviewed changes

diosmosis mentioned this pull request Jan 7, 2019

Revisit INI config recognized in #13923 after merged #13937

Closed

tsteur reviewed Jan 7, 2019

View reviewed changes

diosmosis added 6 commits February 14, 2019 18:21

Add config to use FringersCrossedHandler (untested)

46b7ec5

Get to work in different contexts.

ca8f85b

Add changelog note.

d97a603

Make sure more exceptions make it to the logs, make backtrace include…

2cc9a02

… previous exceptions, do not use screen writer if in cli mode, always print backtrace if in CLI mode and archivephp triggered.

Add log capturing handler.

2df2a21

Remove options from global.ini.php sibnce they may be temporary.

9a81562

diosmosis added 3 commits February 14, 2019 18:21

Add UI test triggering an error w/ the screen handler.

dc76e4b

Add some more log statements, ignore logs in screen writer, replace p…

9eaa2ef

…art of message in ExceptionToTextProcessor instead of whole message.

Add missing license.

0868898

diosmosis force-pushed the fingers-crossed branch from f95a0f7 to 0868898 Compare February 15, 2019 02:24

diosmosis added 9 commits February 14, 2019 19:28

Update changelog, move new item to 3.9

f27d654

Merge branch '3.x-dev' into fingers-crossed

10f989b

Fixing some integration tests.

e4cef65

Fix another unit test.

34cb9ed

One more test fix.

39c4c2e

Try to get rid of xss testing warning.

5741d69

Try again to get rid of warning.

1729fa4

Try again to get rid of warning.

22e36eb

Try again to get rid of warning.

9fc8abd

diosmosis merged commit 8d3fe62 into 3.x-dev Feb 18, 2019

diosmosis deleted the fingers-crossed branch February 18, 2019 14:02

fdellwing mentioned this pull request Mar 19, 2019

Notice - unserialize(): Error at offset 0 of 53988 bytes #14229

Closed

GTVolk mentioned this pull request Apr 22, 2022

[Snyk] Fix for 6 vulnerabilities DevVault/matomo#1

Open

Enable fingers crossed handler via INI config and show backtrace in logs/archive api output #13923

Enable fingers crossed handler via INI config and show backtrace in logs/archive api output #13923

Conversation

diosmosis commented Dec 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsteur commented Dec 30, 2018

diosmosis commented Dec 30, 2018

tsteur commented Dec 31, 2018

diosmosis commented Dec 31, 2018

tsteur commented Dec 31, 2018

tsteur commented Dec 31, 2018

diosmosis commented Jan 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsteur commented Jan 3, 2019

diosmosis commented Jan 3, 2019

Findus23 commented Jan 3, 2019 • edited Loading

diosmosis commented Jan 3, 2019

diosmosis commented Jan 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diosmosis commented Feb 15, 2019

tsteur commented Feb 15, 2019

mattab commented Dec 2, 2019 • edited Loading

diosmosis commented Dec 2, 2019

diosmosis commented Dec 30, 2018 •

edited

Loading

Findus23 commented Jan 3, 2019 •

edited

Loading

mattab commented Dec 2, 2019 •

edited

Loading