QueuedTracking does not handle mysql COMMIT failures

[infomotions]

Expected Behavior

The queues would be processed without error.

Current Behavior

Intermittently the Queued tracking plugin throws errors while processing its queue. When this occurs, the processing call fails and the queues grow without being relieved. Sometimes, the call will eventually go through and things will continue to process normally, though backlogs can get very large and sometimes the error persists. Below is an example of the error.

[2021-04-29 11:09:51] piwik.ERROR: Uncaught exception: /path_to_matomo/core/Tracker/Db/Mysqli.php(432): Commit failed Commit failed #0 /path_to_matomo/plugins/QueuedTracking/Queue/Processor/Handler.php(109): Piwik\Tracker\Db\Mysqli->commit() #1 /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php(208): Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->commit() #2 /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php(143): Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets() #3 /path_to_matomo/plugins/QueuedTracking/Commands/Process.php(86): Piwik\Plugins\QueuedTracking\Queue\Processor->process() #4 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\QueuedTracking\Commands\Process->execute() #5 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run() #6 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand() #7 [internal function]: Symfony\Component\Console\Application->doRun() #8 /path_to_matomo/core/Console.php(130): call_user_func() #9 /path_to_matomo/core/Access.php(673): Piwik\Console->Piwik{closure}() #10 /path_to_matomo/core/Console.php(131): Piwik\Access::doAsSuperUser() #11 /path_to_matomo/core/Console.php(82): Piwik\Console->doRunImpl() #12 /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun() #13 /path_to_matomo/console(32): Symfony\Component\Console\Application->run() #14 {main} {"exception":"[object] (Piwik\Tracker\Db\DbException(code: 0): Commit failed at /path_to_matomo/core/Tracker/Db/Mysqli.php:432)","ignoreInScreenWriter":true} {"class":"Piwik\FrontController","request_id":901677}

[2021-04-29 11:09:51] piwik.DEBUG: Loaded plugins: CoreHome, IntranetMeasurable, Actions, Referrers, UserLanguage, DevicesDetection, Goals, Events, UserCountry, GeoIp2, VisitTime, VisitorInterest, Login, UsersManager, SitesManager, PrivacyManager, Contents, BulkTracking, Resolution, Heartbeat, Marketplace, UserId, PagePerformance, CustomDimensions, AbTesting, ActivityLog, DeviceDetectorCache, FormAnalytics, MarketingCampaignsReporting, MediaAnalytics, QueuedTracking, ReferrersManager, CorePluginsAdmin, CoreAdminHome, WebsiteMeasurable, Diagnostics, CoreVisualizations, Proxy, API, Widgetize, Transitions, LanguagesManager, Dashboard, MultiSites, SEO, VisitsSummary, VisitFrequency, RssWidget, Monolog, TwoFactorAuth, Installation, CoreUpdater, CoreConsole, ScheduledReports, UserCountryMap, Live, ImageGraph, Annotations, MobileMessaging, SegmentEditor, Insights, Morpheus, Intl, CustomJsTracker, DBStats, CustomAlerts, CustomiseTranslations, CustomReports, Funnels, InvalidateReports, LogViewer, MultiChannelConversionAttribution, RollUpReporting, SearchEngineKeywordsPerformance, SecurityInfo, TasksTimetable, TreemapVisualization, UsersFlow, WhiteLabel [] {"class":"Piwik\Plugin\Manager","request_id":901677}

[Piwik\Tracker\Db\DbException]
Commit failed

Exception trace:
() at /path_to_matomo/core/Tracker/Db/Mysqli.php:432
Piwik\Tracker\Db\Mysqli->commit() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor/Handler.php:109
Piwik\Plugins\QueuedTracking\Queue\Processor\Handler->commit() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php:208
Piwik\Plugins\QueuedTracking\Queue\Processor->processRequestSets() at /path_to_matomo/plugins/QueuedTracking/Queue/Processor.php:143
Piwik\Plugins\QueuedTracking\Queue\Processor->process() at /path_to_matomo/plugins/QueuedTracking/Commands/Process.php:86
Piwik\Plugins\QueuedTracking\Commands\Process->execute() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257
Symfony\Component\Console\Command\Command->run() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:874
Symfony\Component\Console\Application->doRunCommand() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:195
Symfony\Component\Console\Application->doRun() at n/a:n/a
call_user_func() at /path_to_matomo/core/Console.php:130
Piwik\Console->Piwik{closure}() at /path_to_matomo/core/Access.php:673
Piwik\Access::doAsSuperUser() at /path_to_matomo/core/Console.php:131
Piwik\Console->doRunImpl() at /path_to_matomo/core/Console.php:82
Piwik\Console->doRun() at /path_to_matomo/vendor/symfony/console/Symfony/Component/Console/Application.php:126
Symfony\Component\Console\Application->run() at /path_to_matomo/console:32

Context

This may occur at random times and requires manual intervention with the queues to recover by disabling the queues and manually clearing them.

Your Environment

Matomo 4.2.1
PHP Version: 8.0.3

[diosmosis]

Hi @infomotions, sorry you're experiencing this and thanks for reporting the issue. It does seem like QueuedTracking doesn't handle failed COMMITs, I'll mark this as a bug.

[infomotions]

Has there been any progress on this issue? It makes the plugin unusable as once it happens in a specific queue, it can grow until redis runs out of memory and deletes records. Is there no way to clear a specific "bad" item from a queue?

[tsteur]

@infomotions to understand this issue better do you maybe any more information why the commit would fail?