Add possibility to run multiple archiver in parallel

[tsteur]

Add possibility to run multiple archiver via cli in parallel to make archiving of >10k websites faster.

If a server has enough power / resources a user will be able to start for instance 5 archiver in parallel and they will all archive different site ids. Currently, all started archiver would archive the same site ids.

[tsteur]

In 5fcbc09: refs #4903 started to work on the possibility to run multiple archivers in parallel for faster archiving. There were multiple issues, for instance there were arrays of siteIds read and written in Options but options do cache all values in a class property so an update of an option does not get updated on another running archiver. Also all sites were reprocessed because of the time_before_today_archive_considered_outdated setting if the last archiving by another archivier was 10 seconds or longer ago. To prevent this only maintaining a list of to be processed siteids in db / filesystem helps so far

[tsteur]

In 3827424: refs #4903 added some comments and better naming

[hpvd]

Great idea!!
What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max by Piwik?

• this may allow max performance for piwik
• but do not slow down other tasks of the server/ the complete server...

Maybe this is not only interesting for archiving, but also for other processes in Piwik...
=> should be open a other ticket for the general task and collection of ideas for the global topic "parallelization of processes for best performance"?

[hpvd]

have opened a new global ticket for the collection of ideas:
#4905

[tsteur]

In 08a0e53: refs #4903 added tests and fixed some issues

[tsteur]

In 70cd91b: refs #4903 added cli command to trigger archiving

[tsteur]

In 77d9bd0: refs #4903 sleep 25 milliseconds not microseconds

[tsteur]

To run for instance 10 archiver in parallel one can execute the following command:

for ((n=0;n<10;n++)); do (/path/to/php /path/to/piwik/console core:archive --url=http://example.org/piwik &); done

FYI: It is not a good idea to do this in case your archiving tables are using the Myisam engine. More on this will follow

[tsteur]

In c2b6c25: refs #4903 forward archive.php to new console command

[tsteur]

In 42ff0ac: refs #4903 those tests are actually database tests

[tsteur]

In 25c3a80: refs #4903 this might fix the tests

[tsteur]

In 213b5c7: refs #4903 make archiving via web possible again, will provide a better solution later

[tsteur]

In 7a2bf71: refs #4903 better solution for not working web archiving

[tsteur]

In 4bbdd90: refs #4903 removed no longer needed methods, make sure logger uses correct log level in case logger instance was already created, if the test proxy is used do not go two directories upwards

[tsteur]

In d40e217: refs #4903 log if we resume an already existing queue

[tsteur]

In 3bfbb7b: refs #4903 forgot to commit this change

[tsteur]

In 754cf8a: refs #4903 added test for web archiving. Added screen to log writers hoping no test fails because of this

[tsteur]

In a0acaac: refs #4903 removing this lock as it does not work anyway. After locking the table the lock will be immediately released when inserting the new archiving id as there is a second get_lock call. According to MySQL and tests have proven it a lock is released when you execute a new GET_LOCK()

[tsteur]

In 934caa2: refs #4903 before executing a scheduled task reschedule it to prevent multiple executions when multiple archiver are running

[mattab]

In 9fa3b75: Refs #4903 Improving the CronArchive output messages and fixing some issues

[mattab]

In 9d330e8: Refs #4903 Show "visits this week", "visits this month" etc.

[mattab]

In 47cd0bb: Refs #4903 Fix regression + tweaks to debug message
(thanks phpstorm for telling me the variable was unused)

[tsteur]

Ran some tests in a setup with 2 nodes. One node running the PHP and one node running the DB. In this first test the PHP node was rather weak (2 cores, 8GB RAM) and it was not worth running more than 2 workers. There was no performance improvement when running 3 or 4 workers. Here it took about 40hours to archive 4k websites with 1 worker and 20 hours with 2 workers (estimated).

After running everything (PHP+DB) on the powerful DB node (16 cores, 64GB RAM) this changed. It was easily possible to run 12 workers (maybe even 16 or 20 would work, depends amount of data etc). Here it took about 8hours to archive 4k websites with 4 workers, 4.5hours with 8 workers, 3.5 hours with 12 workers (estimated).

More information:

--------------------

WEAK PHP, STRONG DB NODE

After 1 hour with 2 workers: 232/4005: 116 per worker
compared to with only 1 worker all the time 145/4005: => increase 160%

Added 3rd worker after 1hour with 2 workers...

After 2 hours with an additional 3rd worker:  449/4005: 72 per worker
compared to with only 1 worker all the time: 225/4005 ==> increase 200%

Added 4rd worker after 2hours

After 3 hours with an additional 4th worker:  676/4005: 57 per worker
compared to with only 1 worker all the time: 308/4005 ==> increase 220%

MySql load average: 0.43, 0.53, 0.58
PHP load average: 5.68, 5.19, 4.91

Estimated time to complete all 4k sites with only 1 worker: 39hours
Estimated time to complete all 4k sites with only 2 worker: 17hours
Estimated time to complete all 4k sites with only 3 worker: 18hours
Estimated time to complete all 4k sites with only 4 worker: 17.5hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 4 workers: 490/4005: 122.5 per worker
After 2 hours with 4 workers: 966/4005: 121 per worker

load average: 5.41, 4.81, 4.54

Estimated time to complete all 4k sites with only 4 worker: 8hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 8 workers: 910/4005: 113 per worker
After 2 hours with 8 workers: 1884/4005: 117 per worker

load average: 10.22, 10.16, 10.12

Estimated time to complete all 4k sites with only 4 worker: 4.5hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 8 workers: 1224/4005: 102 per worker
After 2 hours with 8 workers: 3020/4005: 189 per worker (smaller websites were coming)

load average: 9.97, 9.79, 9.72

Estimated time to complete all 4k sites with only 4 worker: 3.5hours
4K sites were actually finished after 2.25 hours as there were small websites at the end

[mattab]

Such a job well done, Thomas!

[mattab]

In 135624d: Fixes #5035 CliMulti can now trigger a limited number of concurrent requests.
This is set to three concurrent processes in core:archive command.
Refs #4903