Parallelization of processes for best performance

[hpvd]

For best speed on todays multicore/mulithread servers,
piwik should be able to run as many processes as possible in parallel mode.

But which are the processes which can run in parallel mode??

This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea.

it was inspired from ticket:
#4903

One side question which has to be answered:
how to tell piwik how many cores/threads should be used max?

What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max. by Piwik?

• this may allow max performance for piwik
• but do not slow down other tasks of the server/ the complete server...

[hpvd]

1. archiver in parallel (see Add possibility to run multiple archiver in parallel #4903)

[hpvd]

2. activate multicore support but also core limitation (see ticket) by default for image processing (image magic)

this has another goal but the details/code maybe interesting: http://www.daniloaz.com/en/617/systems/high-cpu-load-when-converting-images-with-imagemagick/

[hpvd]

3. activate multicore support but also core/thread limitation (see ticket) by default for zip processing

[hpvd]

maybe one should modify the title of this ticket to:
"Controlled Parallelization of processes for best performance"
because it should not only be possible to minify the time used by an action by maximized parallelization
but it is also very important to be able to control the load applied by piwik to the sever by limiting the used cores/threads

[mattab]

Thanks for the report, but I think this ticket is too broad.
so far we are only thinking of the archiver.

What parts do you mean we can parallelize?

[hpvd]

hmm maybe you can have a look in the ticket text again:
"This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea."

=> The first concrete ideas can already be found within the comments.

If theses approaches are fine, I or everyone else can add single tickets for each and make a link to these ticket within these ticket comments

And when there are one day

• no more ideas regarding the topic Parallelization / Multicore utilization and
• each approach has his own ticket
  then it's time to close this ticket :-)
  What do you think?

[mattab]

I should have left it open as it's an interesting discussion!
cheers for your active participation here also.

[mattab]

Now core:archive can be parallelised for already a few years 🚀
the parameters you can use are:

 --concurrent-requests-per-website  When processing a website and its segments, number of requests to process in parallel (default: 3)
 --concurrent-archivers             The number of max archivers to run in parallel. Depending on how you start the archiver as a cronjob, you may need to double the amount of archivers allowed if the same process appears twice in the `ps ex` output.

see ./console help core:archive or: https://matomo.org/docs/setup-auto-archiving/