Improve multi threading in archiving to be more linear

[mgazdzik]

Currently we can trigger manually many core:archive and each of them would trigger separate process for archiving idsites using common queue. However, each of those commands can spawn up to 3 (or more - if changed in file) processes computing segments. This causes that we cannot trigger as many archiving processes for idsites as we would like, because in worst case we will end up having 3x more processes computing segmented data at the same time. Therefore it would be good to have common limit saying how many process total can be spawned. Basically it should work the same for idsites as it works for segments now. That way we would be able to set top limit of process number regardless it's working on idsites of segments. Also it would allow easier managing number of processes - instead of multiple lines in crontab, only change single param to increase number of threads.

[hpvd]

maybe this could fit also into this topic:
"Parallelization of processes for best performance"
#4905

[mattab]

See related ticket where this originates from: #5396

[mattab]

Maybe we need a new parameter --max-threads=X that would limit the max number of concurrent segment processing threads started by this main core:archive thread?

[mgazdzik]

Yes, I think it would be what is needed. Also it would be great if this parameter would limit total number of threads, i.e. - we can process 10 segments or 10 sites or mixed in any other proportion.

[hpvd]

of course we like that idea :-)
see #4905
.......
What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max. by Piwik?
-this may allow max performance for piwik
-but do not slow down other tasks of the server/ the complete server
........
=> Maybe the max thread value could be a global thing?

[mattab]

It's a bit more complicated than we initially thought. I'm moving this to 2.8.0 to prevent a change in core:archive prior our release due Wed.

[mattab]

Moved to 2.10.0 as we don't have enough time left

[mattab]

I'm moving out of current milestone, because we need to think a bit more about this project, in particular our mid-term goals around scheduling archiving jobs.

Notes:

• The feature of having concurrent / threaded / queued jobs, does not need to work in core
• it can be done in a plugin, provided by Piwik PRO. Doing such added complexity as a plugin re-inforces the Piwik platform and keeps core small is our goal List all the code that should be moved from Piwik Core to specific Plugins #6782
• for 99.9% of users, running jobs synchronously (or with our basic threading already implemeted) works very well
• this issue is tightly coupled to other issues of core:archive at scale such as Computation load after adding new segments on long existing Piwik instance #6638 and Make archiving faster when thousands of websites with low or no traffic #5922 .... maybe we could somehow find a job queue library (laravel queue? the symfony bundle? ...) that we can reuse.
  • dont want to reinvent the weel, it's time consuming to create tests, docs, good code for such feature
  • we have requirements around scheduling thousands of jobs, with priorities, etc. dont want to implement this?
  • If there is an existing job queue, that doesnt yet have mysql support, maybe we could add such support. Or if we can find one on redis, maybe we could use redis. whatever

Maybe this issue depends on discussion in #6638

cc @diosmosis @mnapoli @tsteur

[mattab]

I guess here what we need is a simple new plugin that implements a Job queue for certain climulti:request calls. This is kinda similar to how QueuedTracking has a Redis queue with pending requests.

[mnapoli]

@mattab I don't understand how the work queue is going to help? I think I don't understand the issue actually, from what I could gather from the ticket the problem is because of too many threads/not enough control on those threads?

[mattab]

Right now each core:archive script will trigger 1-N new calls and there could be already some running in the background, which could overload the server.

Having a queue (eg. FIFO) gives us ability to decide process 1 or N jobs at a time (at most) giving us a controlled environment. does it make sense?

[mnapoli]

Thanks I see the point of the queue now.

each core:archive script will trigger 1-N new calls and there could be already some running in the background

Is it because core:archive doesn't wait for the requests to finish processing? Or is it because we can run multiple core:archive in parallel (but in that case this is a problem we create ourselves)? Or is there another reason why "there could be already some running in the background" (assuming archiving in the browser is disabled)?

[mattab]

there are several reasons this could happen, for exahmple if users add 100 sites overnight, or many segments (see also #7483 )...

it is by design that user can trigger several archiving scripts, eg. to make better use of multiple CPUs on the server it could be useful. When a script is already running (for example it could run for days), if we block other core:archive calls, then data for 'yesterday', 'today', etc. may be missing. This would not be good solution, that's why I think we need some kind of Job Queue and then a way to order the jobs (eg. FIFO or some other logic we decide at the time)

I think I will leave it in Mid term for now until we hit again a problem that will require this solution.

[mattab]

Not needed for now