archive.sh script should loop over all websites and trigger archiving in a separate request for each

[mattab]

Currently, archive.sh simply loops over the periods and triggers archiving for all websites: https://github.com/piwik/piwik/blob/master/misc/cron/archive.sh

because there are memory leaks issues during archiving, see #766, when a Piwik has hundreds or thousands of websites in the system, the memory usage peaks and can lead to ridiculous values.

Until #766 is addressed and as a useful safe guard, we should looop over each website and trigger archiving separately for each. This is a quick fix that will help many users get past the memory usage issues.

[anonymous-matomo-user]

Subscribing to this ticket as I would really appreciate that method. Let me know if I can / should test something :)

[mattab]

(In [2025]) Fixes #1227
the archive script now loops over all websites and triggers a different request for each archive, helping out with the archive memory exhausted issue (refs #766)

[anonymous-matomo-user]

Thanks for your work on this. Overall I would say its better. Its still aching slow running over the sites and my most active site (15k visits / 20k actions per day, out of around 150k per day) is taking 2 GB to process the yearly stats.

[anonymous-matomo-user]

I tried the new archive.sh script, but it didn't work for me, because the list of site id's seems to be utf encoded it doesn't pass the isnumeric test. The convertToUnicode=0 param to the api call doesn't seem to do anything here.

[mattab]

alx, did you apply the full patch? only applying a subset will obviously fail. does it work for you?

[anonymous-matomo-user]

Oops, my bad. I didn't apply the full patch, sorry 'bout that.