Provide extended core:archive logger output messages for easier monitoring

[mattab]

As a Piwik developer
I'd like to have better archiving output
to make debugging easier.

What could be added:

• log each segment archiving with time (rather than log them all at once), Log which segments are currently being archived #7536
• ability to print API calls used to archive -> use --v to see climulti:request calls
• during archiving - which plug-in is currently being archived, -> the plugin being archived will be logged if level is DEBUG
• ability to easily print all queries executed during archiving, -> done in logger in verbose mode
• Provide better information regarding memory usage by each plugin Provide better information regarding memory usage by each plugin #6708
• while requesting from API with debug parameter - information about selected archives (idarchive, archive type, missing archives) - such debug was available in Piwik 1.*

[diosmosis]

Current status:

log each segment archiving with time (rather than log them all at once),

This is done in the #5363 pull request (#6755)

ability to print API calls used to archive (it used to work with --v option)

If debug logging level is used, the API requests will be printed out.

during archiving - which plug-in is currently being archived,

I added this a week or two ago, the plugin being archived will be logged if level is DEBUG.

ability to easily print all queries executed during archiving,

This is a duplicate of the above itme.

[mnapoli]

For the record I'm working on the logger refactoring and in CLI the log will be outputted according to verbosity options: i.e. it will not depend on the configuration anymore for logging to stdout (all other backends will have their logging level still configured with ini config).

So by default it will log to the console with INFO level, if you use -v (verbose) it will log with DEBUG level.

[mnapoli]

• maybe displaying current RAM usage, see also Provide better information regarding memory usage by each plugin #6708

Monolog provides a MemoryUsageProcessor to add the memory usage in log entries, so this part could be covered by #6622.

[mgazdzik]

👍 for this, it would be great to have insight into which segment is currently processed (for ex. processing segment ABCD 1 of 10).
In long running archivings it's hard to determine where archiving is, which segment is heavier than other (it may give us some data to analyse segments complexity impact on processing time) and if archiving is making any progress with segments.

[quba]

+1

[mattab]

good idea, see proposal in: Log which segments are currently being archived #7536

[mattab]

#7536 is last remaining sub-task. If we have any new idea to improve core:archive logger (eg. display new stats or useful info), let's create a new issue

[mattab]

All done! if you have further suggestion on how to improve core:archive log output, feel free to create new issues