Parallel execution of plugins

[kynikos]

Plugins that are safe to be executed together with other plugins (e.g. the various instances of UpdateCategoryTree) should just signal the "end" of execution before they actually finish their action, so that, when executed through the "*" button or a plugin group button, the following plugin in the chain can start in parallel.

A downside is of course the messing of the log.

Note that this can already be achieved by clicking each plugin's button separately.

[lahwaacz]

Parallel execution will also increase the server load, even more than running serially with shorter timeouts (see http://www.mediawiki.org/wiki/API:Etiquette#Request_limit).

[kynikos]

To be honest I'm just trying to solve a personal problem, because where I live now I get very high ping times to archlinux.org (0.3-0.5s), and I thought sending parallel requests would help a lot. I think I'll limit the feature to special maintenance functions that are run sporadically, possibly with a bot account. I don't think I'll apply it to normal bot plugins.

[kynikos]

Even better, I'll make this optional and disabled by default.

[lahwaacz]

Perhaps there are some other optimizations possible, e.g. combining things into a single query (not applicable to edit queries) or using a keep-alive connection (I have no idea about how to do this in javascript, perhaps it's already done)?

[kynikos]

I think I had already combined the queries as much as possible; about keep-alive connections, Javascript may simply be relying on the browser engine, and I think all modern ones use persistent connections by default, but I haven't investigated actually...
Anyway, this isn't urgent, since I can let the bot work while I do other stuff. Also, that kind of maintenance scripts should probably be ported to wiki-scripts (which wouldn't fix the problem per se, but move it to your bug report lol ;) ).

[lahwaacz]

If I'm reading the code correctly, it seems that getParentsAndInfo is still called individually per each category, while all the information can be obtained using a generator, which generally takes less requests to the server. You can also get the last revision of all "Table of contents" pages using a single query, and only then parse & update & submit in the per-page fashion.

I know that applying the above ideas will take even more work than implementing the parallel execution, but I hope they will be addressed before the maintenance burden is thrown off :P

[kynikos]

True, a generator should be able to fix that, but the plugin should be rewritten almost completely... Probably it would be more worth to spend the time to rewrite it in Python then :) I'll see how it goes, after all "it ain't broken" right now, so... ;)