Implement multithreading in zimdump

[lidel]

Issue extracted from ipfs/distributed-wikipedia-mirror#66

zimdump feels slower than it could be.
Below some notes from my tests and ideas how to improve its performance.

Single thread? Lack of buffer in front of disk writes?

I have SSD but my disk I/O remains pretty slow (iotop shows pretty slow disk writes at <400 K/s!).
Tool seems to be limited by the CPU: a single core is used and is constantly at 100%. Remaining 7 cores remain unused. Looks like it is single-threaded and perhaps flushing after each write to disk?

Benchmarks

Unpacking wikipedia_en_top_mini_2019-09.zim (250M) took nearly 30 minutes:

$ zimdump -V
1.0.5
$ time zimdump -D out-zimdump wikipedia_en_top_mini_2019-09.zim
1573.26s user 47.65s system 99% cpu 27:14.82 total

$ du -sh out-zimdump
758M    out-zimdump

This is super slow comparing to rust-based multicore extract_zim from dignifiedquire/zim. It produces some errors and skips some files (tool is not maintained anymore), but is able to extract most of it under 10 seconds(!):

$ time extract_zim --skip-link wikipedia_en_top_mini_2019-09.zim --out out-extract_zim
Extracting file: wikipedia_en_top_mini_2019-09.zim to out-extract_zim

  Creating map
  Extracting entries: 243
  Spawning 243 threads
...
5.91s user 1.87s system 635% cpu 1.223 total

$ du -sh out-extract_zim
726M    out-extract_zim

Things to try

Applying some/all optimizations from dignifiedquire/zim should make zimdump much, much faster:

• builds a map of the file first, identifying individual clusters
• creates a pool of 16 workers that process clusters in parallel
• all writes to disk are buffered in memory and periodically flushed
  (in Rust this is provided by BufWriter)

[kelson42]

@lidel Thank you for this quality ticket, I'm supportive. Will do my best to get this don in January.

[momack2]

Thanks @kelson42!! Curious how things are evolving now that we're in the new year - is this still on your agenda this month?

[kelson42]

@momack2 I would like, but we are a bit short on C++ resources currently. It has been posponed to Febuary for the moment. If you can recommend someone, please tell us!

[momack2]

I don't know of any C++ devs with bandwidth, but @jnthnvctr might be able to suggest other routes to get this work increased attention. We'd really love to update our distributed wikipedia mirror with snapshots more recent than 2017... ;)

[kelson42]

@momack2 It is just a "small" delay and working already to find someone. Maybe you can retweet https://twitter.com/KiwixOffline/status/1214826834417860609

[mgautierfr]

I didn't know about extract_zim tool. It's nice to see some rust around zim (even if it is not maintained anymore).

I agree with this ticket, zim_tools is a small set of tools and we can improve it a lot.
Looping articles based on the cluster index order instead of the url order is already used in zimrecreate tool. It should not be too difficult to reuse it.
At least we would use the libzim cache system and avoid to decompress the same cluster several times.

[kelson42]

@mgautierfr Considering that this ticket will need to get the articles in the order they are in the files (to save cluster decompression and that this is as well needed by (at least) zimrecreate and zimcheck I think it would be smart to deliver this kind of iterator within the libzim. Please confirm.

[kelson42]

Most important part of the improvement will be achieved by implementing openzim/libzim#300

[kelson42]

@lidel The zimdump speed has been improved a lot (around 15x) but without adding multithreading. The reason is that we need to revamp our libzim cache strategy to really benefit of it in zimdump. For the moment this is on hold and I will move it out the IPFS project as I believe the speed is acceptable now.

[lidel]

@kelson42 I can confirm, it is now within the same order of magnitude as the rust library, and can be used for practical purposes.

wikipedia_tr_all_maxi_2020-04.zim (4.4G):

• dignifiedquire/zim: 1m13s
• zimdump v1.1.1: 2m8s

Still slower, but that will change with multithreading.

[kelson42]

@lidel Thx for sharing the benchmark.

[kelson42]

@mgautierfr @veloman-yunkan @MiguelRocha AFAIK the libzim cache has been improved to allow to get the full potential a multithreading in zimcheck and zimdump in last version 6.2.0. Therefore would that not be a good time t reconsider this ticket?

[veloman-yunkan]

@Kelson I can work on this

[kelson42]

@veloman-yunkan Good for me but @MiguelRocha had started something which I believe is available at https://github.com/openzim/zim-tools/tree/speed-up-zimdump. @MiguelRocha Do you rememember what was the code status here?

[MiguelRocha]

@kelson42 Yes at that time I created a thread pool to handle the decompression of the clusters in a multi-thread way. Since then zimdump changed quite a bit so its just a matter of re basing the branch with master and fixing potential conflicts.

[kelson42]

@MiguelRocha Thank you for the update
@veloman-yunkan The same multi-threading approach should be used as well for zimcheck.

[kelson42]

@veloman-yunkan I’m unsure about the status here. Could you enlight me please?

[veloman-yunkan]

@kelson42 I am going to do the zimcheck part over this weekend.

[kelson42]

@veloman-yunkan Any news on this front?

[veloman-yunkan]

@veloman-yunkan Any news on this front?

@kelson42 A prototype implementation was ready, but then the libzim_next branch was merged and I had to rebase my branch and resolve a lot of large conflicts. I was waiting for any ripple effects from the merged libzim_next branch to fade out before continuing/finishing the work on this.

[kelson42]

@veloman-yunkan I don't think there is any big plan for now to make big changes again. @mgautierfr has to implement openzim/libzim#397 but (beside a few other details), this is going to be the only thing to change before release mid of January. I think you can continue on this ticket.

[kelson42]

So, we need that now for the zimdump binary (it's done for the zimcheck), but this is less urgent. So we can keep that for the future.