Bad multithreading?

[juangea]

Hi there.

I just downloaded the binaries and I´m trying them with a small project I did recently with RC.

The first thing I noticed was that it was slower than RC; but then I checked out the CPU usage and it was between 2% and 10%, I have 16 threads with 8 cores (i7-5960x).

Is this a bug or is it normal because it depends on the complexity of the reconstruction?

Cheers!

[natowi]

I am also interested in this. Having access to a computer with a 48 core cpu I noticed that Meshroom is not faster compared to my i7. Since Meshroom can be run on a renderfarm where multiple node threads are computed in parallel, it would be a huge speed up to allow multi-thread processing on a desktop computer. At the moment the node threads are computed one at a time on all cores with low cpu usage.

[fabiencastan]

Which node are you talking about?

[natowi]

Is it possible to run the Chunks in the FeatureExtraction, FeatureMatching, PrepareDenseScene, DepthMap, DepthMapFilter nodes in parallel? At the moment they run 0->x.

Also multiple graphs can not be run in parallel...

[fabiencastan]

Each node has internal multi-threading and checks the amount of RAM available to estimate how many threads can be used within the RAM limits.

If you run multiple chunks in parallel, it will start to swap and then probably crash.

[natowi]

Is there a way to check how many threads are used? I can run a test on the 48 core machine with and 512GB ram tomorrow. Last time computation was not faster compared to my i7 machine with 32ram.

[fabiencastan]

Oh yes, on very large computers like that, the chunks are probably too small and it could make sense to change them or compute multiple ones in parallel.

[natowi]

Would it be possible to add an option to manually enable parallel computation in the future?

[fabiencastan]

yes but how to estimate how many of them can be computed in parallel... as the memory requirements are very specific to each node.

[natowi]

The memory requirements can be guessed by trial and error.
This is not so different from the "red bar error" people get on low spec machines, they also need to adjust the settings to match their hardware. There could be an "chunk multi thread number" range slider in the advanced settings.

Once we have this option we could collect test data from demo datasets to estimate the memory requirements for different dataset sizes. Similar to this.

[natowi]

Here are the results for a test with 2000 images:

• FeatureExtraction/ultra: 2h, CPU up to 100% and up to 200GB ram
• FeatureMatching (10^9 features), 100 chunks, 3h per chunk... (aborted) resources for one chunk: CPU@1GHz 5% ram 26GB -> computer has the capacity to run 10-20 chunks in parallel.

[natowi]

Option to make chunks optional #778