[Enhancement] Network processing client

[natowi]

A simple Meshroom client for network processing to distribute the workload on multiple machines would be useful.

The idea is to utilize the computers you have at home/in the office/school/university/...
This could be done by directly connecting running Meshroom instances and set one as master and all the others as computing clients or by installing MR computing clients and connect them to MR.

The installation and setup should be as simple as possible.

(Not everyone has access to a renderfarm/workstation)

[fabiencastan]

That was the idea of PR #81, without re-inventing the wheel. Fireworks allows to launch a master and slaves on other machines easily.
But I never managed to finalize it...

[fabiencastan]

If you can test fireworks, I would love to have other feedback to decide if it's the right way to go!
From my test, it was easy to install and setup (I have only tested on linux for now). The UI for monitoring is limited, but as we have progression feedback within meshroom, that's not a too big issue.

[natowi]

@fabiencastan Thank you. I installed fireworks and added the fireworks submitter to Meshroom. However Meshroom does not find the fireworks installation. Do I have to add/edit a settings file somewhere to point Meshroom to fireworks? I'll give it another try in a few weeks when I have more time.

[ShalokShalom]

Interested in this

[natowi]

Where do I have to install fireworks to allow this submitter to find the fireworks module and the mongodb yaml config file?
I get the error WARNING:root:== The following plugins could not be loaded ==

• fireworksSubmitter: No module named 'fireworks'

I think I need to set an environment variable...?

--
fireworks 1.8.7
mongo 3.7.2
python 3.6.2

[thegallivanter]

Natowi did you ever get distributed processing with Fireworks set up and working? Would be a great speed up for me as I have several comparable workstations and a single workstation takes weeks processing my projects. Could cut down my processing time to a single week.

[natowi]

@thegallivanter No, I did not figure out how to link Meshroom and fireworks (yet), but I will continue looking into this, as is on the to-do list for the documentation. This should not be too difficult, but without an (working) example it is time consuming to figure out.

[natowi]

Reference #213

[natowi]

@sergiy-nazarenko I read you successfully use meshroom on a render farm. Can you describe your render farm setup? The documentation (wiki) on this feature is quite thin and could use some input.

[sergiy-nazarenko]

Hej, natowi
Sure, I can describe. I actually modified the simpleFarmSubmitter.py , here is the link of our slurmFarmSubmitter :
https://raw.githubusercontent.com/sergiy-nazarenko/hafarm/0.3.3/scripts/meshroom/submitters/slurmFarmSubmitter.py
We use slurm on render farm. Slurm needs bash script to load jobs with specific comments such #SBATCH (https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html) . slurmFarmSubmitter converts the meshroom nodes to those scripts through python + jinja template
https://github.com/sergiy-nazarenko/hafarm/blob/master/slurm_job.schema

Cheers,
Sergiy

[spreck]

@natowi @sergiy-nazarenko
Is there any more info on how to implement this?

I looked through the linked documents, but frankly they are beyond my level of expertise. If this could be made more easily accessible that would be great. Any tips on how to go about using multiple computers for meshroom reconstructions?

[natowi]

@spreck https://github.com/alicevision/meshroom/wiki/Renderfarm-submitters