Add support for SGE resource request

[rernst]

Hi,

Our cluster setup forces us to specify sge resources when submitting job, for example h_rt and h_vmem. Therefore, I added an -l or --resource option to runWorkflow.py through makeRunScript.py.

I have tested my change on our sge cluster and it works fine.

Robert

[ctsa]

Thanks for highlighting this issue and submitting a workaround, SGE portability outside of our own clusters hasn't received as much attention as it should recently. I would like to understand a little more to see if we can adjust pyflow to better handle your case as well.

Right now pyflow will set h_vmem automatically based on each task's memory requirement. If your cluster also requires h_rt, this is one case where it would be difficult to set automatically, and the flag you've added in this PR might provide a better solution.

Is h_rt the only other setting you need for your SGE system or is there a fairly long list? My only concern with accepting this PR directly is that it creates the potential for complex bugs where the requested resource settings conflict with settings that pyflow is already submitting (this is why queue support was wrapped behind a fairly narrow option). We could reduce this problem by making the --resource option a bit less visible in some way, to clarify this is an advanced feature.

[rernst]

Thanks for looking into this. I did not know about pyflow setting h_vmem, that is a nice feature!

When submitting jobs to our cluster we have to set h_vmem, h_rt and tmpspace. Both tmpspace and h_vmem have quite decent default values (1GB and 10GB) but runtime is set to only 10 minutes by default.

I have also thought about specifically creating a h_rt setting but this would require more settings if other user need to specify other resources, therefore I choose to mimic the -l option of qsub.

[ctsa]

Sorry for the delay bringing this into a release. A somewhat more restricted version of your patch is available starting with the v1.0.0 release. The runWorkflow.py script now includes:

    --maxTaskRuntime=hh:mm:ss
                        Specify scheduler max runtime per task, argument is
                        provided to the 'h_rt' resource limit if using SGE (no
                        default)

I hope this provides enough flexibility to workaround your SGE cluster defaults, otherwise please reopen the issue.

[rernst]

Looks good, thanks for the heads up!