Always write number of nodes on expanse

[cbkerr]

Description

Expanse expects the `--nodes`` option on all partitions, as poorly hinted at in documentation here: https://www.sdsc.edu/support/user_guides/expanse.html

Using the Shared Partition

#!/bin/bash
#SBATCH -p shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --mem=40G
#SBATCH -t 01:00:00
#SBATCH -J job.8
#SBATCH -A <<project*>>
#SBATCH -o job.8.%j.%N.out
#SBATCH -e job.8.%j.%N.err
#SBATCH --export=ALL

export SLURM_EXPORT_ENV=ALL

module purge
module load cpu
module load gcc
module load mvapich2
module load slurm

srun -n 8 ../hello_mpi

Motivation and Context

Was getting this error:

$ python project.py submit --partition=shared --template "myexpanse.sh" -n 1
Querying scheduler...
Submitting cluster job 'Project/e8f01a59fb6540081c69624e3f6d39f1/run/2497f00b00d4ee4751b82804f9c9f360':
 - Group: run(e8f01a59fb6540081c69624e3f6d39f1)
Traceback (most recent call last):
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/scheduling/base.py", line 216, in _call_submit
    subprocess.check_output(submit_cmd, stderr=subprocess.STDOUT, text=True)
  File "/home/cbkerr/miniconda3/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/cbkerr/miniconda3/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['sbatch', '/tmp/tmpjuv4dllz']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/expanse/lustre/scratch/cbkerr/temp_project/microswimmer-assembly/signac/project.py", line 2056, in <module>
    Project().main()
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/project.py", line 5230, in main
    args.func(args)
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/project.py", line 4941, in _main_submit
    self.submit(jobs=aggregates, names=names, **kwargs)
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/project.py", line 4227, in submit
    status = self._submit_operations(
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/project.py", line 4144, in _submit_operations
    return self._environment.submit(
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/environment.py", line 346, in submit
    if cls.get_scheduler().submit(script, flags=flags, *args, **kwargs):
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/scheduling/slurm.py", line 148, in submit
    return _call_submit(submit_cmd, script, pretend)
  File "/home/cbkerr/miniconda3/lib/python3.9/site-packages/flow/scheduling/base.py", line 218, in _call_submit
    raise SubmitError(
flow.errors.SubmitError: Error when calling submission command sbatch:
sbatch: error: bank_limit plugin: Please set the nodes parameter
sbatch: error: Batch job submission failed: Requested operation not supported on this system

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• [] The changes introduced by this pull request are covered by existing or newly introduced tests.
• The package documentation and framework documentation in signac-docs are up to date with these changes.
• I have updated the changelog and added any related issue and pull request numbers for future reference.

[joaander]

You need to run tests/generate_template_reference_data.py to update the test reference data. If you have an account in $HOME/.signacrc, temporarily remove that first.

[joaander]

Thanks!

Perhaps in #819 we should make this change in the base slurm template. I don't think this would cause a problem on any systems we support. flow requires an explicit partition selection and with homogeneous partitions, we can compute the number of nodes correctly.