Add compiler modifier for MAX_TASKS_PER_NODE, MAX_MPITASKS_PER_NODE, and mpirun arguments

[mrnorman]

On Summit, we have an issue where the PE layout will be different for GPU and CPU runs. CPU runs should use 84 tasks per node while GPU runs should use 36 or fewer. I'd like to use the compiler as the modifier for this, e.g.,

<MAX_TASKS_PER_NODE compiler="pgigpu">36</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE compiler="pgigpu">36</MAX_MPITASKS_PER_NODE>
<MAX_TASKS_PER_NODE compiler="!pgigpu">84</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE compiler="!pgigpu">84</MAX_MPITASKS_PER_NODE>

<mpirun mpilib="spectrum-mpi">
<!-- Use a helper script to tweak jsrun options -->
<executable>/gpfs/alpinetds/world-shared/csc190/e3sm/mpirun.summit</executable>
<!-- <executable>jsrun</executable> -->
<arguments compiler="pgigpu">
  <arg name="num_tasks"> -n {{ total_tasks }} -N $MAX_MPITASKS_PER_NODE -gpu </arg>
</arguments>
<arguments compiler="!pgigpu">
  <arg name="num_tasks"> -n {{ total_tasks }} -N $MAX_MPITASKS_PER_NODE </arg>
</arguments>
</mpirun>

It seems that CIME is not currently able to use a compiler modifier for these fields when parsing the XML file, but I think this will be necessary for CIME to support Summit's configurations.

[jedwards4b]

Have you considered defining two different machines? summit and summit-gpu? I think that this might be a cleaner solution.

[mrnorman]

That's what I'm having to do now. But I don't want the user to have to specify -mach "summit" when they're already on summit. It would seem better to allow the compiler to control these things and leave the machine defaulting to "summit". It would avoid duplicating entries in the XML files as well.

[jedwards4b]

That specification of machine only needs to happen at create_newcase time. In your proposed change you would have to specify a compiler option in the same location - so what would you gain? In the case of two machines I would presume summit would be the default and the machine only needs to be specified for summit-gpu.

[mrnorman]

You avoid large duplications in the XML files...

There's no need to litter things further with duplicate batch, compiler options, modules, and environment variables when most of that will be identical.

[jedwards4b]

How much duplication is there? The compiler and compiler options are different, many of the modules will be different and I suspect many of the environment variables will be too.

[mrnorman]

I've already mentioned what's duplicated, and no the modules and environment variables are not that different. I should also mention a compiler modifier is normal for many other XML entries in the machine file. I don't think I'm requesting something foreign to current practice in the CIME infrastructure.

[jedwards4b]

We have several machines with similar issues - cori and stampede, pleiades are three I can think of. The current practice is to have multiple machine definitions.

[mrnorman]

I think that's a poor practice. If it's simply too much work to add this functionality to CIME, I suppose I can understand that. But that should be the basis of discussion, not duplicating entire machine entries because of a different compiler choice that changes only a few entries in the machine file.

[jedwards4b]

Not too much work, I am willing to yield but feel a need to argue for the current practice.

[jedwards4b]

I think that it may work to put the compiler argument in the mpilib statement - have you tried that?

<mpirun mpilib="spectrum-mpi" compiler="pgigpu">

[mrnorman]

The code above should be the only things different between the GPU and CPU runs. For modules, we just need to add cuda for GPU runs. Otherwise, they're identical. I feel it will be easier to support and cleaner in the XML files not to duplicate the machine files. That situation may be different for other machines.

I'll give the mpirun modification a try rather than the arguments. Thanks for that suggestion. I suppose the max task entries are the only things that would need a change then.

[jgfouca]

We also need to consider how to make it more clear which attribute "selectors" are available for which XML entries.

[mrnorman]

Thanks Jim. Yeah, that would be helpful as well.

[rljacob]

@jgfouca the ECP project needs this.