HPC Resource Redesign

[islas]

This redesign takes the concept of argpacks and extends it generally to step arguments as well as hpc arguments.

SubmitArgpacks.py hosts the generalized implementation and HpcArgpacks.py contains the hpc-specifics both for formatting as well as config structure.

• normal script argacks remain unchanged
• hpc arguments is no longer a single string input
  • A nested dictionary approach is used here
  • the first dictionary entries function as the top-level argpacks for hpc arguments
  • only the first entry under these argpacks is considered, which should be a option : {} where option is the flag to use and any further optional hpc resource specifications should be entered into the {}
  • additional hpc resources listed in the option dictionary are considered both the resource name and an argpack. If regex argpacks are used here, at hpc submission time they will be sanitized to only use the base argpack name
  • resource argpack names, regex or not, must be unique to avoid redeclaration of resources
  • outer hpc argpacks may reuse names but at final selection of argpacks only unique basenames must appear (no duplicates argpacks with different regexes)

Naturally, the join and override command line options must be updated.

The command line join under the hood has been entirely reworked to aggregate resources based on expected runtimes, much like the step thread pool. This differs from the original implementation of just taking the top N resources where N is the pool size.

The override input to the join command no longer expects a resource string as input into the config. The input SHOULD still mirror the config input, however. Instead, the input is now the equivalent of the json dictionary in the config. Argpacks referenced need only use the basename of the argpack (no regex syntax) since at final join uniqueness is maintained and regexes stripped