SignificantChanges

Significant Changes

Semicolon Normalisation

In order to keep syntactic consistency, parameter and variable statements are no longer allowed to end with semicolons. They are newline-separated.

Removal of rootstart and rootfinish

TL;DR: Use a workflow.

Nimrod is a parameterised job scheduler, it is not it’s job to handle a job dependency graph. If you need such functionality, consider using a workflow engine that has Nimrod integration such as Kepler. Additionally, the experiment master now manages ALL experiments, not just a single one, which greatly increases the complexity of the state logic. Removing rootstart and rootfinish make the code significantly simpler.

Removal of the substitution command from the Planfile syntax

The old Nimrod allowed substitutions in a so-called skeleton file. Substitutions in this file would be replaced with the values of their corresponding variable at execution time.

This functionality was removed in the new version due to the inherent complexity of such an apparently-simple operation - there are too many ways for this to go wrong:

• What if the user tries to substitute a binary file?
  • A multi-terabyte file?
    • Is the file streamed in chunks? This adds complexity such as if a substitution is at the end of a chunk.
    • What’s the optimal chunk size?
    • Inefficient for small files.
  • A small-file?
    • Load the entire file into memory and operate there.
    • Won't work for large files.
  • A file with specific escaping requirements?
    • Think SQL injection, etc.

This is now the user's responsibility to tailor any substitutions to the specific experiment. It is strongly recommended to use a specific tool such as sed or awk, as they can do a much better job than Nimrod can.

Execution Changes

The previous syntax used for execution is no longer value due to architectural changes of Nimrod. Thus, execute:node and execute:root are no longer recognised by the parser.

Combining of Planfiles and Runfiles

Overview

The old Nimrod had two types of files, planfiles, and runfiles.

• Parameters are the traditional parameter x range from 0 to 5 step 1-type statements.
• Variables are the syntax used by the old runfiles, variable x index 0 list "0" "1" "2" "3" "4" "5".

Think of a variable statement as a processed parameter statement.

Planfiles were "compiled" into runfiles. New Nimrod does a similar thing, except into a structured JSON format.

Below are examples of equivalent parameter and variable statements.

parameter x integer range from 0 to 5 step 5
variable x index 0 list "0" "1" "2" "3" "4" "5"

Runfiles contained an additional section, a listing of jobs:

jobs
	<jobindex> <var0_value_index> [<var1_value_index> [<var2_value_index> [...]]]
	<jobindex> <var0_value_index> [<var1_value_index> [<var2_value_index> [...]]]
	...
	...
	<jobindex> <var0_value_index> [<var1_value_index> [<var2_value_index> [...]]]
endjobs

Changes

The new Nimrod has done away with that and uses a slightly modified file.

Both variable and parameter statements can be mixed, and are no longer mutually exclusive. This is due to them being essentially the same thing. The primary difference is the index specification in a variable and the explicit list of jobs.

The way this is handled:

• Variables are processed first.
  • Variables must have consecutive indices, starting at 0. The statements themselves may be out of order or mixed with parameter statements.
  • There may not be any two variables with the same index.
• Each entry in the jobs table (if any) will reference a variable and the corresponding value.
• If and only if the variable statements and job indices are valid, are parameter statements processed.
• The parameter domains are expanded and jobs created.
  • If any jobs were specified initially, each will be duplicated with the cartesian product of all the parameters.