Roles of Job vs Execution Context

[godber]

This has been bothering me for a while and I know I've brought it up before but now I think I realize the reason the way we currently store and validate jobs is not quite right given that we also have this notion of the execution context. The thing that bothers me is that a users job gets validated and expanded and stored as a "Job" and then copied over as an execution context. I suggest that we change how this works and only ever store the users "raw" (unvalidated) job on the job object and then do the validation and expansion on the execution context. Conceptually speaking the job represents the users unmodified request and the execution context represents the the expansion of the job and thing that actually gets executed.

The realization I had was that the validation/expansion step then more tightly couples the job to implementation details of a specific version of the code. You could be left with jobs that don't run after upgrading teraslice or with defaults from earlier versions. Whereas if the job had not been validated/expanded it would be more likely to run after code changes. Now, I don't consider this a real driving reason to make this change, it is more of a hint or clue that the current behavior is not quite right.

I realize this complicates the validation at submission time but that is probably manageable.

[kstaken]

I agree with this and this behavior is definitely causing a few issues.

[kstaken]

This isn't fully resolved. Updating existing jobs still converts names to IDs and stores the validated job.