deprecate client-side interpolation for known values in lieu of HCL2

[tgross]

In #20344 (review) @schmichael noted:

I wonder if we should deprecate this form of interpolation in this place in favor of HCL variables? I think an HCL variable with a default value could specify both the group and service names in a way that would make them static by the time they're submitted to Nomad. It would be nice to have a path away from this extremely buggy phased interpolation approach.

There are broadly two kinds of interpolation that happen in Nomad jobs:

• HCL2 variables interpolation, which populates the job specification before it's submitted to the server.
• runtime interpolation, which populates fields on the job after it's been turned into a running allocation on the client. The runtime environment is populated similarly.

This two-phase interpolation has been a source of confusion and bugs for a while now. Any field that's being interpolated on the client cannot be used by the server. One particularly confusing thing is that this information can't be used for scheduling decisions, even if the server could do the interpolation, because the allocation hasn't been placed yet! But some of that information is available already (like task name), but because we do all the interpolation on the client we haven't built the mechanisms on the server to use it.

All of which is to say this is a bit of a mess.

To untangle this, we could:

• Retire the environment variable syntax for the jobspec (we'd probably need to think about what this looks like for task args fields)
• Add all the runtime environment variables for allocation and node data as ${env.*} attributes, but not the submit-time attributes like task name and group name
• Add functions to our HCL2 that allow users to refer to submit-time attributes like task name and group name

An alternate approach could be:

• Leave all the existing syntax in place
• Rework the client's taskenv builder so that it only interpolates runtime-side variables
• Add a submit-time equivalent as the first job submit hook, that does interpolation of submit-time data like task name and group name, so that these never reach the client side uninterpolated.

Either way we could also use more documentation here. There's probably some subsets of fields that we could say "you're allowed to interpolate these fields with node metadata, and these fields only with HCL2 (or submit time)". Being able to document which those are would be nice for users and for us developers.