Jobspec HCLv2 Support

[schmichael]

Nomad uses hcl v1, a version that's in maintenance mode now. HashiCorp has introduced a new major version, hclv2. hclv2 has introduced many nice features: basic templating, variable interpolation, and functions. Terraform and Packer have adopted hclv2 already and Nomad should follow suit.

The change introduces a risk: hcl2 is not drop in replacement for hcl1. We risk breaking existing hcl configurations by using hcl2.

Nomad uses hcl in few places:

• Agent configurations - https://www.nomadproject.io/docs/configuration
• ACL Policies - https://learn.hashicorp.com/nomad/acls/policies#acl-policy-specification
• Job specification - https://www.nomadproject.io/docs/job-specification

The agent configuration and ACL policies are low risk to migrate first. They are very constrained, and easily testable. Also, the agent configuration backward compatibilities are straightforward to detect when operators test/vet an upgrade - if hcl2 has syntax incompatibility, the agent will fail to start and the operator can fix it.

The Job specification hcl is more nuanced. Nomad job submission endpoint takes in a special json format rather than hcl! The hcl is only used by nomad job {run|plan} commands and /jobs/render API endpoint.

Adopting hcl2 will primarily affect users of the CLI. Any incompatibilities introduced by hcl2 will NOT affect API job submission tools/frameworks that typically use the json format. Also, Incompatibilities will not affect operators running old nomad CLIs hitting servers that adopted hcl2.

Backward compatibility constraints

hcl2 introduces few incompatibilities - this section serve as a tally of them and how they affect Nomad:

• hcl2 blocks attributes must be unquoted - hcl2 forces operator to use attribute assignment syntax

# in agent configuration
## valid in hcl1 but not hcl2
http_api_response_headers {
  "Access-Control-Allow-Origin" = "*"
}

## hcl2 use map assignment
http_api_response_headers = {
  "Access-Control-Allow-Origin" = "*"
}

# in job spec configuration
# valid in hcl1 but not hcl2
env {
  "my_key" = "my-value"
}

# the following is valid in both
env {
  my_key = "my-value"
}

• hcl2 parser validates schema and fails for unexpected keys by default, unlike hcl1:

# acl policy - valid in hcl1 but not hcl2 by default
# description is't a valid field here
namespace "default" {
  description = "something that should have been a comment"
  policy = "write"
}

• json representation - we don't have solid tests for the json input of hcl, and we've been hit by this in the past.

• interpolation: more on that in the section beloow.

Interpolation

hcl2 added support for variable interpolation in a first class way; but its current approach is in conflict with Nomads multi-phase interpolation.

Unlike hcl1, hcl2 takes over interpolating variables found in text (e.g. ${NOMAD_ALLOC_ID}) in a globally consistent way. Due to lack of interpolation in hcl1, Nomad uses helper functions to interpolate each field individually. The hcl2 is a simpler approach that eliminates classes of bugs (e.g. not interpolating a new added field).

However, it requires that all variables are known ahead of time, as hcl2 expects the variables and their values to be passed when parsing the config. If a variable is not defined, hcl2 treats it as an empty value with a diagnostic error. In Nomad, we must process variables in multiple phases, e.g. some variables are known at job submission time (some user defined variables), some at scheduling time (e.g. ${attr.kernel.name}, ${node.datacenter} in constraints), and some only at execution time (e.g. NOMAD_ALLOC_DIR). See the Variable Interpolation doc for current list. These scheduling/node variables should not be interpolated when the config is processed the first time.

A single phase of interpolation is not sufficient. We must update the hcl2 library to allow for multi-phase processing of variables.

TODO:

• update ACL policy parser to hcl2
• update agent config parsing
• update job spec parser

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.