HCL in UI: filesystem function disabled

[lubervn]

Nomad version

1.7.2

Operating system and Environment details

Amazon Linux 2023.2

Issue

We are using external yaml files to store version of artifact on s3 like this

locals {
  service_name  = "servier"
  image_version = yamldecode(file("service.yml")).version
  url_prefix    = "s3://company.s3.ap-northeast-1.amazonaws.com/service"
}

When we submit job with nomad job run job.hcl, it works properly, but when we stop & start job in nomad UI, it shows

Could Not Start Job
Failed to parse job: input.hcl:2,23-28: Error in function call; Call to function "file" failed: filesystem function disabled.

Reproduction steps

Create service.yml with the following content:

version: 0.1.1

Create job.hcl with the following lines:

locals {
  service_name  = "servier"
  image_version = yamldecode(file("service.yml")).version
  url_prefix    = "s3://company.s3.ap-northeast-1.amazonaws.com/service"
}

job "test" {
  type = "service"
  group "test" {
    task "test" {
      driver = "raw_exec"
      config {
        command = "/bin/bash"
        args = [
          "-c",
          "sleep 3600"
        ]
      }
    }
  }
}

run nomad job run job.hcl

Go to nomad UI, Click Stop job and then Start Job

Expected Result

Job starts properly

Actual Result

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[tgross]

Hi @lubervn! The file function gets called on the same host where you're running the CLI. So although we're submitting the original jobspec HCL to the UI, the file contents aren't coming with it. See HCL Parsing Context.

However, because we're now submitting the jobspec HCL so it can be shown in the UI, that makes for a misleading interaction where it looks like this will work. But I'm also not sure we want to accept the file contents directly, as they can be arbitrarily large and that could be a problem by allowing job submitters to DOS Raft. So I don't have a good answer as to a solution here, but I'm going to mark this issue for further roadmapping and discussion.

[lubervn]

@tgross before we were using 1.6.1 and it worked well, why doesn't it work in 1.7.2?

[lubervn]

every job has its Full Definition which parsed, why doesn't nomad start job with it(I guess before Nomad used that since nomad 1.6.1 worked well)

[lubervn]

@tgross I found that the issue came from that PR https://github.com/hashicorp/nomad/pull/18120/files

[tgross]

@lubervn right. Before #18120, we weren't sending the HCL at all, so when you tried to start/stop the job in the UI only the JSON version of the job existed which had the contents of the file already parsed locally on your machine.

Now that we've got the HCL in the UI, we probably need to introduce some kind of fallback behavior so that when the UI can't submit a HCL job (because of restricted functions), we submit the JSON version of the job instead. That would break editing the HCL in the UI though, so it's not quite a matter of retrying with JSON. I'm going to bring this to the attention of @philrenaud and get it added to our UI roadmap for further discussion.

[lubervn]

@tgross , should we use full_definition for restart job instead of job spec?

[tgross]

If you have files included, yeah that's the only way to do that currently.

[beninghton]

@tgross please tell if any updates on this? How to start/stop job from UI if it has file(config.json) in it's jobpsec? The only way we do it now is through redeploy via CI/CD, or we restart only allocation.

[tgross]

No updates, @beninghton. Generally speaking if there are status updates to an issue, you'll find them in the issue.