Nomad config validate does not check for missing host_volume directories and daemon fails to start

[maxadamo]

Proposal

nomad config validate <config_file> does not detect missing files/directories for the host_volume parameter and Nomad fails to start

Use-cases

if a directory/file is missing, nomad won't start.

Attempted Solutions

With Puppet we created a custom config validation, which checks that the paths are valid on the system:
voxpupuli/puppet-nomad#85

but I'd expect this same functionality to be available on nomad config validate

[tgross]

Hi @maxadamo! I just verified that the IsValidConfig doesn't just do a static config check and it does things like check the serf keyfile config exists if provided. So this feature totally makes sense to add for host volumes and probably other options as well.

I'm going to mark this for roadmapping but also mark it as a good first issue for community contributions.

[dttung2905]

Hi team,

I would like to attempt to fix this issue. I just added a PR above 🙏

[gulducat]

By my eye, that PR ensures that path is present in host_volume config and not an empty string, which is a nice improvement, but it doesn't check for the presence of the directory on disk.

Seems the latter is the desired outcome of the issue, so I'll go ahead and reopen.

Can repro with this (partial) config:

client {
  enabled = true

  host_volume "nope" {
    path = "/nope" # assuming your system lacks this dir
  }
}

$ nomad config validate client.hcl
Configuration is valid!

but when trying to start the agent:

$ nomad agent -config client.hcl
.....
[ERROR] agent: error starting agent: error="client setup failed: node setup failed: failed to validate volume nope, err: stat /nope: no such file or directory"

[lgfa29]

Ops, my bad I missed that in the review.

@dttung2905 would you be interested in extending the work you did in #17393 to also check if the host volume path is actually a directory that exists?

[dttung2905]

Sure @lgfa29 , please allow me some time to take a look at it again 🙏

[maxadamo]

@dttung2905 would you be interested in extending the work you did in #17393 to also check if the host volume path is actually a directory that exists?

@lgfa29 , and @dttung2905 for clarity: it's doesn't have to be a directory. It can also be a file.

[lgfa29]

Ah interesting, like a symlink to a directory? I don't think I've ever tried that 😄

[maxadamo]

@lgfa29 I just checked my configuration, to ensure that I was remembering correctly.
Maybe this is different with CSI volumes, but host_volume can also be a file, not only a directory. See below, my configuration:

    "host_volume": [
      {
        "cams_bootstrap_ldif_master": {
          "path": "/depot/cams/ldif.master",
          "read_only": true
        },
        "cams_bootstrap_ldif_replica": {
          "path": "/depot/cams/ldif.replica",
          "read_only": true
        },
        "cams_etc_ldap_master": {
          "path": "/depot/cams/etc_ldap.master"
        },
        "cams_slapd_conf_replica": {
          "path": "/depot/cams/etc_ldap.replica/slapd.conf"
        },
        "cams_var_lib_ldap_master": {
          "path": "/depot/cams/var_lib_ldap.master"
        },
        "cams_local": {
          "path": "/depot/cams/local"
        },
        "cams_comanage_conf": {
          "path": "/depot/cams/apache2/000-comanage.conf",
          "read_only": true
        },
        ...