Variable Interpolation error with Consul Workload Identity

[MorphBonehunter]

Nomad version

Output from nomad version

Nomad v1.7.5
BuildDate 2024-02-13T15:10:13Z
Revision 5f5d4646198d09b8f4f6cb90fb5d50b53fa328b8

Operating system and Environment details

Arch Linux

Issue

Today i try to migrate to the workload identity mechanism.
While the vault migration went well, the consul part fails on the variable interpolation stuff.

For easiness and simple change possibility i write my jobfiles with extensive use of NOMAD_TASK_NAME variable,
this doesn't work anymore after migration if you use this in the consul service definition part:

    task "ara" {
      template {
      ...
      }
      driver = "docker"
      config {
        image = ...
        hostname = "${NOMAD_TASK_NAME}"
        command = "/opt/ara/ara-init.sh"
        mount {
          type = "volume"
          source = "${NOMAD_TASK_NAME}"
          target = "/opt/ara/data"
        }
        ports = [
          "http"
        ]
        labels {
          cpu_shares = "${NOMAD_CPU_LIMIT}"
          container_hostname = "${NOMAD_TASK_NAME}"
        }
      }
      service {
        name = "${NOMAD_TASK_NAME}"
        tags = [
          "${node.class}",
          "traefik.enable=true",
          "traefik.http.routers.${NOMAD_TASK_NAME}.entryPoints=https",
          "traefik.http.routers.${NOMAD_TASK_NAME}.tls=true",
          "traefik.http.routers.${NOMAD_TASK_NAME}.rule=Host(`${NOMAD_TASK_NAME}....`)",
          "traefik.http.routers.${NOMAD_TASK_NAME}.middlewares=compression@file,secureheaders@file,local-access-only@file",
          "traefik.tags=host_${attr.unique.hostname}"
        ]
        port = "http"
        check {
          name = "${NOMAD_TASK_NAME}"
          type = "http"
          method = "HEAD"
          path = "/"
          interval = "20s"
          timeout = "5s"
          check_restart {
            limit = 3
            grace = "10s"
          }
        }
      }
    }

After that the following error occurs:

failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service ${NOMAD_TASK_NAME}: Unexpected response code: 500 (rpc error making call: invalid bind name: "${nomad_task_name}")

Reproduction steps

Try to use NOMAD_TASK_NAME in service name definition with active workload identity settings.

Expected Result

The job should start like it does before the migration.

Actual Result

The job fails.

I'm not sure if this is a corner case and the usage of variable interpolation in this scenario isn't something i should do, but as it worked before i assume this is a bug.

[tgross]

Hi @MorphBonehunter! Yeah that looks like a bug for sure (and a regression in any case). We must be missing the interpolation step in the new code. I'll dig into this to verify and then mark this issue for roadmapping (or push a fix sooner rather than later if it's simple).

[tgross]

Hi @MorphBonehunter, I had a chance to investigate this further and it looks like our new consul_hook that we use to derive Consul tokens for WI is indeed missing a call to interpolate. But the identity_hook that signs the workload identity also needs that interpolation step, otherwise we just end up with an error earlier in the process.

I've run out of time for this week to finish my fix (see my working branch), but I should be able to pick it back up next week.

[MorphBonehunter]

Hi @tgross thanks for the update.
Is there any possibility that this also could happen in the vault hook? I do not have an sample for this and also no idea where this could be used but maybe 🤔

[tgross]

Yeah, it'll almost certainly impact Vault as well because of the missing interpolation in the identity_hook. So that'll get picked up in the same work, but I'll double-check (and ensure we have test coverage) that the vault_hook is working fine with interpolation.

[MorphBonehunter]

Hey @tgross are there any news on this issue?

[tgross]

Sorry @MorphBonehunter I didn't get a chance to follow-thru on this. I've got a branch that fixes the issue in the Consul hook but not the WI hook (which is a little trickier). I'll try to get this resolved in the next week.

[tgross]

Hey @MorphBonehunter I've got a draft PR up #20344 which I've still got to do some end-to-end testing on. I'll hopefully have that ready for review tomorrow with an eye towards getting into the next patch release. Thanks for your patience!

[MorphBonehunter]

Hi @tgross, that sounds great. I didn't want to create any pressure here, i just want to ask :)

[tgross]

So the patch I've got up is close but not quite enough, and I'm running into a design issue I'm going to have to work out. I can get all the signing requests from the client to be interpolated just fine. But when it comes to getting those identities signed, we send an "identity handle" up to the server and tell it to sign the identity from the jobspec that matches that handle.

The problem is that the identity handle in the request is interpolated so it won't match the identity handle from the identity in the server's version of the jobspec (which is derived from the service block here). And on the server we can't interpolate anything other than information the server knows that's generic to the jobspec. Ex. we could interpolate $NOMAD_TASK_NAME but we can't interpolate $NOMAD_NODE_ID because that's per-alloc. Even if we could fix the handle, the identity block on the server would also need to be interpolated so that we're not signing for a Consul service named literally ${NOMAD_TASK_NAME}.

I'm bringing this back to the team for some internal discussion around what to do with it, and will report back once I know more.

[tgross]

Ok, I've got that problem fixed in #20344 by adding a field to the WIHandle struct that we pass in the signing request that includes the client-interpolated version of the text, without mangling the original handle. I've smoke-tested that and I've marked the PR for review by my colleagues.

[MorphBonehunter]

@tgross Thanks for figuring this out, great work.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.