[potential-bug] raw_exec artifact not set as executable

[jrasell]

Nomad version

Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f)

Operating system and Environment details

Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core)
Release:	7.5.1804
Codename:	Core

Issue

When using the raw_exec driver and downloading an artifact binary, the binary is not getting set as executable.

The file permissions:

-rw-r--r-- 1 root root 13234400 Oct 26 09:34 nomad-toast_linux_amd64

The job error message:

Error: fork/exec /srv/nomad-client/alloc/2cd4397f-83ff-967a-049c-17d44593825e/nomad-toast-deployments/local/nomad-toast_linux_amd64: permission denied

As far as I am aware when using the raw_exec driver and downloading a binary, nomad/go-getter should be setting the file as executable.

Job file (if appropriate)

artifact {
        source = "https://github.com/jrasell/nomad-toast/releases/download/v0.0.1-rc1/nomad-toast_linux_amd64"

        options {
          checksum = "sha256:4fd86d228e539fad57de3d154c86ccab02d38a6e3d1b815fa34775351f3c287a"
        }
      }

config {
        command = ${NOMAD_TASK_DIR}/nomad-toast_linux_amd64
}

[preetapan]

@jrasell from the following reproduction steps it looks like go-getter does not make the file executable. Nomad does not directly change file permissions in the artifact stanza.

preetha@preetha-work ~ $go-getter https://github.com/jrasell/nomad-toast/releases/download/v0.0.1-rc1/nomad-toast_linux_amd64 .
2018/10/26 14:51:15 Success!
preetha@preetha-work ~ $ls -l nomad-toast_linux_amd64 
-rw-rw-r-- 1 preetha preetha 13234400 Oct 26 14:51 nomad-toast_linux_amd64

[preetapan]

@schmichael pointed out that our executor should mark the file as executable if its not already (

nomad/client/driver/executor/executor.go

Line 284 in e7f6419

if err := makeExecutable(absPath); err != nil {

). any other errors or log messages from the client you can share before it gets to this state?

[jrasell]

@preetapan thanks for the quick responses; i'll run it again tomorrow and capture any relevant logs from the client to add here.

[preetapan]

I was able to reproduce this with the following job spec file, adapted from https://github.com/hashicorp/nomad/blob/community-call-0/community/specs/hello.nomad but using raw_exec instead of exec driver

job "hello2" {
  datacenters = ["dc1"]

  update {
    max_parallel = 1
    min_healthy_time = "15s"
    auto_revert = true
  }

  group "hello" {

    count = 3

    task "hello" {
      driver = "raw_exec"

      config {
        command = "${NOMAD_TASK_DIR}/hellov1"
      }

      artifact {
         source = "https://s3.amazonaws.com/nomad-community-demo/hellov1"
         mode = "file"
         destination = "local/hellov1"
      }

      resources {
        cpu    = 500
        memory = 256
        network {
          mbits = 10
          port "web" {}
        }
      }

      service {
        name = "hello"
        tags = ["urlprefix-/"]
        port = "web"
        check {
          name     = "alive"
          type     = "http"
          path     = "/"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

[preetapan]

After playing around with the above spec file some more, got it to work after changing the command to the following change:

config {
 command = "local/hellov1"
}

That's the relative path to the task directory, use that instead of ${NOMAD_TASK_DIR} and it works.

If that doesn't work for you, please re-open this.

[preetapan]

We can make a fully interpolated path like `${NOMAD_TASK_DIR}/binary' work with a small patch to

nomad/client/driver/executor/executor.go

Line 506 in e7f6419

func lookupBin(taskDir string, bin string) (string, error) {

, so reopening this.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.