client: enforce max_kill_timeout client configuration

[shoenig]

This PR fixes a bug where client configuration max_kill_timeout was
not being enforced. The feature was introduced in 9f44780 but seems
to have been accidentally dropped during the major drivers refactoring.

We can make sure the value is enforced by pluming it through the DriverHandler,
which now uses the lesser of the task::killTimeout or client::maxKillTimeout.
Also updates Event.SetKillTimeout to require both the task.killTimeout and
client.maxKillTimeout so that we don't make the mistake of using the wrong
value - as it was being given only the task.killTimeout before.

Fixes #10005

[shoenig]

Spot checking

nomad.hcl

client {
  max_kill_timeout = "5s"
}

rexec.hcl

job "rexec" {
  datacenters = ["dc1"]

  group "rexec" {
    task "rexec" {
      driver       = "raw_exec"
      kill_timeout = "1m"

      config {
        command = "/bin/bash"
        args    = ["local/script.sh"]
      }

      template {
        data        = <<EOF
trap 'echo "Received SIGINT"' INT
while true
do
  echo 'Running'
  sleep 1
done
EOF
        destination = "local/script.sh"
      }
    }
  }
}

start nomad with config

➜ make dev && sudo nomad agent -dev -config=/home/shoenig/Work/tickets/max-timeout/nomad.hcl

run job

➜ nomad job run rexec.nomad

stop alloc

➜ nomad alloc stop d9

Task is correctly force-killed after max_kill_timeout

Recent Events:
Time                       Type        Description
2022-07-06T16:48:49-05:00  Killed      Task successfully killed
2022-07-06T16:48:49-05:00  Terminated  Exit Code: 137, Signal: 9
2022-07-06T16:48:43-05:00  Killing     Sent interrupt. Waiting 5s before force killing
2022-07-06T16:46:43-05:00  Started     Task started by client
2022-07-06T16:46:43-05:00  Task Setup  Building Task Directory
2022-07-06T16:46:43-05:00  Received    Task received by client

Making sure we respect the task setting if it is less than max_kill_timeout:

      kill_timeout = "2s"

Recent Events:
Time                       Type        Description
2022-07-06T16:52:04-05:00  Killed      Task successfully killed
2022-07-06T16:52:04-05:00  Terminated  Exit Code: 137, Signal: 9
2022-07-06T16:52:02-05:00  Killing     Sent interrupt. Waiting 2s before force killing
2022-07-06T16:51:15-05:00  Started     Task started by client
2022-07-06T16:51:15-05:00  Task Setup  Building Task Directory
2022-07-06T16:51:15-05:00  Received    Task received by client

[lgfa29]

Seems to be working well 🎉

I think it would be worth adding an upgrade notice. Even though this is how it was always supposed to work, this fix will impact any task with kill_timeout > 30s if the client config is not set.

[lgfa29]

Just notice that it doesn't have the backport/1.1.x label, is there a reason we're not able to backport?

[shoenig]

Added backport/1.1.x! totally just forgot

I'll add the upgrade note in a follow up PR ... don't wanna miss this chance for a perfect score on CI

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.