e2e: task schedule: pauses vs restarts

[gulducat]

CE side of ENT PR: hashicorp/nomad-enterprise#2307 - task schedule: pauses are not restart "attempts"

Internal ticket: https://hashicorp.atlassian.net/browse/NET-12119

Distinguish between these two cases:

1. Task dies because we "paused" it (on purpose)
   • should not count against restart attempts, because nothing is wrong.
2. Task dies because it didn't work right
   • should count against restarts, so users can address application issues.

With this, the restart{} block is back to its normal behavior, so its documentation applies without caveat.

Output of the tests:

$ go test ./e2e/task_schedule/ -v -count=1
=== RUN   TestTaskSchedule
=== RUN   TestTaskSchedule/in_schedule
    task_schedule_test.go:47: 2025-02-10T21:04:00Z job options: start="3 21 * * * *" end="5 21"
    task_schedule_test.go:47: 2025-02-10T21:04:01Z ran job "test_task_schedule-080"
    task_schedule_test.go:50: 2025-02-10T21:04:01Z waiting up to 5s: task should start
    task_schedule_test.go:53: 2025-02-10T21:04:02Z waiting up to 1m5s: task should be killed
=== RUN   TestTaskSchedule/in_future
    task_schedule_test.go:72: 2025-02-10T21:05:00Z job options: start="7 21 * * * *" end="8 21"
    task_schedule_test.go:72: 2025-02-10T21:05:01Z ran job "test_task_schedule-163"
    task_schedule_test.go:78: 2025-02-10T21:05:06Z wait a minute
    task_schedule_test.go:80: 2025-02-10T21:06:06Z waiting up to 1m5s: task should start
    task_schedule_test.go:82: 2025-02-10T21:07:00Z waiting up to 1m5s: task should be killed
=== RUN   TestTaskSchedule/job_update
    task_schedule_test.go:102: 2025-02-10T21:08:01Z job options: start="8 22 * * * *" end="8 23"
    task_schedule_test.go:102: 2025-02-10T21:08:02Z ran job "test_task_schedule-277"
    task_schedule_test.go:108: 2025-02-10T21:08:07Z job options: start="8 20 * * * *" end="8 22"
    task_schedule_test.go:108: 2025-02-10T21:08:08Z re-ran job "test_task_schedule-277"
    task_schedule_test.go:109: 2025-02-10T21:08:08Z waiting up to 1m5s: task should start
    task_schedule_test.go:113: 2025-02-10T21:08:08Z job options: start="8 22 * * * *" end="8 23"
    task_schedule_test.go:113: 2025-02-10T21:08:09Z re-ran job "test_task_schedule-277"
    task_schedule_test.go:114: 2025-02-10T21:08:09Z waiting up to 1m5s: task should be killed
=== RUN   TestTaskSchedule/force_run
    task_schedule_test.go:135: 2025-02-10T21:08:09Z job options: start="8 22 * * * *" end="8 23"
    task_schedule_test.go:135: 2025-02-10T21:08:10Z ran job "test_task_schedule-168"
    task_schedule_test.go:136: 2025-02-10T21:08:10Z waiting up to 5s: task should be placed
    task_schedule_test.go:147: 2025-02-10T21:08:10Z waiting up to 5s: task should start
    task_schedule_test.go:153: 2025-02-10T21:08:11Z waiting up to 5s: task should stop
=== RUN   TestTaskSchedule/force_stop
    task_schedule_test.go:175: 2025-02-10T21:08:12Z job options: start="8 20 * * * *" end="8 22"
    task_schedule_test.go:175: 2025-02-10T21:08:13Z ran job "test_task_schedule-428"
    task_schedule_test.go:176: 2025-02-10T21:08:13Z waiting up to 5s: task should start
    task_schedule_test.go:187: 2025-02-10T21:08:13Z waiting up to 5s: task should stop
    task_schedule_test.go:193: 2025-02-10T21:08:14Z waiting up to 15s: task should start
=== RUN   TestTaskSchedule/repeat_pause
    task_schedule_test.go:215: 2025-02-10T21:08:15Z job options: start="8 22 * * * *" end="8 23"
    task_schedule_test.go:215: 2025-02-10T21:08:16Z ran job "test_task_schedule-630"
    task_schedule_test.go:216: 2025-02-10T21:08:16Z waiting up to 5s: task should be placed
=== RUN   TestTaskSchedule/repeat_pause/attempt_1
    task_schedule_test.go:230: 2025-02-10T21:08:16Z waiting up to 5s: task should start
    task_schedule_test.go:235: 2025-02-10T21:08:17Z waiting up to 5s: task should stop
=== RUN   TestTaskSchedule/repeat_pause/attempt_2
    task_schedule_test.go:230: 2025-02-10T21:08:18Z waiting up to 5s: task should start
    task_schedule_test.go:235: 2025-02-10T21:08:19Z waiting up to 5s: task should stop
=== RUN   TestTaskSchedule/repeat_pause/attempt_3
    task_schedule_test.go:230: 2025-02-10T21:08:20Z waiting up to 5s: task should start
    task_schedule_test.go:235: 2025-02-10T21:08:21Z waiting up to 5s: task should stop
=== RUN   TestTaskSchedule/task_dies
    task_schedule_test.go:254: 2025-02-10T21:08:22Z job options: start="8 20 * * * *" end="8 22"
    task_schedule_test.go:254: 2025-02-10T21:08:23Z ran job "test_task_schedule-237"
    task_schedule_test.go:255: 2025-02-10T21:08:23Z waiting up to 5s: task should start
    task_schedule_test.go:263: 2025-02-10T21:08:23Z waiting up to 5s: task should fail
--- PASS: TestTaskSchedule (263.66s)
    --- PASS: TestTaskSchedule/in_schedule (60.15s)
    --- PASS: TestTaskSchedule/in_future (180.29s)
    --- PASS: TestTaskSchedule/job_update (8.03s)
    --- PASS: TestTaskSchedule/force_run (3.04s)
    --- PASS: TestTaskSchedule/force_stop (3.04s)
    --- PASS: TestTaskSchedule/repeat_pause (7.07s)
        --- PASS: TestTaskSchedule/repeat_pause/attempt_1 (2.02s)
        --- PASS: TestTaskSchedule/repeat_pause/attempt_2 (2.02s)
        --- PASS: TestTaskSchedule/repeat_pause/attempt_3 (2.02s)
    --- PASS: TestTaskSchedule/task_dies (2.02s)
PASS
ok      github.com/hashicorp/nomad/e2e/task_schedule    263.665s