Backport of scheduler: annotate tasksUpdated with reason and purge DeepEquals into release/1.5.x

[hc-github-team-nomad-core]

Backport

This PR is auto-generated from #16421 to be assessed for backporting due to the inclusion of the label backport/1.5.x.

The below text is copied from the body of the original PR.

---

reflect.DeepEqual is not suitable for comparing Job structs.

By design, it

• treats nil vs empty slices/maps as different

Note that a non-nil empty slice and a nil slice (for example, []byte{} and []byte(nil)) are not deeply equal.

• compares private struct fields

Struct values are deeply equal if their corresponding fields, both exported and unexported, are deeply equal.

The use of reflect.DeepEqual has been the root of bugs a few times in the past.

#14917
#10896
#9330

This PR introduces the OpaqueMapEquals helper function as a replacement for DeepEqual for the specific cases of comparing map[comparable]interface{}, which is necessary in cases like opaque plugin config, and envoy configs. Unlike DE, OME exhibits the opposite behavior in the two scenarios above. For other (known) types, implement .Equal method on structs and compare values directly, or defer to other sensible helpers.

The tasksUpdated helper has also been modified to return a struct containing the first mis-match of fields triggering a destructive update (as opposed to a simple boolean). Folks tracking down misbehavior now need only to add a log line to understand why tasksUpdated is (or isn't) triggering updates.

Benchmark

before

nomad/scheduler on main
➜ go test -count=3 -bench=BenchmarkTasksUpdated -run BenchmarkTasksUpdated
BenchmarkTasksUpdated-24          105920             14791 ns/op
BenchmarkTasksUpdated-24           82866             14376 ns/op
BenchmarkTasksUpdated-24           89450             14674 ns/op
PASS
ok      github.com/hashicorp/nomad/scheduler    4.485s

after

nomad/scheduler on tasks-updated-annotate
➜ go test -count=3 -bench=BenchmarkTasksUpdated -run BenchmarkTasksUpdated
BenchmarkTasksUpdated-24           54816             24469 ns/op
BenchmarkTasksUpdated-24           47431             24392 ns/op
BenchmarkTasksUpdated-24           48696             24565 ns/op
PASS
ok      github.com/hashicorp/nomad/scheduler    4.436s

The slowdown is primarily caused by OpaqueMapEquals - replacing its implementation with DeepEqual gets the numbers back down. But these numbers are still negligible and the tradeoff for correctness seems worthwhile.

Closes #16290