finalizing scaling API work

[cgbaker]

• adding raft and state_store support to track job scaling events
• updated the ScalingEvent API to record "message string,error bool" instead of cumbersome "reason,error *string"

closed #7422

[drewbailey]

stylistic nit, Error being a bool seems unfamiliar, HasError might be a bit verbose, Successful, Failed maybe? 🤷‍♂️

[cgbaker]

yeah, we played around with a few names there... it indicates that the autoscaler wants the Nomad operator to know that something is wrong. HasError might be better. but Success/Failed doesn't feel quite right.

i wanted to use ThereIsAProblemInTheAutoscalerOrTheScalingPolicy but it seemed... not good.

[jazzyfresh]

ScalingRequest.HasError and ScalingRequest.Failed both read well, and Failed has a precedent in other events structs

[drewbailey]

lgtm

[jazzyfresh]

smol smol ting, just double checking this snapshot case denotes multiple scaling events?

[cgbaker]

these snapshots are a single entry in the table. in this case, all of the scaling events for a single job.

[jazzyfresh]

do we want to panic here? could we just fail the scaling event request and log the error?

[cgbaker]

this is the standard behavior for applying raft log entries to the state store. see, e.g.,:

nomad/nomad/fsm.go

Lines 306 to 311 in 88ff339

	func (n *nomadFSM) applyUpsertNode(buf []byte, index uint64) interface{} {
	defer metrics.MeasureSince([]string{"nomad", "fsm", "register_node"}, time.Now())
	var req structs.NodeRegisterRequest
	if err := structs.Decode(buf, &req); err != nil {
	panic(fmt.Errorf("failed to decode request: %v", err))
	}

it likely means that the raft log entry doesn't contain what the leading byte says it should contain, which likely means a version mismatch or corruption. the reason for the panic, i assume, is that state is undefined if we don't know how to apply a raft entry.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.