Feature request: support Buildkite "Job prioritization"

[artem-zinnatullin]

As discussed with @DrJosh9000 controller should eventually add support for Buildkite "Job prioritization" so that jobs are scheduled onto Kubernetes with respect to their Buildkite priority.

Job priority is considered before jobs are dispatched to agent queues, so jobs with higher priority are assigned before jobs with lower priority, regardless of which has been longest in the queue. Priority only applies to command jobs, including plugin commands.

Right now controller seems to take jobs from Buildkite API in batches and each batch is shuffled (…?) with no respect neither to the creation time nor priority:

agent-stack-k8s/internal/controller/monitor/monitor.go

Lines 216 to 226 in 06a3601

	// Why shuffle the jobs? Suppose we sort the jobs to prefer, say, oldest.
	// The first job we'll always try to schedule will then be the oldest, which
	// sounds reasonable. But if that job is not able to be accepted by the
	// cluster for some reason (e.g. there are multiple stack controllers on the
	// same BK queue, and the job is already created by another controller),
	// and the k8s API is slow, then we'll live-lock between grabbing jobs,
	// trying to run the same oldest one, failing, then timing out (staleness).
	// Shuffling increases the odds of making progress.
	rand.Shuffle(len(jobs), func(i, j int) {
	jobs[i], jobs[j] = jobs[j], jobs[i]
	})