evaluations list pagination and filtering

[tgross]

Fixes #11621

Add pagination to the Eval.List RPC by checking for pagination token
and page size in QueryOptions. This will allow resuming from the
last ID seen so long as the query parameters and the state store
itself are unchanged between requests.

Add filtering by job ID or evaluation status over the results we get
out of the state store.

Add a NextToken field to the structs.QueryMeta so that we have a
common field across RPCs to tell the caller where to resume paging
from on their next API call. Include this field on the api.QueryMeta
as well so that it's available for future versions of List HTTP APIs
that wrap the response with QueryMeta rather than returning a simple
list of structs.

---

Note to reviewers: this is probably best reviewed commit-by-commit. It's a large PR but most of the LoC is tests. @ChaiWithJai I've added you as a reviewer mostly so you can see the implications for how the UI will need to do paging.

[lgfa29]

Nice!

Do you think we're at the point where it would be worth documenting the general flow of pagination in https://www.nomadproject.io/api-docs?

[DerekStrickland]

Found one copy-paste variable naming issue. Made some observations on the filtering method itself around readability and comments.

[lgfa29]

After reading @DerekStrickland's comments I went over the code again to use the api package in a sample app and I noticed that its missing handling the new values in setQueryOptions.

This is the sample I used, but a test in api/evaluations_test.go would be helpful as well.

package main

import (
	"fmt"

	"github.com/hashicorp/nomad/api"
)

func main() {
	c, _ := api.NewClient(api.DefaultConfig())
	eval := c.Evaluations()
	lastToken := ""

	for {
		q := &api.QueryOptions{
			LastToken: lastToken,
			PerPage:   1,
		}

		l, qm, _ := eval.List(q)
		fmt.Printf("%v\n", l[0].ID)

		lastToken = qm.LastToken
		if qm.LastToken == "" {
			break
		}
	}
}

[tgross]

I noticed that its missing handling the new values in setQueryOptions.

Good catch. That also means we have a bug in the existing CSIVolumes.ListExternal API 🤦

[tgross]

Do you think we're at the point where it would be worth documenting the general flow of pagination in https://www.nomadproject.io/api-docs?

I definitely considered that but I figured I'd do that once we had a second example?

[tgross]

Ok @DerekStrickland @lgfa29 this should be ready for re-review. Thanks for sticking this one out... it's been a slog 😁

[DerekStrickland]

[DerekStrickland]

It dawned on me last night, that we might need to keep PreviousToken too. This is not a blocker. Just a thought for conversation.

[tgross]

We can't actually page backwards with the pager though -- you just call it with whatever the previous "next token" was (or with no token for the first page). So for now I'm going to call YAGNI on that and we can see if callers end up really needing it. 😀

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.