API Support for Topology Visualization

[DingoEatingFuzz]

It might be better to have this place to talk generally about the requirements of the topology visualization.

The API goals are to make as few requests as possible and download as many unneeded bits as possible. I guess these are always the goals of an API, but I digress.

Currently the topology visualization does this:

1. Fetch /v1/nodes to get all nodes
2. For each node, fetch /v1/node/:id to get the resources data (see Add Resources, ReservedResources, and Reserved to the /v1/nodes endpoint conditionally #8995)
3. For each node, fetch /v1/node/:id/allocations to get allocations including resources data

This results in 2n + 1 requests.

This is what I would like to do:

1. Fetch /v1/nodes?resources=true
2. Fetch /v1/allocations?resources=true

This would be just the two requests.

If I'm dreaming big, there is still a lot of information here I'm not sure I need, such as task events and node events, but this is a good starting point.

[schmichael]

These fields?

• Node.NodeResources
• NodeReservedResources

(Node.Resources and Node.Reserved are deprecated as of 0.9 although that could be made more clear.)

• Allocation.AllocatedResources

(Allocation.{Resources,SharedResources,TaskResources} are all deprecated although that could be made more clear.)

Sample Allocation.AllocatedResources JSON:

{
  "Tasks": {
    "redis": {
      "Cpu": {
        "CpuShares": 100
      },
      "Memory": {
        "MemoryMB": 300
      },
      "Networks": null,
      "Devices": null
    }
  },
  "TaskLifecycles": {
    "redis": null
  },
  "Shared": {
    "Networks": null,
    "DiskMB": 300,
    "Ports": null
  }
}

[DingoEatingFuzz]

Good to know!

The UI currently uses Node.Resources, Node.Reserved, and Allocation.AllocationResources. Looks like we should update to use Node.NodeResources and Node.ReservedResources.

It's totally understandable if you don't want to drag the deprecated fields into this new endpoint variant so I can update what we're using as part of this.

[schmichael]

Does #9055 seem to meet your needs?

I fear /v1/allocations?resources=true may be quite expensive for users with large clusters. Some future improvements that might help:

• ?terminal=false - only return DesiredStatus=running allocations. Same logic could be applied to Nodes.
• ?task_states=false - strip task states from the response as they're often the largest part of the response.

In addition or alternatively you may want to continue doing n + 1 requests: /v1/nodes?resources=true + 1 node/allocs request per node.

You could limit the n requests to just the first "page" of node results (and perhaps prefetch another page or two). Obviously this limits visualizations and sorting to Node data which might be a nonstarter.

[DingoEatingFuzz]

This meets my needs! I just updated my data fetching and verified it all.

I agree about the massiveness of the allocations payload, and both of those additional toggles would help--especially task states which can be quite large and (at least for the topo viz) I'm not even looking at.

This new n + 1 approach is interesting. It definitely makes sense to progressively download things in some manner, but I still get squeamish thinking about doing this for any cluster where n is > 100. That's a lot of requests in rapid succession.

The other other approach, if we don't want to pollute the public API with combinatorial bells and whistles, is to make this an "internal" "ui" endpoint. There's precedence for this in Vault and Consul already.

I don't like the UI getting VIP treatment, but it might be nice as a stop-gap while we figure out some API configurations we actually want to support long term.

[schmichael]

Altering projection (the fields like task_states) is pretty easy on top of what I've built, so I'll add that.

Altering selection (which objects like terminal=false) is a new thing altogether, so I'm going to skip it for now. Feel free to open up a new issue if you think it's worth going down that path.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.