Need separate request/limit values for worker pods

[Fryguy]

The work done in #20193 and ManageIQ/manageiq-pods#532 introduced limits, however, we need a way to be able to specify requests as well. @carbonin laid out some details here: #20193 (comment):

So this now adds a CPU and memory limit to all the workers, but unfortunately this makes the requests default to the limits. ref

This means that by default you would need a cluster with at least 6 cores and 6Gib of memory to even schedule the default set of workers, nevermind adding provider workers or scaling. This also doesn't include the case where you set requests on the rest of the pods in the project.

When we deploy the non-worker containers (postgres, httpd, orchestrator, etc.) we default to no limits/requests, but they can be easily set in the CR at deploy time. I'm not sure how we would make a similar change for the workers. I don't think it's a good experience to have to deploy the whole app then have to go into advanced settings to mess about with resource constraints.

Maybe we have a single flag that tells the orchestrator to either add the constraints or not? Then that can be set in settings at deploy time?

We need a way for a user to specify these in the worker configuration so that it can be used in a kube deployment, but also possibly in a systemd deployment. Additionally, we might need to find a way to specify these in the operator CR, but I'd be ok with that as a follow-up exercise.

cc @jrafanie @agrare @bdunne

[kbrock]

Currently we configure the worker with hard limits:

:workers:
  :worker_base:
    :defaults:
      :cpu_threshold_percent: 50 # used for all
      :memory_threshold: 400.megabytes # default value but most override

we would like to add minimum values as well, which is called requests

we have the option of using a ruby range or using a separate value. I'm leaning to separate values since it is difficult to state (400.megabytes)..(600.megabytes)

[bdunne]

👍 to separate values

[kbrock]

PRs:

• Add a memory and cpu limits on deployments  #20661
• I think the work in Add a flag to gate worker resource constraints manageiq-pods#532 will support this change and nothing needed there

[chessbyte]

The PRs in the comment above have been merged. Are we waiting for other work or can this issue be closed now?