WIP: Experimental k8s config (DO NOT MERGE)

[i-chvets]

WIP: Exeprimental K8S config

Description

Configuration options for CPU resources for OWGW.

Summary of changes:

• Modified code to use env vat to specify number of CPUs for generic scheduler. If not set, system value is taken.
• Modified code to use configuration parameter to specify number of CPUs. If not set, system value is taken.

[stephb9959]

I don't think you want to tell the app to use X number of cores. This should be done through k8s or anything else during deployment. Then, let the app decide how to use them. The app already knows how many cores are available. This can lead to a problem where someone changing the number of cores in k8s has no effect in the app. This would only be useful if the container would report the wrong number of cores. And for this to be useful we would also need to specify affinity. I do nothing this is a good idea. If you want we can have a chat on this.

[i-chvets]

I don't think you want to tell the app to use X number of cores. This should be done through k8s or anything else during deployment. Then, let the app decide how to use them. The app already knows how many cores are available. This can lead to a problem where someone changing the number of cores in k8s has no effect in the app. This would only be useful if the container would report the wrong number of cores. And for this to be useful we would also need to specify affinity. I do nothing this is a good idea. If you want we can have a chat on this.

Fair enough.
The issue is that app takes all cores that are available on the system, i.e. the node. It does not know about what has been requested in K8S deployment. If you are in container and look at /proc/cpuinfo you will see all the cores on the node the container is running on.
There is another approach, but it will still require similar changes, i.e. reading number of cores not from system info.
If we introduce usage of Downward API (https://kubernetes.io/docs/concepts/workloads/pods/downward-api/) this will allow the app to access what is actually being requested by K8S. I will have another experimental PR using that method and we can compare what makes better sense.

[stephb9959]

I don't think you want to tell the app to use X number of cores. This should be done through k8s or anything else during deployment. Then, let the app decide how to use them. The app already knows how many cores are available. This can lead to a problem where someone changing the number of cores in k8s has no effect in the app. This would only be useful if the container would report the wrong number of cores. And for this to be useful we would also need to specify affinity. I do nothing this is a good idea. If you want we can have a chat on this.

Fair enough. The issue is that app takes all cores that are available on the system, i.e. the node. It does not know about what has been requested in K8S deployment. If you are in container and look at /proc/cpuinfo you will see all the cores on the node the container is running on. There is another approach, but it will still require similar changes, i.e. reading number of cores not from system info. If we introduce usage of Downward API (https://kubernetes.io/docs/concepts/workloads/pods/downward-api/) this will allow the app to access what is actually being requested by K8S. I will have another experimental PR using that method and we can compare what makes better sense.

Is this real? I am shocked. So limiting anything in k8s means nothing be default? That's a new one for me. I did not know that.

[i-chvets]

Is this real? I am shocked. So limiting anything in k8s means nothing be default? That's a new one for me. I did not know that.

Well, container will be throttled and will be killed if it goes over memory limit, for example. But nothing prevents the app from creating more threads that there are cores (which is what is actually happening). But when app is oversubscribing, it is getting throttled and all those threads do not have chance to run.
If K8S request/limit is 4 cores and deployment is pinned to the node with 4 cores all is exactly what we want. However, if there is no pinning and K8S decides to put this pod on 16 core node, the app in the pod will take them all.

[i-chvets]

Closing.