Cannot Specify Kubelet Policies in Config File

[DanielLee343]

What happened:
I want to specify policies for kubelet for example CPUManagerPolicy, and TopologyManagerPolicy. I've tried two ways to specify in the config file but neither works.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "TopologyManager": true
  "CPUManager": true
  "MemoryManager": true
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        topology-manager-policy: single-numa-node
        cpu-manager-policy: static

This one times out in ✗ Starting control-plane and journalctl -xeu kubelet shows:

...
-- 
-- A start job for unit kubelet.service has finished successfully.
-- 
-- The job identifier is 54164.
May 17 23:04:39 nu-ins-1 systemd[3052607]: kubelet.service: Failed to execute command: No such file or directory
May 17 23:04:39 nu-ins-1 systemd[3052607]: kubelet.service: Failed at step EXEC spawning /var/lib/minikube/binaries/v1.26.3/kubelet: No such file or directory
-- Subject: Process /var/lib/minikube/binaries/v1.26.3/kubelet could not be executed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The process /var/lib/minikube/binaries/v1.26.3/kubelet could not be executed and failed.
-- 
-- The error number returned by this process is ERRNO.
May 17 23:04:39 nu-ins-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit kubelet.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 203.
May 17 23:04:39 nu-ins-1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit kubelet.service has entered the 'failed' state with result 'exit-code'.

Then I tried replacing kind: InitConfiguration to kind: KubeletConfiguration and turns out creating cluster fine. However, when I look to see if the policy is configured correctly by kubectl get --raw "/api/v1/nodes/kind-control-plane/proxy/configz" | jq, it shows

...
"cpuManagerPolicy": "none",
...
"topologyManagerPolicy": "none",
...
 "featureGates": {
      "CPUManager": true,
      "MemoryManager": true,
      "TopologyManager": true
    },

As it shows the feature gates are all enabled, but the policies of both cpuManagerPolicy and topologyManagerPolicy are not set up properly.

To testify, I deploy a simple stress test like this:

spec:
      containers:
      - name: stress-test
        image: progrium/stress:latest
        args: ["--vm", "2", "--vm-bytes", "5G", "--timeout", "30s"]
        resources:
          limits:
            memory: "12Gi"
            cpu: "2"
          requests:
            memory: "12Gi"
            cpu: "2" 

Like what I specified for topology-manager-policy, I expect the pod to consume on a single NUMA node memory (otherwise would never be admitted), however, pcm-memory shows:

that the pod spins on cross-NUMA nodes. I guess I didn't specify the config file correct.

What you expected to happen:
kubelet configuration will show the policies are recognized and pod should behave as it the policy indicates.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version):
  kind v0.18.0 go1.20.2 linux/amd64
• Runtime info: (use docker info or podman info):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-107-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.7GiB
 Name: nu-ins-1
 ID: ab710971-b6a8-41c3-9c27-580cae8cb3e8
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: lyuze
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Kubernetes version: (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-30T06:34:50Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

[BenTheElder]

-- Subject: Process /var/lib/minikube/binaries/v1.26.3/kubelet could not be executed

Ummm, minikube? KIND does not use this path. What is this log?

[BenTheElder]

To specify in a config file use like:
#2999 (comment)
#2999 (comment)

(note the issue linked is about a bug that is fixed, but in the issue various configurations enabling static CPUmanager are used).

Also note that featuregates are kubernetes version dependent, and eventually become default and go away as the feature GAs.

[BenTheElder]

For memory limits, see also #3169 #1963

[DanielLee343]

Thanks @BenTheElder ! That fixed the issue.