Odd error with crun update setting memsw #326
Comments
|
I think the error could be caused by trying to write to the root cgroup. I am going to take a look, are you able to easily reproduce it? |
giuseppe
added a commit
to giuseppe/crun
that referenced
this issue
Apr 14, 2020
if the write to the memory limit fails with EINVAL, try to reverse the order the two files are written. Closes: containers#326 Signed-off-by: Giuseppe Scrivano <[email protected]>
|
@giuseppe I'm working on adding crun support to Kubespray project. After doing a deployment on Ubuntu 20.04 LTS, I'm getting a similar error This is the information about the worker node: $ crun --version
crun version 0.15
commit: 56ca95e61639510c7dbd39ff512f80f626404969
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
$ uname -a
Linux aio 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ kubelet --version
Kubernetes v1.19.2And this is the pod definition: apiVersion: v1
kind: Pod
metadata:
name: crun-pod
spec:
runtimeClassName: crun
containers:
- name: test
image: busybox
command: ["sleep"]
args: ["infity"]So I'm getting this error: BTW, I tried also with podman in the node to verify crun. I suspect that I missing a value in crio configuration file $ grep -v ^\# /etc/crio/crio.conf | grep .
[crio]
log_dir = "/var/log/crio/pods"
version_file = "/var/run/crio/version"
version_file_persist = "/var/lib/crio/version"
[crio.api]
listen = "/var/run/crio/crio.sock"
stream_address = "127.0.0.1"
stream_port = "10010"
stream_enable_tls = false
stream_tls_cert = ""
stream_tls_key = ""
stream_tls_ca = ""
grpc_max_send_msg_size = 16777216
grpc_max_recv_msg_size = 16777216
[crio.runtime]
default_runtime = "runc"
no_pivot = false
decryption_keys_path = "/etc/crio/keys/"
conmon = "/usr/bin/conmon"
conmon_cgroup = "system.slice"
conmon_env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
]
default_env = [
]
selinux = false
seccomp_profile = ""
cgroup_manager = "systemd"
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"NET_RAW",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"SYS_CHROOT",
"KILL",
]
default_sysctls = [
]
additional_devices = [
]
hooks_dir = [
"/usr/share/containers/oci/hooks.d",
]
default_mounts = [
]
pids_limit = 1024
log_size_max = -1
log_to_journald = false
container_exits_dir = "/var/run/crio/exits"
container_attach_socket_dir = "/var/run/crio"
bind_mount_prefix = ""
read_only = false
log_level = "info"
log_filter = ""
uid_mappings = ""
gid_mappings = ""
ctr_stop_timeout = 30
manage_ns_lifecycle = false
namespaces_dir = "/var/run"
pinns_path = ""
[crio.runtime.runtimes.runc]
runtime_path = "/usr/sbin/runc"
runtime_type = "oci"
runtime_root = "/run/runc"
[crio.runtime.runtimes.crun]
runtime_path = "/usr/bin/crun"
runtime_type = "oci"
runtime_root = "/run/crun"
[crio.image]
default_transport = "docker://"
global_auth_file = ""
pause_image = "k8s.gcr.io/pause:3.3"
pause_image_auth_file = ""
pause_command = "/pause"
signature_policy = ""
image_volumes = "mkdir"
registries = [
]
[crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dirs = [
"/opt/cni/bin",
"/usr/libexec/cni",
]
[crio.metrics]
enable_metrics = false
metrics_port = 9090I'll appreciate any pointer and help on this. |
|
thanks for the report. I wonder if it is related to missing the swap memory limit in Ubuntu. I've tried to reproduce with Podman on Ubuntu 20.04 but I've not managed. Were you able to reproduce it only through Podman? |
Maybe I can deploy this in another distro, what Kubespray's OS supported do you suggest? |
|
Fedora 32 would be ideal for me. I've never used Kubespray before, if you have a configuration I can more use locally, I'll be happy to try reproducing the issue here. |
|
Ok, I'll try to deploy it with Fedora32 but I've to admit that its cri-o support in Kubespray has not been well tested |
|
I had some issues using cgroups v1, I've submitted a PR to fix that. Once those changes are applied it's possible to deploy crun in Fedora +31 with Kubespray: $ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aio Ready master 40m v1.18.9 10.10.16.4 <none> Fedora 32 (Thirty Two) 5.8.13-200.fc32.x86_64 cri-o://1.18.3
$ ps -ef | grep crun
root 22494 1 0 01:23 ? 00:00:00 /usr/libexec/crio/conmon -s -c f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef -n k8s_POD_crun-pod_default_b3f6b503-8116-467f-bde8-53a1c26c6d8d_0 -u f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef -r /usr/bin/crun -b /var/run/containers/storage/overlay-containers/f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef/userdata --persist-dir /var/lib/containers/storage/overlay-containers/f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef/userdata -p /var/run/containers/storage/overlay-containers/f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef/userdata/pidfile -P /var/run/containers/storage/overlay-containers/f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef/userdata/conmon-pidfile -l /var/log/pods/default_crun-pod_b3f6b503-8116-467f-bde8-53a1c26c6d8d/f75f523ce80b3ba33706235b708eced8a3bdb6822e09fc96df410c78d1de97ef.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level info --runtime-arg --root=/run/crun
root 22578 1 0 01:23 ? 00:00:00 /usr/libexec/crio/conmon -s -c cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270 -n k8s_test_crun-pod_default_b3f6b503-8116-467f-bde8-53a1c26c6d8d_0 -u cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270 -r /usr/bin/crun -b /var/run/containers/storage/overlay-containers/cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270/userdata --persist-dir /var/lib/containers/storage/overlay-containers/cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270/userdata -p /var/run/containers/storage/overlay-containers/cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270/userdata/pidfile -P /var/run/containers/storage/overlay-containers/cbfe13ae988928634818765e63f2627ac32b16f3fee4b0c9cd7da844b1aea270/userdata/conmon-pidfile -l /var/log/pods/default_crun-pod_b3f6b503-8116-467f-bde8-53a1c26c6d8d/test/0.log --exit-dir /var/run/crio/exits --socket-dir-path /var/run/crio --log-level info --runtime-arg --root=/run/crun |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am seeing
on:
crun 0.13 and master
cgroup v1
CRI-O CI on cri-o/cri-o#3564 (not locally)
Through applying the patch in #325, I have determined it is the
opencall in src/libcrun/utils.c:195 sending the EINVALIt seems to happen when we set a memsw on the initial container creation,:
and then call a crun update where we specify Memory.Limit and Memory.Swap:
The text was updated successfully, but these errors were encountered: