kube generate: privileged: false is not correct

[dilyanpalauzov]

I run podman in rooltess mode and kubernetes in rootful mode. To start podman I use the instructions at https://mail.aegee.org/cgit/aegee-ldap/tree/readme.md#n13 . Then I call podman generate kube. The output contains:

apiVersion: v1        
kind: Pod            
spec:     
  containers:
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
        - CAP_MKNOD
        - CAP_NET_RAW
        - CAP_AUDIT_WRITE
      privileged: false
      readOnlyRootFilesystem: true 
      runAsGroup: 2000
      runAsUser: 2000
      seLinuxOptions: {}
    volumeMounts:
    - mountPath: /data
      name: home-d-data-host-0
  volumes:
  - hostPath:
      path: /home/d/data
      type: Directory
    name: home-d-data-host-0

When I give /home/, /home/d and /home/d/data mode 777, and insert this into kubernetes, it does not work, since the privileges to access the volume are not sufficient. I have to change privileged: false to privileged: true.

Perhaps podman generate kube shall accept more command line opitons to know how to emit yaml.

[umohnani8]

@dilyanpalauzov what new options are you looking for? The privileged option in the generated kube yaml is set based on what value you pass to the --privileged flag in podman when creating the container. By default the privileged value is set to false, so it looks like the kube yaml was generated correctly.

I tested it out, by setting the --privileged flag to true in podman and my generated kube yaml showed privileged as true:

➜  ~ podman create --pod new:foo --privileged=true alpine ls
fdd52ebb0e2141e9de7f99c1780504ffce5a5e14b7cfef01aaaf857366fc9417
➜  ~ podman ps -a
CONTAINER ID  IMAGE                            COMMAND     CREATED        STATUS      PORTS       NAMES
07ef2c7e4562  k8s.gcr.io/pause:3.5                         3 seconds ago  Created                 530a111026c0-infra
fdd52ebb0e21  docker.io/library/alpine:latest  ls          3 seconds ago  Created                 suspicious_blackwell
➜  ~ podman generate kube foo
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.0.0-dev
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2021-10-11T16:12:33Z"
  labels:
    app: foo
  name: foo
spec:
  containers:
  - command:
    - ls
    image: docker.io/library/alpine:latest
    name: suspiciousblackwell
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
    workingDir: /
  dnsConfig: {}
  restartPolicy: Never
status: {}

[dilyanpalauzov]

In my particular case, I use rootless podman, which bind-mounts a directory (with selinux label?):

podman run --read-only=true --mount type=bind,src=data,dst=/data,relabel=private --read-only-tmpfs=false -d -p=3890:3890 localhost/openldap:2021-09-21

podmaan generate kube emits:

metadata:
  annotations:
    bind-mount-options:/home/d/data: Z
spec:
  …
  volumes:
  - hostPath:
      path: /home/d/data
      type: Directory
    name: home-d-data-host-0

however kubernetes does not handle this annotation in any special way, so it is ignored, when resolving the hostPath. Without the Z label, the hostPath volume/mount does not work. In turn inserting the yaml file in Kubernetes, when kubernetes mounts the volume, the pod sees “permission denied”.

I do not know how to solve this, since I do not understand selinux and volumes very well.

If the mount:Z/relabel=private relabelling are a matter of selinux labels, then insert that selinux (or seccomp) property in spec.container.securityContext.

All in all, the output of podman kube generate migth work very well as input to podman play kube, but it does not work as input to Kubernetes.

To be precise, when trying to inject a yaml file with

  annotations:
    bind-mount-options:/home/d/data: Z

in kubectl-1.22.2 apply -f, the answer is:

The Pod "ldap" is invalid: metadata.annotations: Invalid value: "bind-mount-options:/home/d/data": a qualified name must consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9.]*)?[A-Za-z0-9]') with an optional DNS subdomain prefix and '/' (e.g. 'example.com/MyName')

[rhatdan]

Kubernetes does not do any relabeling with SELinux now. There have been proposals but nothing has been accepted. We could potentially add functionality to CRI-O to make this happen and have it understand the annotation, but it is not likely that this would be accepted upstream. @mrunalp @haircommander @saschagrunert what do you think about adding this support to CRI-O.

[rhatdan]

@dilyanpalauzov to make this work in kubernetes, you could simple
chcon -T container_file_t -R /home/d/data
And then the kubernetes.

[dilyanpalauzov]

I think that the overall direction of Kubernetes is to allow to run kubelet in userspace and the SELinux relabelling of directories/hostPath-volumes is a milestone in the rootless journey.

Another part in writing down how cri-o shall be setup in rootless mode - cri-o/cri-o#5399.

[haircommander]

Kubernetes does not do any relabeling with SELinux now

this is not quite right: many volume plugins do relabel selinux, but hostPath does not. The proposals are in attempt to not relabel in some cases for those that do, as it can take a long time if the volume is sufficiently large.

as far as the annotation, I am open to it. I think we'll need to talk about the structure of it--I don't know if an annotation can contain a :.

chcon -T container_file_t -R /home/d/data

frankly, I see this as the most idiomatic option. SELinux relabeling can take a long time, and it is more efficient to just do it the once and then every pod can have acccess to it, rather than relabel for each pod (which would happen if we added an annotation workaround)

the SELinux relabelling of directories/hostPath-volumes is a milestone in the rootless journey.

I don't know of any effort to relabel hostPath directories, though I may be wrong.

[rhatdan]

The issue I have with the labeling we have now is that it is always shared, which means that all Pods can attack each other. I would like to have CRI-O understand that it could do Private labeling also. I am fine with the labeling for annotations to only use the top level labeling support. IE If directory is currently labeled correctly then do nothing.

[rhatdan]

@dilyanpalauzov SELinux has nothing to do with rootless versus rootfull. It is needed in either situation. I actually am not a big proponant of rootless kubernetes. I am a big fan of running all workloads either as non root or within a separate usernamespace where real root is not mapped.

[haircommander]

The issue I have with the labeling we have now is that it is always shared

I am under the impression CRI-O always label privately (though we'll add additional contexts as they're requested)

[dilyanpalauzov]

My reading is, that in order to use the output of podman generate kube, which contains spec.volumes[].hostPath as input to kubectl apply, the caller must either call chcon -t container_file_t -R <PATH> on the host, before uploading the yaml file, or set priviliged: true. podman does relabelling for the directory, while for kubelet this is slightly out of scope.

I propose:

• include in the output of podman generate kube comments about privileged: true/manual call of chcon -t container_file_t -R <directory>, telling the user (how) to call chcon (in the case where podman is rootless and unprivileged)
• include the details about privileged: false/chcon -t container_file_t -R in the manual page for podman-generate-kube.

The issue I have with the labeling we have now is that it is always shared, which means that all Pods can attack each other.

--mount=type=bind does accept "relabel=private". Likewise -v appepts Z- private unshared, or z - shared. What does the above citation mean?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@umohnani8 PTAL
A better solution would be to run the container with type:spc_t rather then forcing privileged. But I think we now have a label/attribute that will tell podman play kube to do a relabel.

[rhatdan]

@baude committed 1ff6a50 but there is no documentation in the man pages.

[umohnani8]

@rhatdan got it, will update my PR to relabel instead of forcing privileged

[umohnani8]

@rhatdan we can check for rootless and selinux and add the annotation, but when the yaml is used in a k8s cluster, k8s won't recognize it and will still cause the error mentioned in this issue. So might be best to just add a note to let the user know to make the pod privileged or to relabel the volume with chcon. We can generate the annotation so that it is used with play kube, but doesn't look like podman has an issue with the volume here. WDYT?

[rhatdan]

It sounds fine with me. But it could get a little wordy.