1.6.0 broke selinux again

[coeki]

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

What happened?

On the master, where you run kubeadm init, setenforce 0 is still needed as it will be denied to read the certs.

ausearch -m avc

type=AVC msg=audit(1490892442.272:1130): avc:  denied  { open } for  pid=5137 comm="kube-controller" path="/etc/kubernetes/pki/sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c483,c527 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1
----
time->Thu Mar 30 16:47:49 2017
type=AVC msg=audit(1490892469.704:1170): avc:  denied  { read } for  pid=7452 comm="kube-apiserver" name="apiserver.crt" dev="sda1" ino=1573691 scontext=system_u:system_r:container_t:s0:c11,c526 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1
----
time->Thu Mar 30 16:47:49 2017
type=AVC msg=audit(1490892469.704:1171): avc:  denied  { open } for  pid=7452 comm="kube-apiserver" path="/etc/kubernetes/pki/apiserver.crt" dev="sda1" ino=1573691 scontext=system_u:system_r:container_t:s0:c11,c526 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1
----
time->Thu Mar 30 16:47:52 2017
type=AVC msg=audit(1490892472.814:1173): avc:  denied  { open } for  pid=7326 comm="kube-controller" path="/etc/kubernetes/pki/sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c391,c896 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1
----
time->Thu Mar 30 16:47:52 2017
type=AVC msg=audit(1490892472.814:1172): avc:  denied  { read } for  pid=7326 comm="kube-controller" name="sa.key" dev="sda1" ino=1573694 scontext=system_u:system_r:container_t:s0:c391,c896 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=1

What you expected to happen?

Seems similar to #107, and expected to run with setenforce 1

How to reproduce it (as minimally and precisely as possible)?

Every time

Anything else we need to know?

[timothysc]

/cc @dgoodwin @pmorie

[dgoodwin]

Not able to reproduce immediately with a fully up to date CentOS 7 vm, how does this compare to your environment @coeki?

(root@centos1 ~) $ rpm -qa | grep selinux
libselinux-python-2.5-6.el7.x86_64
selinux-policy-3.13.1-102.el7_3.16.noarch
libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-targeted-3.13.1-102.el7_3.16.noarch
container-selinux-2.9-4.el7.noarch

(root@centos1 ~) $ rpm -qa | grep kube
kubernetes-cni-0.5.1-0.x86_64
kubelet-1.6.1-0.x86_64
kubectl-1.6.1-0.x86_64
kubeadm-1.6.1-0.x86_64

(root@centos1 ~) $ kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

(root@centos1 ~) $ ls -lZ /etc/kubernetes/pki
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  apiserver.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  apiserver.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  apiserver-kubelet-client.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  apiserver-kubelet-client.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  ca.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  ca.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  front-proxy-ca.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  front-proxy-ca.key
-rw-r--r--. root root unconfined_u:object_r:cert_t:s0  front-proxy-client.crt
-rw-------. root root unconfined_u:object_r:cert_t:s0  front-proxy-client.key
-rw-------. root root unconfined_u:object_r:cert_t:s0  sa.key
-rw-------. root root unconfined_u:object_r:cert_t:s0  sa.pub

kubeadm init completed successfully for me and no denials in audit.log.

[coeki]

@dgoodwin Sorry, since I use vagrant, and some ansible, that particular environment is long gone, which was based on the just released kubeadm/kubernetes 1.6.0 (not alpha).
But I set my ansible script to just setenforce 0, since I thought it was broken, haven't tested it proper again.

I have tested it just now with a fresh environment, an updated centos7, and kubeadm/kubernetes 1.6.1, and can confirm it works with setenforce 1.

So I'll close this one out, it must have been a fluke.

Thanks for having a look.

[coeki]

Hi @dgoodwin So I refactored my vagrant dev pipeline, and it broke on fedora 25 (updated specs below), which is weird since the rpm's are build on fedora. Tested with weave plugin.

centos/7 (updated):

libselinux-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
container-selinux-2.10-2.el7.noarch
selinux-policy-3.13.1-102.el7_3.16.noarch
selinux-policy-targeted-3.13.1-102.el7_3.16.noarch
libselinux-python-2.5-6.el7.x86_64

fedora 25 (updated):

libselinux-python-2.5-13.fc25.x86_64
selinux-policy-targeted-3.13.1-225.13.fc25.noarch
libselinux-utils-2.5-13.fc25.x86_64
libselinux-python3-2.5-13.fc25.x86_64
rpm-plugin-selinux-4.13.0.1-1.fc25.x86_64
container-selinux-2.10-1.fc25.noarch
selinux-policy-3.13.1-225.13.fc25.noarch
libselinux-2.5-13.fc25.x86_64

docker inspect on Centos/7:

[root@Centos1 vagrant]# docker inspect 9f79beb28ce4 |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt'
        "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c27,c200",
        "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c27,c200",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/kubernetes:/etc/kubernetes/:ro",
                "/etc/ssl/certs:/etc/ssl/certs",
--
            "SecurityOpt": [
                "seccomp=unconfined",
                "label=user:system_u",
                "label=role:system_r",
                "label=type:svirt_lxc_net_t",
                "label=level:s0:c27,c200"

            ],

on fedora 25:

[root@master-01 vagrant]# docker inspect 362a29628561  |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt'
        "MountLabel": "system_u:object_r:container_file_t:s0:c815,c881",
        "ProcessLabel": "system_u:system_r:container_t:s0:c815,c881",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/kubernetes:/etc/kubernetes/:ro",
                "/etc/ssl/certs:/etc/ssl/certs",

        "SecurityOpt": [
            "seccomp=unconfined",
            "label=user:system_u",
            "label=role:system_r",
            "label=type:container_t",
            "label=level:s0:c815,c881"
        ],

after adjusting, so everything with volume mount spc_t:

[root@master-01 vagrant]# docker inspect 8cc809a58e0c |grep -i -A 6 'mountlabel\|processlabel\|SecurityOpt'                     "MountLabel": "system_u:object_r:container_file_t:s0:c81,c416",
        "ProcessLabel": "system_u:system_r:spc_t:s0:c81,c416",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/kubernetes:/etc/kubernetes/:ro",
                "/etc/ssl/certs:/etc/ssl/certs",
--
            "SecurityOpt": [
                "label=type:spc_t",
                "seccomp=unconfined",
                "label=user:system_u",
                "label=role:system_r",
                "label=type:spc_t",
                "label=level:s0:c81,c416"

So why does it work on Centos/7 and not on fedora 25 out of the box? It's ridiculous, the rpm's are build on fedora, centos and fedora should have similar selinux policies, and I think in general hostPath and selinux clash whatever. Sorry to rant but his is just not ok.

cc @rhatdan @dgoodwin @jbeda @luxas 
 

[dgoodwin]

I can understand the frustration as this has seemingly broke out from underneath us several times now, but first lets clarify exactly what broke? Are there denials in /var/log/audit/audit.log? Are there any other errors in docker's journald? What containers were running at the time? (docker ps) Were any containers failing, (docker ps -a) and if so, what was in their docker logs?

I tried to reproduce:

• vagrant init fedora/25-cloud-base; vagrant up --provider libvirt
• full dnf update and reboot
• setenforce 1
• follow kubeadm instructions: https://kubernetes.io/docs/getting-started-guides/kubeadm/ (minus the setenforce 0)

It may be worth noting Fedora is not listed on this getting started guide, I know when we created the rpms we discussed that as far as we knew, they should work on Fedora, but CentOS was definitely considered the more useful target. You will probably have a smoother experience there, but we can still try to keep Fedora operational.

For my test I did find kube-apiserver pod crashing, and the following in audit.log:

type=AVC msg=audit(1493643899.467:671): avc:  denied  { read } for  pid=7848 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0
type=AVC msg=audit(1493643911.826:674): avc:  denied  { read } for  pid=8149 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0
type=AVC msg=audit(1493643940.435:678): avc:  denied  { read } for  pid=8418 comm="kube-apiserver" name="apiserver.crt" dev="vda1" ino=1573656 scontext=system_u:system_r:container_t:s0:c67,c939 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

All other pods seem to come up so this would appear to be a relatively isolated issue with the new certs.

[coeki]

@dgoodwin, I'm sorry for my rant, I admit I was frustrated, cause last time I tested it on fedora/25-cloud-base, it seemed to work. But this was in one of the 1.6.alpha releases, where I build the binaries myself.

I know Fedora is officially not supported, but well centos and fedora having worked smiliar in the past, I still run it.

Probably something changed policy wise, at least the selinux labels differ between centos/7 and fedora/25, which might be the issue.

So I did some more investigations and trying to piece it together.

Fedora/25 uses the newer scheme of container_t for the process label and container_file_t for the mount label. Centos/7 uses svirt_lxc_net_t for process and svirt_sandbox_file_t for the mount label. Not sure if this matters though, they should be the same basically.

BTW I'm seeing the same avc's for kube-apiserver on fedora/25:

type=AVC msg=audit(1493669542.347:1525): avc: denied { read } for pid=7005 comm="kube-apiserver" name="apiserver.crt" dev="sda1" ino=1573608 scontext=system_u:system_r:container_t:s0:c400,c673 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

After some tries the kube-apiserver dies, not even running in docker. Now if I manually adjust the manifest for kube-apiserver and add:

  securityContext:
    seLinuxOptions:
      type: spc_t

It runs the kube-apiserver, but if the edit was not quick enough, you might have to restart kubelet. Now it works with setenforce 1. This also affects kube-controller-manager.

type=AVC msg=audit(1493673979.866:3337): avc: denied { read } for pid=23013 comm="kube-controller" name="sa.key" dev="sda1" ino=1573672 scontext=system_u:system_r:container_t:s0:c535,c704 tcontext=unconfined_u:object_r:cert_t:s0 tclass=file permissive=0

Fixed that manifest like kube-apiserver and then the cluster runs again.

So I can't put my finger on it.
Conceptually conflicting selinux labels on hostPath should raise an AVC, that's why we introduced the spc_t container type for etcd, especially for write. So all containers/pods using a hostPath mount for writing, should be run as spc_t (weave-1.6 plugin also does that now).
In this case however reading the hostPath, /etc/kubernetes/, is prohibited on fedora/25, but not Centos/7.

So what is broken? That remains a good point.

Thanks for your patience ;)

[dgoodwin]

Not a problem, it is frustrating keeping this functional. I'm not great with selinux but this is good info above, I will try to find some time to debug soon. We could perhaps just go spc_t for the apiserver it's just unfortunate to have to open that one up as well. (we do this for etcd so it can write to host disk, but thus far that was it)

[coeki]

@dgoodwin I found the root cause in Fedora 25, it's the fact that process type container_t, can't read a file that is not labeled type container_file_t or container_ro_file_t, which clashes, as the certs created on the host are labeled cert_t.

This post from @rhatdan explains a tightening of the container_selinux policy:
http://www.projectatomic.io/blog/2017/02/selinux-policy-containers/

Some diff between fedora and centos:

Fedora:

sesearch -ACR -s container_t -t "cert*" -c file -p read
Found 4 semantic av rules:
   allow svirt_sandbox_domain exec_type : file { ioctl read getattr lock execute execute_no_trans entrypoint open } ;
   allow svirt_sandbox_domain virsh_t : fifo_file { ioctl read write getattr lock append open } ;
   allow svirt_sandbox_domain exec_type : lnk_file { read getattr } ;
   allow domain tmpfile : file { ioctl read getattr lock append } ;

Centos:

[root@Centos1 vagrant]# sesearch -ACR -s svirt_lxc_net_t -t "cert*" -c file -p read
Found 12 semantic av rules:
   allow svirt_sandbox_domain exec_type : file { ioctl read getattr lock execute execute_no_trans open } ;
   allow nsswitch_domain cert_t : lnk_file { read getattr } ;
   allow svirt_sandbox_domain exec_type : lnk_file { read getattr } ;
   allow svirt_sandbox_domain systemd_unit_file_type : file { ioctl read getattr lock open } ;
   allow svirt_sandbox_domain virsh_t : fifo_file { ioctl read write getattr lock append open } ;
   allow svirt_sandbox_domain systemd_unit_file_type : lnk_file { read getattr } ;
   allow domain tmpfile : file { ioctl read getattr lock append } ;
   allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ;
DT allow nsswitch_domain slapd_cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain slapd_cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]

If the fedora container_selinux policy would be mainstream for centos/rhel as well, docker just mounting a directory on the host as a volume with read-only permissions, would still be blocked, if it's not (re)labeled container_ro_file_t or container_file_t

And then there is something I found, but did not test:

[root@master-01 vagrant]# sesearch -ACR -s  container_auth_t  -t "cert*" -c file -p read
Found 10 semantic av rules:
   allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ;
   allow daemon initrc_transition_domain : fifo_file { ioctl read write getattr lock append } ;
   allow container_auth_t cert_t : lnk_file { read getattr } ;
   allow container_auth_t cert_t : file { ioctl read getattr lock open } ;
   allow nsswitch_domain cert_t : lnk_file { read getattr } ;
   allow domain tmpfile : file { ioctl read getattr lock append } ;
DT allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain slapd_cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain slapd_cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]

[root@Centos1 vagrant]# sesearch -ACR -s  container_auth_t  -t "cert*" -c file -p read
Found 10 semantic av rules:
   allow container_auth_t cert_t : file { ioctl read getattr lock open } ;
   allow nsswitch_domain cert_t : lnk_file { read getattr } ;
   allow daemon initrc_transition_domain : fifo_file { ioctl read write getattr lock append } ;
   allow container_auth_t cert_t : lnk_file { read getattr } ;
   allow domain tmpfile : file { ioctl read getattr lock append } ;
   allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ;
DT allow nsswitch_domain slapd_cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain slapd_cert_t : lnk_file { read getattr } ; [ authlogin_nsswitch_use_ldap ]
DT allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]

So, I'm kinda stuck, couple of solutions as far as I can see are:

• • Run everything that needs access to any directory mounted as a docker/kubernetes volume on the host with label spc_t or label (maybe) container_auth_t
• • Manually change stuff on Fedora:
  • • change label
  • • change policy, so container_t can read cert_t
• • Or inject the certs with configMaps

Any thoughts?

[dgoodwin]

Really good info, thanks!

spc_t would be the quick and easy path forward, quite an easy change and precedent already there for etcd pod.

Certs injected as secrets or configmaps would likely be the ideal solution.

[rhatdan]

container_auth_t is a context assigned to docker authorization plugins, not to container processes.

[rhatdan]

Simplest case is to run as spc_t, but we need to make it easier to extend the SELinux policy for container processes to add these types of rules. I will take a quick look at doing this.

[luxas]

@rhatdan @coeki @dgoodwin I'd love to proceed on this (see #279)

Can I assign this to any of you?

[coeki]

As it stands this only affects Fedora25, but as we seen before, it could trickle down to Centos7.
@rhatdan might know better if this will be so?

The fix should be trivial, as we just need to add spc_t to the static pod specs of the kube-apiserver and I think the kube-controller, so any pod that even wants to read /etc/kubernetes/pki.

I would like to try and see if I can make a PR, but since I have never done it before, I'll have to setup some stuff.

[luxas]

@coeki A PR from you fixing that would be awesome! Contact me privately on Slack in case you run into errors when creating your dev env.
Might be of interest: https://github.com/kubernetes/community/tree/master/contributors/devel

[jamiehannaford]

@coeki Are you working on this? Would be awesome to get this fixed. We now store PKI assets as secrets for self-hosted clusters (see kubernetes/kubernetes#50766), so hopefully that should also fix some of this.

[0xmichalis]

@luxas still an issue?

[jamiehannaford]

I think we can close due to inactivity.