containerd 1.4.4 errors out with "Selinux is not supported" due to missing build tags

[anusha-ragunathan]

What happened:
Created a cluster with containerd as CRI.
Enabled selinux with enable_selinux = true (in /etc/containerd/config.toml)
SELinux does not work.

containerd version baked in the AMI is 1.4.4

# cat /etc/containerd/config.toml 
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    enable_selinux = true

However, containerd spits out a warning in /var/log/messages like so:

Aug 27 18:38:51 ip-10-225-48-48 containerd: time="2021-08-27T18:38:51.407197323Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false BaseRuntimeSpec:} UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false BaseRuntimeSpec:} Runtimes:map[runc:{Type:io.containerd.runc.v2 Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:0xc0003dee10 PrivilegedWithoutHostDevices:false BaseRuntimeSpec:}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.intuit.com:{Endpoints:[https://docker.intuit.com https://docker-readonly.intuit.com]} docker.io:{Endpoints:[https://registry-1.docker.io]}] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:true SelinuxCategoryRange:1024 SandboxImage:k8s.gcr.io/pause:3.2 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true IgnoreImageDefinedVolumes:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Aug 27 18:38:51 ip-10-225-48-48 containerd: time="2021-08-27T18:38:51.407381546Z" level=warning msg="Selinux is not supported" <====

Further more, verified that the process and filesystem labels expected on Pods (an example nginx deployment) is not set:

# listing from within the nginx container
# ls -Z /usr/sbin/nginx
system_u:object_r:container_var_lib_t:s0 /usr/sbin/nginx

# listing from the node where SELinux is enabled on containerd
# ps -eZ
LABEL                              PID TTY          TIME CMD
system_u:system_r:spc_t:s0           1 ?        00:00:00 nginx
system_u:system_r:spc_t:s0          31 ?        00:00:00 nginx
system_u:system_r:spc_t:s0          32 pts/0    00:00:00 sh
system_u:system_r:spc_t:s0         422 pts/0    00:00:00 ps

What you expected to happen:
containerd supports Selinux without warnings/errors

Anything else we need to know?:
It's possible that the Amazon Linux version of containerd does not build with Golang BUILD_TAGS set to selinux. This is essential to get SELinux support. Else, only stubbed out Go methods are built as part of containerd, which results in the "Selinux is not supported" error.

I built containerd 1.4.4 from source and this version works fine. Simply issuing make on https://github.com/containerd/containerd/tree/v1.4.4 should be good.

Deploying an nginx Daemonset on the node with the containerd fix shows the tests work as expected.

The process labels show up as expected. This is the process listing on the node.

# ps -eZ | grep nginx
system_u:system_r:container_t:s0:c511,c746 286060 ? 00:00:00 nginx
system_u:system_r:container_t:s0:c511,c746 286118 ? 00:00:00 nginx

The file labels show up as expected. This is from inside the container.

$ k exec -it  nginx-daemon-set-pzhlj -- bash
root@nginx-daemon-set-pzhlj:/# which nginx
/usr/sbin/nginx
root@nginx-daemon-set-pzhlj:/# ls -Z /usr/sbin/nginx
system_u:object_r:container_file_t:s0:c511,c746 /usr/sbin/nginx

Environment:

• AWS Region: us-west-2
• Instance Type(s): Issue seen across several instances such as t2.large, m5.xlarge and m5.xlarge
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): "eks.6"
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): "1.19"
• AMI Version: custom built
• Kernel (e.g. uname -a): 5.4.141-67.229.amzn2.x86_64 SMP Mon Aug 16 12:51:43 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /etc/eks/release on a node):

# cat /etc/eks/release 
BASE_AMI_ID="ami-015412692e01778b4"
BUILD_TIME="Fri Sep 10 01:53:55 UTC 2021"
BUILD_KERNEL="5.4.141-67.229.amzn2.x86_64"
ARCH="x86_64"

[anusha-ragunathan]

Turns out that if I build containerd 1.4.4 tag from https://github.com/containerd/containerd/tree/v1.4.4, the selinux tags are automatically included and it works fine. Maybe Amazon Linux is stripping out the build tags when building containerd ?

I couldn't find the amazon source pkg/rpm for containerd-1.4.4, so I'm not sure how EKS builds containerd from source.

[anusha-ragunathan]

Evidence that the containerd binary baked with the AMI does not include the expected getEnabled selinux method, which is critical to enable SELinux support

1. strings on AWS containerd 1.4.4

# strings /bin/containerd > containerd.log

# grep -i "initPlatform" containerd.log | more
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).initPlatform
github.com/containerd/containerd/runtime/v1/shim.(*Service).initPlatform
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).initPlatform.stkobj
initPlatform
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).initPlatform
github.com/containerd/containerd/runtime/v1/shim.(*Service).initPlatform

# grep -i getEnabled containerd.log
<empty>

2. Build OSS containerd from v1.4.4 tag

# GO111MODULE=off BUILDTAGS="no_btrfs selinux" make

# strings ./bin/containerd > containerd.log

# grep -i getEnabled containerd.log 
github.com/containerd/containerd/vendor/github.com/opencontainers/selinux/go-selinux.(*selinuxState).getEnabled
github.com/containerd/containerd/vendor/github.com/opencontainers/selinux/go-selinux.(*selinuxState).getEnabled
github.com/containerd/containerd/vendor/github.com/opencontainers/selinux/go-selinux.getEnabled
github.com/containerd/containerd/vendor/github.com/opencontainers/selinux/go-selinux.GetEnabled

[anusha-ragunathan]

I looked more into why selinux is not added as a BUILDTAG, even when it's set in the Makefile
https://github.com/containerd/containerd/blob/v1.4.4/Makefile#L78-L79

It's because of how BUILDTAGS is being processed in the Makefile.

# Build tags apparmor and selinux are needed by CRI plugin.
GO_BUILDTAGS ?= apparmor selinux

Now what does "?=" in a Makefile mean? According to the make man page, https://www.gnu.org/software/make/manual/make.html#index-_003f_003d

There is another assignment operator for variables, ‘?=’. This is called a conditional variable assignment operator, because it only has an effect if the variable is not yet defined. This statement:

FOO ?= bar
is exactly equivalent to this (see The origin Function):

ifeq ($(origin FOO), undefined)
  FOO = bar
endif
Note that a variable set to an empty value is still defined, so ‘?=’ will not set that variable.

This means, when containerd is built with BUILDTAGS, the GO_BUILDTAGS for "apparmor" and "selinux" are not added.

[anusha-ragunathan]

Dumping out tags in the Makefile, shows that the "selinux" tag is INDEED not added.

# git diff
diff --git a/Makefile b/Makefile
index c0fecb9a6..329eea561 100644
--- a/Makefile
+++ b/Makefile
@@ -185,6 +185,9 @@ FORCE:
 
 define BUILD_BINARY =
 @echo "$(WHALE) $@"
+@echo "***BUILDTAGS: $(BUILDTAGS)***"
+@echo "***GO_BUILDTAGS: $(GO_BUILDTAGS)***"
+@echo "***GO_TAGS: $(GO_TAGS)***"
 @go build ${DEBUG_GO_GCFLAGS} ${GO_GCFLAGS} ${GO_BUILD_FLAGS} -o $@ ${GO_LDFLAGS} ${GO_TAGS}  ./$<
 endef

# GO111MODULE=off make BUILDTAGS=no_btrfs
+ bin/ctr
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs ***
***GO_TAGS: -tags no_btrfs ***
+ bin/containerd
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs ***
***GO_TAGS: -tags no_btrfs ***
+ bin/containerd-stress
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs ***
***GO_TAGS: -tags no_btrfs ***
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries

Now, I made a change to remove the conditional assignment. The selinux and apparmor tags are added!

# git diff
diff --git a/Makefile b/Makefile
index c0fecb9a6..2b3bf52ba 100644
--- a/Makefile
+++ b/Makefile
@@ -76,7 +76,7 @@ ifdef BUILDTAGS
     GO_BUILDTAGS = ${BUILDTAGS}
 endif
 # Build tags apparmor and selinux are needed by CRI plugin.
-GO_BUILDTAGS ?= apparmor selinux
+GO_BUILDTAGS += apparmor selinux
 GO_BUILDTAGS += ${DEBUG_TAGS}
 GO_TAGS=$(if $(GO_BUILDTAGS),-tags "$(GO_BUILDTAGS)",)
 GO_LDFLAGS=-ldflags '-X $(PKG)/version.Version=$(VERSION) -X $(PKG)/version.Revision=$(REVISION) -X $(PKG)/version.Package=$(PACKAGE) $(EXTRA_LDFLAGS)'
@@ -185,6 +185,9 @@ FORCE:
 
 define BUILD_BINARY =
 @echo "$(WHALE) $@"
+@echo "***BUILDTAGS: $(BUILDTAGS)***"
+@echo "***GO_BUILDTAGS: $(GO_BUILDTAGS)***"
+@echo "***GO_TAGS: $(GO_TAGS)***"
 @go build ${DEBUG_GO_GCFLAGS} ${GO_GCFLAGS} ${GO_BUILD_FLAGS} -o $@ ${GO_LDFLAGS} ${GO_TAGS}  ./$<
 endef

# GO111MODULE=off make BUILDTAGS=no_btrfs
+ bin/ctr
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs apparmor selinux ***
***GO_TAGS: -tags no_btrfs apparmor selinux ***
+ bin/containerd
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs apparmor selinux ***
***GO_TAGS: -tags no_btrfs apparmor selinux ***
+ bin/containerd-stress
***BUILDTAGS: no_btrfs***
***GO_BUILDTAGS: no_btrfs apparmor selinux ***
***GO_TAGS: -tags no_btrfs apparmor selinux ***
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries

[anusha-ragunathan]

The issue reveals only when you build by explicitly adding BUILDTAGS in your make environment. In this case, the Amazon Linux 2 host that's used for compiling does not have the btrfs module loaded and I had to build with BUILDTAGS=no_btrfs.

Maybe that's how the containerd 1.4.4 package is also built ?

[anusha-ragunathan]

Verified that a yum install of containerd version 1.4.6-5 available via amzn2-extras.repo has fixed this issue and provides SELinux support.

[jj-fontana]

Hi,
facing the same issue here but the only version I can find on amzn2 repo is the 1.4.6-3.amzn2.
I tested the centos package containerd.io.x86_64-1.4.12-3.1.el8 and it seems to work fine but I rather like use a amzn package here..