The nonroot scale-1 variant does not seem to work in Minikube

[polarn]

First, I think it's awesome to stop running as root inside the pods.

But I can't seem to get the nonroot to work in Minikube. Is it a general minikube problem? Zookeeper complains about not being able to create things under the /var/lib/zookeeper mountpoint, and if I go in there (had to edit the statefulset to do a sleep instead of the normal command) the folder has different permissions compared to a working setup in AWS:

nonroot@pzoo-0:/opt/kafka$ ls -ld /var/lib/zookeeper
drwxrwxrwx 4 root root 4096 May 15 09:02 /var/lib/zookeeper
nonroot@pzoo-0:/opt/kafka$ ls -ld /var/lib/zookeeper/data
drwxrwxr-x 2 root root 4096 May 15 09:02 /var/lib/zookeeper/data

And here is how it looks in the working setup in AWS:

nonroot@pzoo-0:/opt/kafka$ ls -ld /var/lib/zookeeper
drwxrwsr-x 5 root nogroup 4096 May 12 11:30 /var/lib/zookeeper
nonroot@pzoo-0:/opt/kafka$ ls -ld /var/lib/zookeeper/data
drwxrwsr-x 3 root nogroup 4096 May 12 11:30 /var/lib/zookeeper/data

As seen, in Minikube the nonroot:nogroup user can't write to the data folder. And I assume this is because the init-container when it created the folder, in AWS it respects the fsGroup parameter for the volume, and in Minikube it ignores it.

By adding chown -R 65532:65534 /var/lib/zookeeper to the zookeeper init file, it works, but since it seems to be working in AWS, I guess this is just affecting Minikube, or just me? 😄

[solsson]

Thanks for the report. We've abandoned minikube in favor of k3s so you might actually be the first to test the nonroot kustomization on minikube. Maybe kubernetes/minikube#1990 confirms your suspicion?

[polarn]

Sorry for not getting back sooner. Yes I think you are correct... I went back to running as root inside minikube instead.

Do you want me to test the nonroot-entrypoints branch?

[solsson]

I found the issue with k3s locally on Ubuntu as well. Yes please, since you're running a dev cluster I'm interested in how nonroot-entrypoints works for you after 83af767.

[polarn]

Seems to be working, I deployed it in my minikube, created a topic and could produce and consume to it. However pzoo-0 is logging this a few times, I assume that is fine?

[SyncThread:0] WARN org.apache.zookeeper.server.ZooKeeperServer - Failed to register with JMX
java.lang.NullPointerException
	at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
	at org.apache.zookeeper.server.ServerCnxnFactory.registerConnection(ServerCnxnFactory.java:183)
	at org.apache.zookeeper.server.ZooKeeperServer.finishSessionInit(ZooKeeperServer.java:726)
	at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:207)
	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:182)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:527)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

It's also seems to be a new docker base image without any linux tools, so I cant check the permissions on the /var/lib/zookeeper but I assume it's all good because it works. :)

[solsson]

Thank you. I documented the JMX issue in solsson/dockerfiles#29 (comment). But I also haven't seen any actual malfunction :)