Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

microk8s + kerberos integration issues #705

Closed
fabioscaccabarozzi opened this issue Dec 12, 2023 · 14 comments · Fixed by #709, #711 or #714
Closed

microk8s + kerberos integration issues #705

fabioscaccabarozzi opened this issue Dec 12, 2023 · 14 comments · Fixed by #709, #711 or #714

Comments

@fabioscaccabarozzi
Copy link

What happened: when trying to setup PV+PVC with Kerberos under microk8s, nodes cannot mount the share, and pods stay in "ContainerCreating" state, until they fail and are restarted.

What you expected to happen: share is mounted fine, pods can correctly start

How to reproduce it:

Steps to reproduce
Assuming we're on a host where Kerberos is configured, and cifs-utils and keyutils are installed.
Install microk8s via snap: snap install microk8s --classic
Install CSI driver with values supplied below.
Use the driver-parameters docs for the kerberos part (kinit, secret, ...)
Create the resources below. PV and PVC will show "Bound".
Attempt to create a deployment that uses the PVC (see example below).
Whichever node attempts to run the pod will fail with the following error:

microk8s.daemon-kubelite[1422]: E1212 11:06:34.115973    1422 csi_attacher.go:364] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = Internal desc = Error writing kerberos cache: rpc error: code = Internal desc = Directory for kerberos caches must exist, it will not be created: /var/lib/kubelet/kerberos/: stat /var/lib/kubelet/kerberos/: no such file or directory

Investigation & Explanation
In microk8s, by default /var/lib/kubelet is a symlink pointing to /var/snap/microk8s/common/var/lib/kubelet.
Kerberos was configured according to the driver-parameter docs linked above, so we have:

[libdefaults]
        debug = true
        default_realm = <realm.com>
        default_ccache_name = FILE:/var/lib/kubelet/kerberos/krb5cc_%{uid}
        kdc_timesync = 1
        ccache_type = 4
        forwardable = true
        proxiable = true

[realms]
...
[domain_realm]
...

The first problem is that if we supply linux.kubelet at chart installation, that folder will be mounted by the chart in the csi-smb-node pod verbatim via hostPath (host: /path/A -> pod: /path/A).

The second problem is we can't seem to find a way to alter the Kerberos cache path. This means that if we move the kubelet path via linux.kubelet, the /var/lib/kubelet path won't exist in the csi-smb-node pods, making it impossible to create and save kerberos tickets from the pod to the host system, which ultimately is the one performing the mounts via cifs-utils.

Further, currently in the Kerberos cache folder absolute symlinks are used from krb5cc_UID files to the actual token files (e.g: /var/lib/kubelet/kerberos/krb5cc_1000 -> /var/lib/kubelet/kerberos/aW1nLXNoYXJlLWRldm1xcw==), so the source path from the host must be exactly the same also in the pod, otherwise the absolute symlinks won't work for cifs-utils.

As it stands now, csi-smb-node expects the the kerberos cache to be at /var/lib/kubelet/kerberos and nowhere else, making it impossible the use of Kerberos with a different linux.kubelet.


Fixing attempt
I have tried to modify the helm chart template for csi-smb-node to mount the Kerberos cache folder as an additional volume, or the same volume with subPath, but it doesn't work -> touch /var/lib/..../kerberos/test would yield no such file or directory, I guess because I cannot mount the same path twice for the same pod. Anyway that would clash with the hard-coded path in the file referenced above.


Proposed solution
The better solution would be to make the Kerberos cache path configurable as the linux.kubelet path is, and ideally discourage the use of /var/lib/kubelet/kerberos, as in microk8s that path is a symlink to the snap folder.


Objects used
The helm chart values, as per docs for microk8s:

$ helm get values -nkube-system csi-driver-smb
USER-SUPPLIED VALUES:
linux:
  kubelet: /var/snap/microk8s/common/var/lib/kubelet

These are the credentials, PV and PVC I'm trying to use:

apiVersion: v1
kind: Secret
stringData:
  krb5cc_1000: __SMB_SHARE_KRB5CC__
metadata:
  name: smbcreds-krb5
  namespace: proj-ns
type: Opaque
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-smb-krb5
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: smb
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - noperm
    - mfsymlinks
    - cache=strict
    - noserverino # required to prevent data corruption
    - sec=krb5
    - uid=1000
    - cruid=1000
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    volumeHandle: pv-proj-ns-krb5 # make sure it's a unique id in the cluster
    volumeAttributes:
      source: "__SMB_SHARE_URL__"
    nodeStageSecretRef:
      name: smbcreds-krb5
      namespace: proj-ns
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-smb-krb5
  namespace: proj-ns
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: pv-smb-krb5
  storageClassName: smb

An example deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: proj-deploy
  namespace: proj-ns
spec:
  selector:
    matchLabels:
      name: proj-deploy
  replicas: 2
  revisionHistoryLimit: 0
  template:
    metadata:
      labels:
        name: proj-deploy
    spec:
      restartPolicy: Always
      volumes:
        - name: smb
          persistentVolumeClaim:
            claimName: pvc-smb-krb5
      containers:
        - name: test-deploy
          image: nginx
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - mountPath: /data
              name: smb

Anything else we need to know?:

Environment:

  • CSI Driver version: csi-driver-smb-v1.13.0
  • Kubernetes version (use kubectl version): 1.27.8 (snap)
  • OS (e.g. from /etc/os-release): Ubuntu 20.04 LTS
  • Kernel (e.g. uname -a): 5.4.0-169-generic
@andyzhangx
Copy link
Member

andyzhangx commented Dec 14, 2023

@fabioscaccabarozzi current kerberos path is /var/lib/kubelet/kerberos, what's the expected path in your env? is it /var/snap/microk8s/common/var/lib/kubelet/kerberos?

anyway, let me make it configurable first.

@fabioscaccabarozzi
Copy link
Author

@andyzhangx I do not have an expected path - I followed the guides and bits found for microk8s and for kerberos, and I finally realized that the two cannot work together at the moment.

I guess that either providing a configurable path indipendent of linux.kubelet (e.g: host: /var/lib/kerberos, pod: /var/lib/kerberos) or keeping the kerberos path relative to the linux.kubelet path, e.g:

  • linux.kubelet: /var/snap/microk8s/common/var/lib/kubelet
  • host: /var/snap/microk8s/common/var/lib/kubelet/kerberos
  • pod: /var/snap/microk8s/common/var/lib/kubelet/kerberos)

should work.

@andyzhangx
Copy link
Member

so if I set /var/lib/kubelet/kerberos as configurable in this csi driver, it shoud work @fabioscaccabarozzi ?

@fabioscaccabarozzi
Copy link
Author

yes, that's my expectation -> if the CSI driver and the host kerberos/cifs-utils can use the same cache folder with the proper symlinks, everything should work

@andyzhangx
Copy link
Member

#709 could fix this issue

@fabioscaccabarozzi
Copy link
Author

The PR looks good to me, I believe it's the right approach.

One minor nitpick: as it is now, if linux.krb5CacheDirectory is not a sub-path of linux.kubelet, the csi-smb-node pod won't have the mount to share with the host.

I believe we need to add an additional volume here in case users specify something like:

  • linux.kubelet: /var/snap/microk8s/common/var/lib/kubelet
  • linux.krb5CacheDirectory: /var/lib/kerberos

How can I test the PR? Is there some image.*.tag override I can use in the helm chart values?

I should be able to start testing it tomorrow.

Thank you!

@andyzhangx
Copy link
Member

@fabioscaccabarozzi can you move kerberos to /var/snap/microk8s/common/var/lib/kubelet/kerberos path, and then set --krb5-cache-directory=/var/snap/microk8s/common/var/lib/kubelet/kerberos with image andyzhangx/smb-csi:v1.14.0

https://github.com/kubernetes-csi/csi-driver-smb/blob/4a84e512499af588303fb2e31b9b6df547f50538/charts/latest/csi-driver-smb/templates/csi-smb-node.yaml#L111C16-L111C39

@andyzhangx
Copy link
Member

wait, I don't think this PR would work since /var/lib/kubelet/kerberos/ is hardcoded in /etc/krb5.conf in the driver image, I think this requires another PR to map kublet path to /var/lib/kubelet/ inside driver.

RUN echo "[libdefaults]\n    default_ccache_name = FILE:/var/lib/kubelet/kerberos/krb5cc_%{uid}\n" > /etc/krb5.conf

https://github.com/kubernetes-csi/csi-driver-smb/blob/4a84e512499af588303fb2e31b9b6df547f50538/cmd/smbplugin/Dockerfile#L25C116-L25C116

@fabioscaccabarozzi
Copy link
Author

fabioscaccabarozzi commented Dec 15, 2023

Ah, you're right, I didn't even see that part, sorry.

Yes, agree that part also needs to be changed.

Perhaps it could be mounted via a configmap via the helm chart?

@andyzhangx
Copy link
Member

andyzhangx commented Dec 15, 2023

#711 should fix this issue, so you don't need to change any setting, as long as kerberos path is /var/snap/microk8s/common/var/lib/kubelet/kerberos, it should work since inside driver, it's mapping to /var/lib/kubelet/kerberos

@fabioscaccabarozzi
Copy link
Author

#711 will not fix everything -> kerberos utils will create cache files in /var/lib/kubelet/kerberos with absolute symlinks pointing to /var/lib/kubelet/kerberos/<file>, but if host path is mounted from, e.g: /var/lib/kerberos, the host cifs-utils won't be able to use them.

@andyzhangx
Copy link
Member

#714 could fix this issue, this PR adds a mount path for krb5CacheDirectory if it's not empty

@fabioscaccabarozzi
Copy link
Author

This fixes the mount+symlinks, the only outstanding part now is the /etc/krb5.conf

This can be done by via the helm chart, using a ConfigMap to mount the file with the custom config. This way the container image contains the default, and gets overridden if required.

@andyzhangx
Copy link
Member

This fixes the mount+symlinks, the only outstanding part now is the /etc/krb5.conf

This can be done by via the helm chart, using a ConfigMap to mount the file with the custom config. This way the container image contains the default, and gets overridden if required.

@fabioscaccabarozzi what about this fix: #715 ?
inside the driver, the path is always /var/lib/kubelet/kerberos, that would work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants