Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

manifests: updater mount the directory, not the socket #207

Merged

Conversation

ffromani
Copy link
Collaborator

@ffromani ffromani commented Aug 2, 2023

Quoting kube docs:

when accessing the `/var/lib/kubelet/pod-resources/kubelet.sock` from DaemonSet
or any other app deployed as a container on the host, which is mounting socket as
a volume, it is a good practice to mount directory `/var/lib/kubelet/pod-resources/`
instead of the `/var/lib/kubelet/pod-resources/kubelet.sock`. This will ensure
that after kubelet restart, container will be able to re-connect to this socket.

Container mounts are managed by inode referencing the socket or directory,
depending on what was mounted. When kubelet restarts, socket is deleted
and a new socket is created, while directory stays untouched.
So the original inode for the socket become unusable. Inode to directory
will continue working.

Quoting kube docs:

```
when accessing the `/var/lib/kubelet/pod-resources/kubelet.sock` from DaemonSet
or any other app deployed as a container on the host, which is mounting socket as
a volume, it is a good practice to mount directory `/var/lib/kubelet/pod-resources/`
instead of the `/var/lib/kubelet/pod-resources/kubelet.sock`. This will ensure
that after kubelet restart, container will be able to re-connect to this socket.

Container mounts are managed by inode referencing the socket or directory,
depending on what was mounted. When kubelet restarts, socket is deleted
and a new socket is created, while directory stays untouched.
So the original inode for the socket become unusable. Inode to directory
will continue working.
```

Signed-off-by: Francesco Romani <[email protected]>
@ffromani ffromani force-pushed the more-robust-podres-mount branch from 6edffb0 to 02bbea5 Compare August 2, 2023 16:16
@ffromani ffromani requested review from Tal-or and swatisehgal August 2, 2023 17:03
@Tal-or
Copy link
Contributor

Tal-or commented Aug 2, 2023

This is interesting. So IOW mounting a volume as socket and restarting kubelet causes the container to failed to connect the socket?

@ffromani
Copy link
Collaborator Author

ffromani commented Aug 3, 2023

This is interesting. So IOW mounting a volume as socket and restarting kubelet causes the container to failed to connect the socket?

I never tested or seen this myself in the wild, but there are reports in the community of similar incidents with other software components mounting in a similar way. The rationale for the change seems solid and the change is expected to make no difference in the happy path, so I'm proposing the fix preemptively.

@Tal-or
Copy link
Contributor

Tal-or commented Aug 3, 2023

/approve
/lgtm

@ffromani ffromani merged commit de34913 into k8stopologyawareschedwg:main Aug 3, 2023
@ffromani ffromani deleted the more-robust-podres-mount branch August 3, 2023 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants