No easy way how to update CSI driver that uses fuse

[jsafrane]

We recommend to use DaemonSet to run CSI drivers on node. If a driver runs fuse daemon, it's almost impossible to update it, as killing a pod with the driver kills the fuse daemons too and it will kill all mounts, possibly corrupting application data.

We need a documented and supported way how to update such CSI drivers. Note that the update process can be manual or the code can live somewhere else, we just need it to to be documented and supported so people don't loose data.

/sig storage
@msau42 @davidz627 @saad-ali @pohly @vladimirvivien @verult @lpabon @jingxu97 @gnufied

[lpabon]

@jsafrane I think that is the responsibility of the driver, in my opinion. Although we could definitely provide guidelines, we shouldn't provide a solution, since we cannot control their release or update processes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[davidz627]

/remove-lifecycle stale

[fredkan]

Expect there is a common solution for this issue.

One workaround: https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[farcaller]

/remove-lifecycle stale

[jim3ma]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.

Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jim3ma]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[davidz627]

/remove-lifecycle stale

[andyzhangx]

I hit similar issue, make fuse driver standalone is one solution;
another problem is that how to recover when those mounting paths are already broken when fuse csi driver is restarted:
so when fuse driver is broken(sometimes due to csi driver pod restarted), the stage volume path is broken, following code will return error directly:

kubernetes/pkg/volume/csi/csi_attacher.go

Lines 231 to 234 in e4a5012

	mounted, err := isDirMounted(c.plugin, deviceMountPath)
	if err != nil {
	klog.Error(log("attacher.MountDevice failed while checking mount status for dir [%s]", deviceMountPath))
	return err

is it possible only return error when it's not IsCorruptedMnt, like what flexvolume did:

kubernetes/pkg/volume/flexvolume/detacher.go

Lines 60 to 61 in e4a5012

	if pathErr != nil && !mount.IsCorruptedMnt(pathErr) {
	return fmt.Errorf("Error checking path: %v", pathErr)

So in that case, even fuse driver is broken, we could make sure if fuse csi driver has remount logic, it could recover after fuse driver is back.
WDYT? I could make code change if that's an acceptable workaround.

[andyzhangx]

I worked out a PR(#88569) to mitigate this issue when fuse driver on the node is restarted(mount point is corrupted), could someone take a look? Thanks.

[andyzhangx]

/open
keep the issue open since PR(#88569) is only a mitigation

[Ark-kun]

/reopen

[k8s-ci-robot]

@Ark-kun: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andyzhangx]

/reopen

[k8s-ci-robot]

@andyzhangx: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@jsafrane: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jim3ma]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.

Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

We implement a solution - transferring fds in csi pods with surging rolling update.

[andyzhangx]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.
Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

We implement a solution - transferring fd in csi pod with surging rolling update.

@jim3ma do you have the details of the solution? thanks.

[jim3ma]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.
Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

We implement a solution - transferring fd in csi pod with surging rolling update.

@jim3ma do you have the details of the solution? thanks.

Currently, this solution is using in Ant Group, we will open source in Dragonfly Image Service some times later.

Another solution: csi container starts a fuse session in another container in host not in pod, then we can update csi pod with keeping a fuse session containers. this likes the solution from Alibaba Cloud OSS plugin but within our control.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xhejtman]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.
Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

We implement a solution - transferring fd in csi pod with surging rolling update.

@jim3ma do you have the details of the solution? thanks.

Currently, this solution is using in Ant Group, we will open source in Dragonfly Image Service some times later.

Another solution: csi container starts a fuse session in another container in host not in pod, then we can update csi pod with keeping a fuse session containers. this likes the solution from Alibaba Cloud OSS plugin but within our control.

did you open source already?

[andyzhangx]

@xhejtman Azure Blob CSI driver uses similar fuse proxy solution, and it's open source, check details here: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/deploy/blobfuse-proxy

[xhejtman]

Thank. I was also interested in truly restartable fuse driver, at least I understand what @jim3ma talk about - track everything fuse needs for restart.

[jim3ma]

Available solution from Alibaba Cloud OSS plugin:
https://github.com/AliyunContainerService/csi-plugin/blob/master/docs/oss-upgrade.md
This solution is making fuse processes running in node's root namespaces and cgroup, so when daemonset pod upgrade, fuse processes will not be killed.
Another solution: Buddy daemonset pods(two daemonset pods work as buddy)
When need upgrade, the buddy pod will take all fd(s) from original pod and then serve csi service.
This solution needs some code changes for fuse drivers.

We implement a solution - transferring fd in csi pod with surging rolling update.

@jim3ma do you have the details of the solution? thanks.

Currently, this solution is using in Ant Group, we will open source in Dragonfly Image Service some times later.
Another solution: csi container starts a fuse session in another container in host not in pod, then we can update csi pod with keeping a fuse session containers. this likes the solution from Alibaba Cloud OSS plugin but within our control.

did you open source already?

We are trying to merge some code into upstream fuse driver to make this solution perfect in some corner cases. And someday we will make our solution open source with a real project.

[ofek]

@jim3ma Is it open source now?