[BUG]: Container is terminated but Pod is stuck in terminating

[hiteshmathur19]

Describe the bug
On deleting a pod which is using powerflex csi for mounting volume, the pod stuck in terminating for ever.

To Reproduce
Steps to reproduce the behavior:
kubectl get po -A -o wide| grep Termin
chf-apps csdpbl002-0 0/1 Terminating 0

kubectl get pvc -A | grep csdpbl002-0
chf-apps pv-fds-csdpbl002-0 Bound ccd-af9659f2b3

On creation of pod :
MountVolume.WaitForAttach succeeded for volume "ccd-af9659f2b3"
device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/ccd-af9659f2b3/globalmount"

On deleting the pod:
/volume-subpaths/ccd-af9659f2b3/sdp/0" is not a mountpoint, deleting

But the mountpoint couldn’t be deleted, since its already in use by the application.
umount: /var/lib/kubelet/pods/69fd83d4-eb2c-4eaf-aa2e-8436319070f6/volumes/kubernetes.io~csi/ccd-af9659f2b3/mount: target is busy.\n"

After sometime volume did get detached. But the pod still get stuck in terminating.
Volume detached for volume "ccd-af9659f2b3\

After restarting the kubelet:

"operationExecutor.UnmountVolume started for volume "ccd-af9659f2b3"
UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/csi-vxflexos.dellemc.com^62ce8cd81be7f10f-eaf5ab8a00000502
failed to open volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/ccd-af9659f2b3/vol_data.json]: open /var/lib/kubelet/plugins/kubernetes.io/csi/pv/ccd-af9659f2b3/vol_data.json: no such file or directory

Only those pods which have error "target is busy" , showing this problem of stuck in terminating.

Expected behavior
On deleting pod, it should successfully get deleted.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
If applicable, submit logs or stack traces from the affected services

System Information (please complete the following information):

• OS/Version: Suse
• Kubernetes 1.21
• Attaching the logs

Additional context
Add any other context about the problem here.
driver.logs.20211227_0335.zip
kubelet.zip

[hiteshmathur19]

Any update on this please ?

[prablr79]

@bharathsreekanth customer is looking for updates here.

[bharathsreekanth]

I will have a look at this today and discuss with the team and get back. Thanks!

[nb950]

as per node logs volume unpublish report "target is busy" . assume this is true , you are shutting down pod with application running ?
next we see driver node log shows "volume XX is not published to node"
time="2021-12-21T16:16:14Z" level=info msg="SDC returned volume eaf5ab8a00000502 on system 62ce8cd81be7f10f not published to node"
the controller unpublish could have done this , however logs are missing , please send us driver controller logs ?
the podmon logs are from 12/27 and the driver logs are from 12/21

can you help retest and get all the logs immediately as the issue occurs ?
is there something different about the app on the pod , can you share any info ,can it cause the node network to appear frozen which it turn forces podmon to do something different ?

[bharathsreekanth]

Based on the last update, the podmon issue needs to to triaged first. Assigning to @alikdell to triage.

[alikdell]

Send email requesting customer to reproduce the issue and send us fresh logs.
Customer agreed to provide us new sets of logs whenever they are able reproduce the issue at their end.

[alikdell]

Not able to reproduce, closing the issue for now.
If issue observed again in future this ticket will be reopened.