in some cases, the work and member cluster workloads will not be deleted

[will-qq]

Please provide an in-depth description of the question you have:
Our colleagues recently reported that when deleting some custom crd task from karmada, karmada workloads and the resourcebinding was deleted successfully, but the works.work.karmada.io and member cluster workloads were not actually deleted.

In most cases, deletion can be successful, but in a few cases, deletion fails. The problem is similar to #708, but we are very sure that the karmada version we are using has covered the relevant code.

Problem Description:
From karmada, the object has been deleted:

# kubectl get xxx.bigdata.xxx.k8s.io -n xxx-xxx-xxx xxx-325610187
Error from server (NotFound): xxx.bigdata.xxxx.k8s.io "xxx-325610187" not found

But the work has not been deleted:

# kubectl get work -n karmada-es-gpu-xx-4   xxxx-325610187-5b5bb5b65   -oyaml | less
apiVersion: work.karmada.io/v1alpha1
kind: Work
metadata:
  creationTimestamp: "2023-12-26T11:55:26Z"
  finalizers:
  - karmada.io/execution-controller
  generation: 1
  labels:
    resourcebinding.karmada.io/name: xxxx-325610187-xxxx
    resourcebinding.karmada.io/namespace: xxxxx
  name: xxxx-325610187-5b5bb5b65
  namespace: karmada-es-gpu-xx-4
  resourceVersion: "5165857701"
  selfLink: /apis/work.karmada.io/v1alpha1/namespaces/karmada-es-gpu-nj-4/works/pytorch-325610187-5b5bb5b65
  uid: b2c96c56-3493-4279-ac20-a1de73cd720a
spec:
  workload:
    manifests:

And From member cluster, the object also not been deleted:

# kubectl get xxx.bigdata.xxx.k8s.io -n xxx-xxx-xxx xxx-325610187
NAME                AGE
xxx-325610187   39h

What do you think about this question?:
Is it a bug or are we using it improperly?

Environment:

• Karmada version:
• Kubernetes version:
• Others:

[chaunceyjiang]

It seems to be the same issue as #4467.

Could you check if your ResourceBinding xxxx-325610187-xxxx still exists, and if so, could you paste it here as well?

[will-qq]

It seems to be the same issue as #4467.

Could you check if your ResourceBinding xxxx-325610187-xxxx still exists, and if so, could you paste it here as well?

ResourceBinding is not exists, but work and member cluster workloads still exists

From karmada, ResourceBinding is not exists:

# kubectl get rb -n xxx-xxx-xxx  xxx-325610187-xxx
Error from server (NotFound): resourcebindings.work.karmada.io "xxx-325610187-xxx" not found

[zhzhuang-zju]

It's very strange that after rb has been deleted, the work still does not have a DeletionTimestamp. Can you provide the log of karmada-controller-manager when performing the deletion operation? Or confirm that the namesapce and name of the rb of the work labels are consistent with the deleted ones.

[liangyuanpeng]

@will-qq It is resolved? It would be great if you would share how to resolve it.