[BUG] KubeBlocks Uninstall - CRDs delete gets stuck

[gstefanoff]

Describe the bug
Installed KubeBlocks (version 0.9.2 ) via helm using these instructions

When I tried uninstalling with helm, I was not able to delete CRDs. Every attempt resulted in delete processes being stuck. I am not able to delete the namespace because k8s is waiting for 23 resources to be removed, the problem is I cannot remove them either because they get stuck as well the only solution, I have not tried yet is the one mentioned here.

To Reproduce
Steps to reproduce the behavior:

1. Delete pods (posgres, redis)
2. Uninstall addons used
3. helm uninstall kubeblocks --namespace kb-system
4. kubectl get crd -o name | grep kubeblocks.io | xargs kubectl delete
5. kubectl delete namespace kb-system

Expected behavior
KubeBlocks should be as easily removed as it is installed. Having to resort to "workarounds" such as removing finalizers is not feasible solution.

Screenshots

Not actually deleted!

Additional context
logs from kb-system namespaces pending deletion.

Name:         kb-system
Labels:       kubernetes.io/metadata.name=kb-system
Annotations:  <none>
Status:       Terminating
Conditions:
  Type                                         Status  LastTransitionTime               Reason                Message
  ----                                         ------  ------------------               ------                -------
  NamespaceDeletionDiscoveryFailure            False   Wed, 15 Jan 2025 10:06:48 -0800  ResourcesDiscovered   All resources successfully discovered
  NamespaceDeletionGroupVersionParsingFailure  False   Fri, 10 Jan 2025 11:38:49 -0800  ParsedGroupVersions   All legacy kube types successfully parsed
  NamespaceDeletionContentFailure              False   Sun, 12 Jan 2025 00:33:40 -0800  ContentDeleted        All content successfully deleted, may be waiting on finalization
  NamespaceContentRemaining                    True    Sun, 12 Jan 2025 00:33:40 -0800  SomeResourcesRemain   Some resources are remaining: configmaps. has 23 resource instances
  NamespaceFinalizersRemaining                 True    Sun, 12 Jan 2025 00:33:40 -0800  SomeFinalizersRemain  Some content in the namespace has finalizers remaining: config.kubeblocks.io/finalizer in 23 resource instances

[shanshanying]

Hi @gstefanoff

It seems even though you uninstall addons, resources like actionssets, addons are not deleted actually.

1. may I know how do you uninstall addons?
2. have you deleted all clusters properly before uninstall addons ? (e.g. k delete cluster )

[gstefanoff]

Hi @shanshanying
I uninstalled addons using kbcli addon uninstall (first disabled) I should mention that some addons (MariaDB) I had to install using helm because the recommended method did not work.
To remove cluster, I did not used the command you're referencing because when I checked if there are any clusters, I got no resource type cluster. So I deleted clusterdefinitions (k delete clusterdefinition )

[shanshanying]

Hi @gstefanoff

So what is the list of clusterdefinition. clusterversion, after uninstalling Addons, leftover in your env?
e.g. using

k get addon,cd,cv,cc,bpt,actionsets

and check the list of helm charts intalled in ns kb-system:

helm -n kb-system list

[gstefanoff]

Hi @shanshanying
It looks like not everything has been removed.
Is there specific order these resources must be removed in?

helm -n kb-system list return empty list

k get bpt => the server doesn't have a resource type "bpt"

k get cc
mongodb-config-constraints           Available   47d
mongodb-env-constraints              Available   47d
oracle-mysql5.7-config-constraints   Available   47d
oracle-mysql8.0-config-constraints   Available   47d
postgresql12-cc                      Available   47d
postgresql14-cc                      Available   47d
postgresql15-cc                      Available   47d
postgresql16-cc                      Available   47d
redis-cluster7-config-constraints    Available   47d
redis7-config-constraints            Available   47d

k get cv
NAME                 CLUSTER-DEFINITION   STATUS      AGE
etcd-v3.5.6          etcd                 Available   47d
mariadb-10.6.15      mariadb              Available   13d
mongodb-4.2          mongodb              Available   47d
mongodb-4.4          mongodb              Available   47d
mongodb-5.0          mongodb              Available   47d
mongodb-6.0          mongodb              Available   47d
mysql-5.7.44         mysql                Available   47d
mysql-8.0.33         mysql                Available   47d
mysql-8.4.2          mysql                Available   47d
postgresql-12.14.0   postgresql           Available   47d
postgresql-12.14.1   postgresql           Available   47d
postgresql-12.15.0   postgresql           Available   47d
postgresql-14.7.2    postgresql           Available   47d
postgresql-14.8.0    postgresql           Available   47d
postgresql-15.7.0    postgresql           Available   47d
postgresql-16.4.0    postgresql           Available   47d
redis-7.0.6          redis                Available   47d
redis-7.2.4          redis 

k get cv
etcd-v3.5.6          etcd                 Available   47d
mariadb-10.6.15      mariadb              Available   13d
mongodb-4.2          mongodb              Available   47d
mongodb-4.4          mongodb              Available   47d
mongodb-5.0          mongodb              Available   47d
mongodb-6.0          mongodb              Available   47d
mysql-5.7.44         mysql                Available   47d
mysql-8.0.33         mysql                Available   47d
mysql-8.4.2          mysql                Available   47d
postgresql-12.14.0   postgresql           Available   47d
postgresql-12.14.1   postgresql           Available   47d
postgresql-12.15.0   postgresql           Available   47d
postgresql-14.7.2    postgresql           Available   47d
postgresql-14.8.0    postgresql           Available   47d
postgresql-15.7.0    postgresql           Available   47d
postgresql-16.4.0    postgresql           Available   47d
redis-7.0.6          redis                Available   47d
redis-7.2.4          redis                Available   47d

k get cd 
NAME         TOPOLOGIES                                     SERVICEREFS   STATUS      AGE
etcd                                                                      Available   47d
mariadb                                                                   Available   13d
mongodb                                                                   Available   47d
mysql                                                                     Available   47d
postgresql                                                                Available   47d
redis        replication,replication-twemproxy,standalone  

k get addon -n kb-system
NAME                           TYPE   VERSION   PROVIDER   STATUS     AGE
alertmanager-webhook-adaptor   Helm                        Disabled   47d
apecloud-mysql                 Helm                        Disabled   47d
apecloud-otel-collector        Helm                        Disabled   47d
aws-load-balancer-controller   Helm                        Disabled   47d
csi-driver-nfs                 Helm                        Disabled   47d
csi-hostpath-driver            Helm                        Disabled   47d
etcd                           Helm                        Enabled    47d
grafana                        Helm                        Disabled   47d
kafka                          Helm                        Disabled   47d
llm                            Helm                        Disabled   47d
mariadb                        Helm                        Enabled    13d
mongodb                        Helm                        Enabled    47d
mysql                          Helm                        Enabled    47d
postgresql                     Helm                        Enabled    47d
prometheus                     Helm                        Disabled   47d
pulsar                         Helm                        Disabled   47d
qdrant                         Helm                        Disabled   47d
redis                          Helm                        Enabled    47d
snapshot-controller            Helm                        Enabled    47d
victoria-metrics-agent         Helm                        Disabled   47d

[shanshanying]

Addons listed are not removed while uninstalling KB using 'helm unintall.
It seems these Addons are annotatied with helm.sh/resource-poliyc:keep.

To uninstall KubeBlocks, one should

1. delete clusters, backups if any
2. uninstall all Addons
3. uninstall KubeBlocks.

Or you can use our command line tool

kbcli kb uninstall --auto-approve

For your case, since KB has been unisntalled and no one handles these CRs anymore. you may patch these reousrce to remove finalizers and remove them, e.g.
e.g.

# patch all clusterdefinition, remove finalizers
 kubectl get cd -oname | xargs -I {} kubectl patch {}  --type=merge -p '{"metadata":{"finalizers":[]}}'
# patch all addons to remove finalizers
 kubectl get addon -oname | xargs -I {} kubectl patch {}  --type=merge -p '{"metadata":{"finalizers":[]}}'

replace addon, cd in above exampels with resource names (crd name) leftover in your envrionemnt.

[gstefanoff]

Hi @shanshanying

I removed cc,cd,addon, crds as you advised. However, configmaps are still in the namespace. Deletes get stuck and the namespace is waiting for them to be removed. These configmaps are managed by helm btw.
What is the best way to remove them?

kubectl api-resources --verbs=list --namespaced -oname | xargs -n 1 kubectl get --show-kind --ignore-not-found -n kb-system
NAME                                        DATA   AGE
configmap/mongodb-env-tpl                   1      49d
configmap/mongodb-metrics-config-new        1      49d
configmap/mongodb-scripts                   6      49d
configmap/mongodb5.0-config-template        2      49d
configmap/mysql-agamotto-configuration      2      49d
configmap/mysql-scripts                     5      49d
configmap/oracle-mysql5.7-config-template   1      49d
configmap/oracle-mysql8.0-config-template   1      49d
configmap/pgbouncer-configuration           1      49d
configmap/postgresql-configuration          4      49d
configmap/postgresql-scripts                4      49d
configmap/postgresql12-configuration        4      49d
configmap/postgresql12-custom-metrics       1      49d
configmap/postgresql14-custom-metrics       1      49d
configmap/postgresql15-configuration        4      49d
configmap/postgresql15-custom-metrics       1      49d
configmap/postgresql16-configuration        4      49d
configmap/postgresql16-custom-metrics       1      49d
configmap/redis-cluster-scripts             4      49d
configmap/redis-cluster7-config-template    1      49d
configmap/redis-metrics-config              1      49d
configmap/redis-scripts                     11     49d
configmap/redis7-config-template            1      49d

Thanks.

[sophon-zt]

Hi @gstefanoff

In kb 0.9, the ownership of these configmaps was transferred from cd/cv to cmpd. Please check the status of cmpd.
using k get cmpd

There's a related bug has been fixed: #8188

Please show the metadata of the leftover configmap(k get cm -n kb-system -l config.kubeblocks.io/config-type=tpl). e.g: mongodb-env-tpl

k get cm -n kb-system mongodb-env-tpl -o jsonpath='{.metadata}' |jq

[gstefanoff]

Hi @sophon-zt

Here's the output

  "annotations": {
    "meta.helm.sh/release-name": "kb-addon-mongodb",
    "meta.helm.sh/release-namespace": "kb-system"
  },
  "creationTimestamp": "2024-12-06T00:05:30Z",
  "deletionGracePeriodSeconds": 0,
  "deletionTimestamp": "2025-01-10T19:38:49Z",
  "finalizers": [
    "config.kubeblocks.io/finalizer"
  ],
  "labels": {
    "app.kubernetes.io/instance": "kb-addon-mongodb",
    "app.kubernetes.io/managed-by": "Helm",
    "app.kubernetes.io/name": "mongodb",
    "app.kubernetes.io/version": "5.0",
    "config.kubeblocks.io/config-type": "tpl",
    "helm.sh/chart": "mongodb-0.9.1"
  },
  "managedFields": [
    {
      "apiVersion": "v1",
      "fieldsType": "FieldsV1",
      "fieldsV1": {
        "f:data": {
          ".": {},
          "f:env": {}
        },
        "f:metadata": {
          "f:annotations": {
            ".": {},
            "f:meta.helm.sh/release-name": {},
            "f:meta.helm.sh/release-namespace": {}
          },
          "f:labels": {
            ".": {},
            "f:app.kubernetes.io/instance": {},
            "f:app.kubernetes.io/managed-by": {},
            "f:app.kubernetes.io/name": {},
            "f:app.kubernetes.io/version": {},
            "f:helm.sh/chart": {}
          }
        }
      },
      "manager": "helm",
      "operation": "Update",
      "time": "2024-12-06T00:05:30Z"
    },
    {
      "apiVersion": "v1",
      "fieldsType": "FieldsV1",
      "fieldsV1": {
        "f:metadata": {
          "f:finalizers": {
            ".": {},
            "v:\"config.kubeblocks.io/finalizer\"": {}
          },
          "f:labels": {
            "f:config.kubeblocks.io/config-type": {}
          }
        }
      },
      "manager": "kubeblocks",
      "operation": "Update",
      "time": "2024-12-06T00:05:30Z"
    }
  ],
  "name": "mongodb-env-tpl",
  "namespace": "kb-system",
  "resourceVersion": "122466013",
  "uid": "6569f1bd-0a03-4cd8-abb7-8519f35bce69"
}