Namespace stuck in the 'Terminating' state forever when Eclipse Che is uninstalled on OSD v4 and there is no way to fully remove it

[ibuziuk]

Describe the bug

Namespace stuck in the 'Terminating' state when Eclipse Che is uninstalled on OSD v4

Steps to reproduce

pre-requirement OSD v4 cluster + dedicated admin / enhanced dedicated admin permissions

1. create a namespace
2. install Eclipse Che via OperatorHub
3. wait until all the resources are provisioned
4. delete the operator
5. delete the namespace
6. ERROR: namespace stuck in the "Terminating" state:

Expected behavior

The namespace is deleted correctly

Runtime

• OSD v4

Installation method

• OperatorHub with enhanced dedicated admin permissions

Additional context

IMPORTANT

• The culprit is likely related to CheCluster CR in the namespace with oauthclients.finalizers.che.eclipse.org finalizer - Prevent operator removal, while it still has some CR's to manage #13717 (do we really need the finalizers?)
• dedicated admin can NOT edit / remove CR directly. The solution might be grating edit / admin rights on the operator level to admin - https://github.com/coreos/prometheus-operator/pull/2350/files
• dedicated admin can not patch namespaces spec and workaround this issue by manual removal of finalizers from the namespace spec

Documentation

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

[ibuziuk]

Looks like on OSD admin must delete CheCluster from the namespace before uninstalling the operator. If the operator is uninstalled first there is no way to clean up the namespace since user will not have the required rights:

[ibuziuk@fedora ~]$ oc edit CheCluster eclipse-che
error: checlusters.org.eclipse.che "eclipse-che" could not be patched: checlusters.org.eclipse.che "eclipse-che" is forbidden: User "ibuziuk" cannot patch resource "checlusters" in API group "org.eclipse.che" in the namespace "eclipse-che"

Reinstalling the operator in the Terminating namespace also will not work:

operatorgroups.operators.coreos.com is forbidden: User "ibuziuk" cannot create resource "operatorgroups" in API group "operators.coreos.com" in the namespace "eclipse-che"

So, two important outcomes:

• dedicated admin must delete CheCluster CR before uninstalling the operator
• it looks if a namespace stuck in the terminating phase due to finalizer in CR it will not be possible to do anything for cleanup with the dedicated admin permissions

[dmytro-ndp]

Che removal sequence:

oc project <project-name>
oc delete deployment che-operator
oc delete checluster eclipse-che
oc delete project <project-name>

• how to delete terminating Che project

oc patch checluster eclipse-che -p '{ "metadata": { "finalizers": null }}' --type merge -n

• how to delete terminating custom resource definition removal

oc patch customresourcedefinition/checlusters.org.eclipse.che -p '{ "metadata": { "finalizers": null }}' --type merge

[ibuziuk]

@dmytro-ndp dedicated admin on OSD 4 does not have perms for patching

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.