Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated" #46

Closed
etoews opened this issue Apr 29, 2020 · 8 comments

Comments

@etoews
Copy link
Contributor

etoews commented Apr 29, 2020

I got 4.3.8 installed and everything is working fine.

However, when I go to upgrade, I get the following error.

$ oc adm upgrade --to-latest=true
Updating to latest version 4.3.13

$ oc describe clusterversion
Name:         version
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterVersion
Metadata:
  Creation Timestamp:  2020-04-28T00:47:17Z
  Generation:          2
  Resource Version:    953549
  Self Link:           /apis/config.openshift.io/v1/clusterversions/version
  UID:                 f67845db-4ecb-4c74-a8c1-73bda2820f2f
Spec:
  Channel:     stable-4.3
  Cluster ID:  af5828d9-24ea-4234-805e-b1d595c528e9
  Desired Update:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61
    Version:  4.3.13
  Upstream:   https://api.openshift.com/api/upgrades_info/v1/graph
Status:
  Available Updates:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61
    Version:  4.3.13
  Conditions:
    Last Transition Time:  2020-04-28T01:36:39Z
    Message:               Done applying 4.3.8
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-04-29T23:18:39Z
    Message:               Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated": Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [hostmount-anyuid]
    Reason:                UpgradePreconditionCheckFailed
    Status:                True
    Type:                  Failing
    Last Transition Time:  2020-04-29T23:18:24Z
    Message:               Unable to apply 4.3.13: it may not be safe to apply this update
    Reason:                UpgradePreconditionCheckFailed
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2020-04-28T00:50:29Z
    Status:                True
    Type:                  RetrievedUpdates
    Last Transition Time:  2020-04-28T02:27:51Z
    Message:               Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [hostmount-anyuid]
    Reason:                DefaultSecurityContextConstraints_Mutated
    Status:                False
    Type:                  Upgradeable
  Desired:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61
    Version:  4.3.13
  History:
    Completion Time:    <nil>
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61
    Started Time:       2020-04-29T23:18:24Z
    State:              Partial
    Verified:           true
    Version:            4.3.13
    Completion Time:    2020-04-28T01:36:39Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:a414f6308db72f88e9d2e95018f0cc4db71c6b12b2ec0f44587488f0a16efc42
    Started Time:       2020-04-28T00:47:25Z
    State:              Completed
    Verified:           false
    Version:            4.3.8
  Observed Generation:  2
  Version Hash:         lnZzahlL8hk=
Events:                 <none>

The SCC was altered by this line in nfs-provisioner-setup.sh.

Is that line effectively preventing the upgrade?

@canit00
Copy link
Contributor

canit00 commented Apr 29, 2020

This is very interesting. Finding myself about to try an upgrade. I don't have intimate knowledge, here to learn & help. @etoews have you come across this BZ-1821905 yet? Seems like back-ported to 4.3, I may be able to test my setup tomorrow.

@etoews
Copy link
Contributor Author

etoews commented Apr 29, 2020

I removed the alteration

$ oc adm policy remove-scc-from-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner
securitycontextconstraints.security.openshift.io/hostmount-anyuid removed from: ["system:serviceaccount:nfs-provisioner:nfs-client-provisioner"]

and a couple of minutes later the cluster upgrade continued.

I'm waiting for that to complete but I expect this will simply come down to a matter of how to document the upgrade steps here and here.

Do you

$ oc adm policy remove-scc-from-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner
$ oc adm upgrade --to-latest=true
$ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner

or

$ oc adm upgrade --to-latest=true --allow-upgrade-with-warnings

@etoews
Copy link
Contributor Author

etoews commented Apr 29, 2020

@canit00 I hadn't come across BZ-1821905. Thanks for the link!

I think it'll provide good input into the decision on which way to document the upgrades steps.

@etoews
Copy link
Contributor Author

etoews commented Apr 30, 2020

My cluster upgraded cleanly to 4.3.13 🎉

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.13    True        False         161m    Cluster version is 4.3.13

effectively using these commands

$ oc adm policy remove-scc-from-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner
$ oc adm upgrade --to-latest=true
$ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner

@christianh814 Given BZ-1821905, do you have a preference for

$ oc adm policy remove-scc-from-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner
$ oc adm upgrade --to-latest=true
$ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner

or

$ oc adm upgrade --to-latest=true --allow-upgrade-with-warnings

to upgrade the cluster?

(I haven't actually tried oc adm upgrade --to-latest=true --allow-upgrade-with-warnings yet)

@christianh814
Copy link
Contributor

@etoews

I think I'd prefer

oc adm upgrade --to-latest=true --allow-upgrade-with-warnings

This is because removing the scc might "break" storage during the upgrade for any apps running. Although this helper is mainly for POCs...we still should try not to break things where possible :)

Although we should also call out the other method and say "only do this if the other method doesn't work" sort of thing.

@etoews
Copy link
Contributor Author

etoews commented Apr 30, 2020

Cool. I'll send a PR early next week after I've had a chance to properly test oc adm upgrade --to-latest=true --allow-upgrade-with-warnings. Currently we tear down our OCP4 PoC cluster on a weekly basis so the upgrade will be the first thing I do next week.

@etoews
Copy link
Contributor Author

etoews commented May 4, 2020

--allow-upgrade-with-warnings didn't work but --force did.

$ oc adm upgrade --to-latest --allow-upgrade-with-warnings
Updating to latest version 4.3.18

$ oc describe clusterversion
Name:         version
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterVersion
Metadata:
  Creation Timestamp:  2020-05-04T03:50:03Z
  Generation:          6
  Resource Version:    306802
  Self Link:           /apis/config.openshift.io/v1/clusterversions/version
  UID:                 1874d0a8-a4fa-480d-9b49-72a0f87c7bcc
Spec:
  Channel:     stable-4.3
  Cluster ID:  1af0959f-0cc1-4a8c-82c4-f2162091e0c9
  Desired Update:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:1f0fd38ac0640646ab8e7fec6821c8928341ad93ac5ca3a48c513ab1fb63bc4b
    Version:  4.3.18
  Upstream:   https://api.openshift.com/api/upgrades_info/v1/graph
Status:
  Available Updates:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:e1ebc7295248a8394afb8d8d918060a7cc3de12c491283b317b80b26deedfe61
    Version:  4.3.13
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:1f0fd38ac0640646ab8e7fec6821c8928341ad93ac5ca3a48c513ab1fb63bc4b
    Version:  4.3.18
  Conditions:
    Last Transition Time:  2020-05-04T05:16:36Z
    Message:               Done applying 4.3.8
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-05-04T22:07:51Z
    Message:               Precondition "ClusterVersionUpgradeable" failed because of "DefaultSecurityContextConstraints_Mutated": Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [hostmount-anyuid]
    Reason:                UpgradePreconditionCheckFailed
    Status:                True
    Type:                  Failing
    Last Transition Time:  2020-05-04T22:07:51Z
    Message:               Unable to apply 4.3.18: it may not be safe to apply this update
    Reason:                UpgradePreconditionCheckFailed
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2020-05-04T03:50:09Z
    Status:                True
    Type:                  RetrievedUpdates
    Last Transition Time:  2020-05-04T05:42:06Z
    Message:               Cluster operator kube-apiserver cannot be upgraded: DefaultSecurityContextConstraintsUpgradeable: Default SecurityContextConstraints object(s) have mutated [hostmount-anyuid]
    Reason:                DefaultSecurityContextConstraints_Mutated
    Status:                False
    Type:                  Upgradeable
  Desired:
    Force:    false
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:1f0fd38ac0640646ab8e7fec6821c8928341ad93ac5ca3a48c513ab1fb63bc4b
    Version:  4.3.18

$ oc adm upgrade --clear
Cleared the update field, still at 4.3.18

Waited a while until the cluster was completely back to 4.3.8.

$ oc adm upgrade --to-latest --force
Updating to latest version 4.3.18

$ oc describe clusterversion
Name:         version
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterVersion
Metadata:
  Creation Timestamp:  2020-05-04T03:50:03Z
  Generation:          8
  Resource Version:    336479
  Self Link:           /apis/config.openshift.io/v1/clusterversions/version
  UID:                 1874d0a8-a4fa-480d-9b49-72a0f87c7bcc
Spec:
  Channel:     stable-4.3
  Cluster ID:  1af0959f-0cc1-4a8c-82c4-f2162091e0c9
  Desired Update:
    Force:    true
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:1f0fd38ac0640646ab8e7fec6821c8928341ad93ac5ca3a48c513ab1fb63bc4b
    Version:  4.3.18
  Upstream:   https://api.openshift.com/api/upgrades_info/v1/graph
Status:
  Available Updates:  <nil>
  Conditions:
    Last Transition Time:  2020-05-04T05:16:36Z
    Message:               Done applying 4.3.18
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-05-04T22:36:00Z
    Status:                False
    Type:                  Failing
    Last Transition Time:  2020-05-04T22:44:57Z
    Message:               Cluster version is 4.3.18
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-05-04T03:50:09Z
    Status:                True
    Type:                  RetrievedUpdates
  Desired:
    Force:    true
    Image:    quay.io/openshift-release-dev/ocp-release@sha256:1f0fd38ac0640646ab8e7fec6821c8928341ad93ac5ca3a48c513ab1fb63bc4b
    Version:  4.3.18

@christianh814
Copy link
Contributor

Closed by f90299c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants