Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1884334: UpdateError: enhance for ability to determine when upgrade failing #486

Merged
merged 1 commit into from
Feb 3, 2021
Merged

Bug 1884334: UpdateError: enhance for ability to determine when upgrade failing #486

merged 1 commit into from
Feb 3, 2021

Conversation

jottofar
Copy link
Contributor

@jottofar jottofar commented Dec 8, 2020 •
edited
Loading

Currently during an upgrade the cluster will be marked as failing when an operator is still updating but not necessarily failing. Add the ability to set whether a given cluster operator error should result in the cluster being set to failing.

The bool UpgradeSucceeding was added to the UpdateError structure to indicate whether the error should result in Failing=True. The name was chosen such that !UpgradeSucceeding== Failing. In this way the default for a given error is that it results in the cluster being marked as failing and explicit action must be taken by a developer, for example when a new UpdateError is added, to determine and set the error such that the cluster is not marked as failing. I chose to stick with a new field in UpdateError as opposed to an UpdateError sibling to avoid having to change method signatures especially since this first round is only addressing cluster operator errors.

This PR also adds the ability to track each cluster operator's sync'ing time such that those marked as ClusterOperatorNotAvailablewill not result in cluster Failing=True unless sync'ing time has exceeded 40 minutes.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 8, 2020
@openshift-ci-robot
Copy link
Contributor

@jottofar: This pull request references Bugzilla bug 1884334, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

WIP: Bug 1884334: still updating is not always failure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 8, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 8, 2020
@jottofar
Copy link
Contributor Author

jottofar commented Dec 8, 2020

/test unit

@jottofar jottofar force-pushed the bug-1884334 branch 6 times, most recently from ad4fd97 to 6c36937 Compare December 10, 2020 14:34
@jottofar
Copy link
Contributor Author

/retitle Bug 1884334: UpdateError: add and use Progressing field

@openshift-ci-robot openshift-ci-robot changed the title WIP: Bug 1884334: still updating is not always failure Bug 1884334: UpdateError: add and use Progressing field Dec 10, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2020
@openshift-ci-robot
Copy link
Contributor

@jottofar: This pull request references Bugzilla bug 1884334, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1884334: UpdateError: add and use Progressing field

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking
wking reviewed Dec 10, 2020
View reviewed changes
pkg/cvo/status.go Outdated Show resolved Hide resolved
@jottofar
Copy link
Contributor Author

/test ci/prow/e2e-agnostic-upgrade

@openshift-ci-robot
Copy link
Contributor

@jottofar: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

  • /test e2e-agnostic
  • /test e2e-agnostic-operator
  • /test e2e-agnostic-upgrade
  • /test e2e-metal-assisted
  • /test gofmt
  • /test images
  • /test unit

Use /test all to run the following jobs:

  • pull-ci-openshift-cluster-version-operator-master-e2e-agnostic
  • pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-operator
  • pull-ci-openshift-cluster-version-operator-master-e2e-agnostic-upgrade
  • pull-ci-openshift-cluster-version-operator-master-gofmt
  • pull-ci-openshift-cluster-version-operator-master-images
  • pull-ci-openshift-cluster-version-operator-master-unit

In response to this:

/test ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jottofar
Copy link
Contributor Author

/test e2e-agnostic-upgrade

1 similar comment
@jottofar
Copy link
Contributor Author

/test e2e-agnostic-upgrade

@jottofar jottofar force-pushed the bug-1884334 branch 5 times, most recently from 8193ef0 to ead8788 Compare December 15, 2020 02:33
@jottofar
Copy link
Contributor Author

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jan 26, 2021
@openshift-ci-robot
Copy link
Contributor

@jottofar: This pull request references Bugzilla bug 1884334, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jan 26, 2021
@jottofar jottofar force-pushed the bug-1884334 branch 3 times, most recently from 8ced187 to c9a55d8 Compare January 27, 2021 16:50
@jottofar
Copy link
Contributor Author

/test e2e-agnostic-upgrade

@jottofar
Copy link
Contributor Author

/test unit

@jottofar
Copy link
Contributor Author

/retest

wking
wking reviewed Jan 29, 2021
View reviewed changes
pkg/cvo/status.go Outdated Show resolved Hide resolved
wking
wking reviewed Feb 3, 2021
View reviewed changes
pkg/cvo/internal/operatorstatus.go
Name: actual.Name,
Nested: nestedMessage,
UpdateEffect: payload.UpdateEffectNone,
Reason: "ClusterOperatorNotAvailable",
Copy link
Member

@wking wking Feb 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want a different reason for this, because we know by this point that it is available and not degraded.
But we can twiddle that in follow-up work.

wking
wking reviewed Feb 3, 2021
View reviewed changes
pkg/payload/task.go
type UpdateEffectType string

const (
// "None" defines an error as having no affect on the update state.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We should add Go linter. I expect it would want these godocs to start with the exposed name, so:

// UpdateEffectNone defines an error as having no affect on the update state.

and similar for the other constants below.

@wking
Copy link
Member

wking commented Feb 3, 2021

/lgtm
/hold

Holding in case you want to address either of the two new nits. I'm fine punting them both to follow up work as well, in which case, just pull the hold.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2021
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 3, 2021
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jottofar, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jottofar
Copy link
Contributor Author

jottofar commented Feb 3, 2021

/unhold

Let's address them in a follow-up so this gets some soak time before code cutoff.

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2021
@jottofar
Copy link
Contributor Author

jottofar commented Feb 3, 2021

/test unit

@openshift-merge-robot openshift-merge-robot merged commit 9f3368d into openshift:master Feb 3, 2021
@openshift-ci-robot
Copy link
Contributor

@jottofar: All pull requests linked via external trackers have merged:

Bugzilla bug 1884334 has been moved to the MODIFIED state.

In response to this:

Bug 1884334: UpdateError: enhance for ability to determine when upgrade failing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wking wking wking left review comments

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@vrutkovs vrutkovs Awaiting requested review from vrutkovs

Assignees

@wking wking

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jottofar @openshift-ci-robot @openshift-merge-robot @sdodson @wking