Drain controller #488
Closed
Drain controller #488
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
@SchSeba this PR doesn't change user-facing API. The goal of this PR is to move drain logic to controller and use SriovNetworkNodeState to store drain status. it's still does drain only on one node at the same time. Once we agree on API changes proposed in #479 next PR will add changes to sriovPoolConfig and implement multiple draining nodes |
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
9059724 to
5b289c5
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
5b289c5 to
13aa522
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
13aa522 to
35163ae
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
35163ae to
429ddaf
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
|
/test-e2e-all |
SchSeba
reviewed
Sep 11, 2023
controllers/drain_controller.go
Outdated
| } | ||
|
|
||
| func (dr *DrainReconciler) drainNode(node *corev1.Node) error { | ||
| glog.Info("drainNode(): Update prepared") |
There was a problem hiding this comment.
please don't combine different logger
controllers/migration_controller.go
Outdated
| return reconcile.Result{}, err | ||
| } | ||
| nodeState.Status.DrainStatus = anno | ||
| mr.ClientSet.SriovnetworkV1().SriovNetworkNodeStates(namespace).UpdateStatus(context.TODO(), nodeState, metav1.UpdateOptions{}) |
There was a problem hiding this comment.
we need to be careful here..
because today there is a clear split between the daemon and the operator.
meaning the operator updates ONLY the spec and the daemon updates the status
e0ne
force-pushed
the
drain-controller
branch
from
429ddaf to
f7645ef
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
f7645ef to
b5174ec
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
b5174ec to
d666eff
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
|
@e0ne can you please rebase this PR? I'm interested in running e2e test lane on this |
zeeke
reviewed
Oct 20, 2023
main.go
Outdated
| glog.Info(fmt.Sprintf("%s pod from Node %s/%s", verbStr, pod.Namespace, pod.Name)) | ||
| }, | ||
| //Out: writer{glog.Info}, | ||
| //ErrOut: writer{glog.Error}, |
There was a problem hiding this comment.
Got a panic while running conformance tests:
E1020 10:20:44.423046 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 237 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2a473e0?, 0x40d9d30})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000544900?})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x2a473e0, 0x40d9d30})
/usr/lib/golang/src/runtime/panic.go:884 +0x213
fmt.Fprintf({0x0, 0x0}, {0x2d42314, 0xc}, {0xc000766898, 0x1, 0x1})
/usr/lib/golang/src/fmt/print.go:225 +0x7a
k8s.io/kubectl/pkg/drain.RunNodeDrain(0xc0008c2dd0, {0xc00067cdb0?, 0x0?})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/k8s.io/kubectl/pkg/drain/default.go:40 +0xd4
github.com/k8snetworkplumbingwg/sriov-network-operator/controllers.(*DrainReconciler).drainNode.func1()
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/controllers/drain_controller.go:168 +0x156
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x0?)
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145 +0x4d
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x2540be400, 0x4000000000000000, 0x0, 0x5, 0x0}, 0x0?)
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:461 +0x5f
github.com/k8snetworkplumbingwg/sriov-network-operator/controllers.(*DrainReconciler).drainNode(0xc000275b00, {0x308f540?, 0xc000513560?}, 0xc000428dc0)
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/controllers/drain_controller.go:161 +0x31d
github.com/k8snetworkplumbingwg/sriov-network-operator/controllers.(*DrainReconciler).Reconcile(0xc000275b00, {0x308f540, 0xc000513560}, {{{0xc0000000fa, 0x20}, {0x2d606eb, 0x1c}}})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/controllers/drain_controller.go:86 +0x51b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x308f540?, {0x308f540?, 0xc000513560?}, {{{0xc0000000fa?, 0x292db00?}, {0x2d606eb?, 0x3079ed8?}}})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00039a960, {0x308f498, 0xc0006929b0}, {0x2b27140?, 0xc0005198a0?})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314 +0x377
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00039a960, {0x308f498, 0xc0006929b0})
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x587
2023-10-20T10:20:44.423098134Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "sriovoperatorconfig", "controllerGroup": "sriovnetwork.openshift.io", "controllerKind": "SriovOperatorConfig", "SriovOperatorConfig": {"name":"drain-upgrade-reconcile-name","namespace":"openshift-sriov-network-operator"}, "namespace": "openshift-sriov-network-operator", "name": "drain-upgrade-reconcile-name", "reconcileID": "a469baca-6d9c-46fc-a2ff-1d81e65d4402"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1048b3a]
Merged
zeeke
reviewed
Oct 24, 2023
controllers/helper.go
Outdated
| return false | ||
| } | ||
|
|
||
| return oldAnno == newAnno |
There was a problem hiding this comment.
From the doc:
// Update returns true if the Update event should be processed
So I'd change it to oldAnno != newAnno, as the event should be processed if a value change has happened. Right?
bn222
reviewed
Oct 24, 2023
| // sort nodeList to iterate in the same order each reconcile loop | ||
| sort.Slice(nodeList.Items, func(i, j int) bool { | ||
| return strings.Compare(nodeList.Items[i].Name, nodeList.Items[j].Name) == -1 | ||
| }) |
There was a problem hiding this comment.
we sort slice of objects here
e0ne
force-pushed
the
drain-controller
branch
from
d666eff to
20a617f
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
20a617f to
a2f0a5c
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
e0ne
force-pushed
the
drain-controller
branch
from
a2f0a5c to
651df2e
Compare
|
Thanks for your PR,
To skip the vendors CIs use one of:
|
Pull Request Test Coverage Report for Build 6768257558
💛 - Coveralls |
Closed
|
closing this one |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.