Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-35343: make shutdown-delay-duration configurable #1685

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

OCPBUGS-35343: make shutdown-delay-duration configurable #1685

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion pkg/cmd/render/render.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"net"
"os"
"path/filepath"
"time"

"github.com/ghodss/yaml"
configv1 "github.com/openshift/api/config/v1"
Expand Down Expand Up @@ -47,6 +48,9 @@ type renderOpts struct {
infraConfigFile string

groupVersionsByFeatureGate map[configv1.FeatureGateName][]schema.GroupVersion

// so we can override bootstrap kube-apiserver argument 'shutdown-delay-duration'
shutdownDelayDuration time.Duration
}

// NewRenderCommand creates a render command.
Expand All @@ -62,6 +66,11 @@ func newRenderCommand(testOverrides ...func(*renderOpts)) *cobra.Command {
lockHostPath: "/var/run/kubernetes/lock",
etcdServerURLs: []string{"https://127.0.0.1:2379"},
etcdServingCA: "root-ca.crt",

// by default, we are giving the load balancer 20s to remove the
// bootstrap kube-apiserver from its pool after TERM signal is
// sent to the kube-apiserver on the bootstrap node.
shutdownDelayDuration: 20 * time.Second,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the default value for shutdownDelayDuration ?
It looks like it is 0, does it mean that the bootstrap api server didn't give any time to LBs to react ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc its ) for SNO and 129 for AWS:

cluster-kube-apiserver-operator/pkg/operator/configobservation/apiserver/observe_termination_duration.go

Lines 37 to 46 in 37df1b1

case infra.Status.ControlPlaneTopology == configv1.SingleReplicaTopologyMode:
// reduce the shutdown delay to 0 to reach the maximum downtime for SNO
observedShutdownDelayDuration = "0s"
case infra.Spec.PlatformSpec.Type == configv1.AWSPlatformType:
// AWS has a known issue: https://bugzilla.redhat.com/show_bug.cgi?id=1943804
// We need to extend the shutdown-delay-duration so that an NLB has a chance to notice and remove unhealthy instance.
// Once the mentioned issue is resolved this code must be removed and default values applied
//
// Note this is the official number we got from AWS
observedShutdownDelayDuration = "129s"

}
for _, f := range testOverrides {
f(&renderOpts)
Expand Down Expand Up @@ -97,6 +106,7 @@ func (r *renderOpts) AddFlags(fs *pflag.FlagSet) {
fs.StringVar(&r.clusterConfigFile, "cluster-config-file", r.clusterConfigFile, "Openshift Cluster API Config file.")
fs.StringVar(&r.clusterAuthFile, "cluster-auth-file", r.clusterAuthFile, "Openshift Cluster Authentication API Config file.")
fs.StringVar(&r.infraConfigFile, "infra-config-file", "", "File containing infrastructure.config.openshift.io manifest.")
fs.DurationVar(&r.shutdownDelayDuration, "shutdown-delay-duration", r.shutdownDelayDuration, "shutdown-delay-duration argument for the bootstrap kube-apiserver.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not to add it directly to the bootstrap-api-server's manifest ? - https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/bindata/bootkube/bootstrap-manifests/kube-apiserver-pod.yaml#L46

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bootkube already has an override -

cluster-kube-apiserver-operator/bindata/bootkube/config/bootstrap-config-overrides.yaml

Lines 79 to 81 in 04eba64

{{- if .ShutdownDelayDuration}}
shutdown-delay-duration:
- {{ .ShutdownDelayDuration }}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, cool!
should we also set shutdown-send-retry-after ?

}

// Validate verifies the inputs.
Expand Down Expand Up @@ -126,6 +136,10 @@ func (r *renderOpts) Validate() error {
return err
}

if r.shutdownDelayDuration <= 0 {
return errors.New("--shutdown-delay-duration must be a positive time duration")
}

return nil
}

Expand Down Expand Up @@ -192,7 +206,7 @@ func (r *renderOpts) Run() error {
BindAddress: "0.0.0.0:6443",
BindNetwork: "tcp4",
TerminationGracePeriodSeconds: 135, // bit more than 70s (minimal termination period) + 60s (apiserver graceful termination)
ShutdownDelayDuration: "", // do not override
ShutdownDelayDuration: r.shutdownDelayDuration.String(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW: Do we want to expose it as a command line arg ? If yes then we have to coordinate updating it with terminationGracePeriodSeconds. Otherwise kubelet might terminate kas process before graceful termination completes.

}

featureGateAccessor, err := r.generic.FeatureGates()
Expand Down