Skip to content

Commit

Permalink
mark server unhealthy as soon as shutdown starts
Browse files Browse the repository at this point in the history
I am not convinced that this is more right than doing it the other way.  Some recent k8s change convinced me this is right, but feedback is welcome here.
  • Loading branch information
jrockway committed Sep 23, 2021
1 parent 97ea3e8 commit 731bb67
Showing 1 changed file with 7 additions and 6 deletions.
13 changes: 7 additions & 6 deletions server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -569,23 +569,24 @@ func listenAndServe(stopCh chan string) error {
servers--
zap.L().Error("server unexpectedly exited", zap.Error(doneErr), zap.Int("servers_remaining", servers))
}

zap.L().Info("marking server unhealthy")
healthServer.Shutdown() // nolint
if t := listenOpts.PreDrainGracePeriod; t > 0 {
// The idea here is that when we are sent the termination signal, our service
// controller also removes you from its list of healthy backends, and so we stop
// receiving new requests. However, this does not always happen instantly. This
// delay allows us to serve normal traffic (including health checks claiming we're
// healthy) for a period of time after we have been removed. Eventually traffic
// volume will trail off as service discovery stops being able to discover us. Then
// we stop accepting connections entirely and allow existing connections to finish
// (the graceful drain), and finally shut down.
// delay allows us to serve normal traffic for a period of time after we have been
// removed. Eventually traffic volume will trail off as service discovery stops
// being able to discover us. Then we stop accepting connections entirely and allow
// existing connections to finish (the graceful drain), and finally shut down.
zap.L().Info("sleeping before drain to wait for network reprogramming", zap.Duration("duration", t))
time.Sleep(t)
}

zap.L().Info("beginning graceful drain", zap.Duration("duration", listenOpts.ShutdownGracePeriod))
tctx, c := context.WithTimeout(context.Background(), listenOpts.ShutdownGracePeriod)
defer c()
healthServer.Shutdown() // nolint
for _, h := range drainHandlers {
h()
}
Expand Down

0 comments on commit 731bb67

Please sign in to comment.