Skip to content

Commit

Permalink
Terminate process correctly from reflector thread
Browse files Browse the repository at this point in the history
Our reflectors run on their own thread, and sys.exit() from
a thread does not actually exit the process - just the thread.
When the reflector has to shut down because it can not talk
to the k8s master anymore for a while, we had just been stopping
the reflector thread and doing nothing else.

This will instead actually cause the hub to stop, and in z2jh
restart - which seems the right thing to do.

Ref 2i2c-org/infrastructure#680,
where we encountered this during a period of time when the
k8s master was unreachable for about 1 minute.
  • Loading branch information
yuvipanda committed Sep 14, 2021
1 parent ebebccb commit c15ad86
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion kubespawner/spawner.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import os
import string
import sys
import signal
import warnings
from asyncio import sleep
from concurrent.futures import ThreadPoolExecutor
Expand Down Expand Up @@ -1927,7 +1928,10 @@ def on_reflector_failure():
"%s reflector failed, halting Hub.",
key.title(),
)
sys.exit(1)
# This won't be called from the main thread, so sys.exit
# will only kill current thread - not process.
# https://stackoverflow.com/a/7099229
os.kill(os.getpid(), signal.SIGINT)

previous_reflector = self.__class__.reflectors.get(key)

Expand Down

0 comments on commit c15ad86

Please sign in to comment.