Gunicorn receiving HUP signal and restarting

[talhachattha]

I have been running a gunicorn server with 4 workers and 40 threads, I receive a request to start a job for which I use pebble and its ProcessPool for work to remain asynchronous.
After running for few hours gunicorn automatically receives HUP signal and restarts causing those processes to stop and thus job failure.
Following is the output of what I receive.

[2018-06-28 09:50:33 -0700] [8266] [INFO] Starting gunicorn 19.8.1
[2018-06-28 09:50:33 -0700] [8266] [INFO] Listening at: http://0.0.0.0:7788 (8266)
[2018-06-28 09:50:33 -0700] [8266] [INFO] Using worker: threads
[2018-06-28 09:50:33 -0700] [8269] [INFO] Booting worker with pid: 8269
[2018-06-28 09:50:33 -0700] [8270] [INFO] Booting worker with pid: 8270
[2018-06-28 09:50:33 -0700] [8271] [INFO] Booting worker with pid: 8271
100%|██████████| 70/70 [03:55<00:00, 3.36s/it]
Run Completed, Saving the result
39%|███▉ | 4042/10240 [15:27<20:52, 4.95it/s]
[2018-06-28 10:27:47 -0700] [8266] [INFO] Handling signal: hup
[2018-06-28 10:27:47 -0700] [8266] [INFO] Hang up: Master
[2018-06-28 10:27:47 -0700] [22456] [INFO] Booting worker with pid: 22456
[2018-06-28 10:27:47 -0700] [22457] [INFO] Booting worker with pid: 22457
[2018-06-28 10:27:47 -0700] [22458] [INFO] Booting worker with pid: 22458

[tilgovi]

Reloading is the expected response to HUP. The reload should be graceful. If it's not graceful, that is a bug with the threaded worker (related: #1236). If you don't want gunicorn to reload, do not send HUP.

[talhachattha]

I am not sending HUP, OS sends it or somehow it automatically receives it.

[tilgovi]

You will have to look into the process pool to figure out why Gunicorn receives HUP. Maybe your application receives HUP and the process pool sends it to all children, or maybe the process pool sends HUP for some other reason.

Is it causing errors, though? Do clients get disconnected in the middle of requests, or is it at least a graceful reload? If it is not graceful, there may a be a bug.

[talhachattha]

I am unable to identify the source of HUP signal. But somehow gunicorn receives it from somewhere and gets restarted. I am unable to find a way for graceful closure. It just kills anything that is running and simply restarts.

[tilgovi]

When you say "kills anything that is running" do you mean requests / connections or some background thread you have started? Are they long requests (websocket, long poll)? It should close connections gracefully, but there is a timeout after which it kills them hard.

[benoitc]

what are these hidden line though? Seems like the master is receiving an HUP signal right after ....

[talhachattha]

@benoitc what hidden lines are you talking about?

[talhachattha]

@tilgovi It simply restarts everything, I do run some background processes well. They also get killed and server as a whole restarts.

[benoitc]

@talhachattha

what is this ^^ ?

[talhachattha]

@benoitc This is the progress of a job that I run in background in form of different processes.

[benoitc]

could that job be responsible of the HUP signal? What if you don't run it?

[talhachattha]

The job is my product.
System is nothing without it.
I have to run it without getting this HUP signal

[benoitc]

well , something in your system is sending that HUP signal. You have to check if it’s not this job.

maybe you can also check the debug log or spew lov to see at which point it os happening

[talhachattha]

What is spew lov?

[benoitc]

@talhachattha pups, it's the --spew function to trace all calls:

http://docs.gunicorn.org/en/stable/settings.html#spew

[benoitc]

closing issue, no more news since.