-
Notifications
You must be signed in to change notification settings - Fork 690
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DAMN ! worker 7 (pid: 1343) died, killed by signal 11 :( trying respawn ... #1792
Comments
Hey @jedie - i get the same error. I am building from python:3.6-alpine as well. My ENV and CMD in Dockerfile looks like this..
I am a bit worried of using this in a PRODUCTION environment. Rob |
I switched to 3.6-slim-stretch as a "work-a-round"... |
Also getting this all of a sudden too.
|
I found that switching the uwsgi config to only one thread makes this go away. Here is my uwsgi config (from Dockerfile)..
|
I can confirm that configuring it to use one thread and this goes away. |
I'm also seeing this on |
|
I ran into a similar issue, though for me segfaults were traced down to anything that tried to use ssl (e.g. to talk to a remote API). Changing to stretch-slim seemed to resolve the issue. |
Just wanted to note I ran across this issue with |
I'm still getting this using |
Still having this problem, is there a way to make uwsgi exit if this happens, I have my service configured to restart on fail. Better than being in an inconsistent state, running but not alive I'm using
|
Can confirm that switching to Alpine 3.9 fixed that problem for me. I had the same symptoms, completely out of the blue. One of the most significant changes in 3.9 is the return to OpenSSL (from LibreSSL), I can imagine how changing such a foundational library could make a difference. It's also completely possible that there is a looming bug somewhere in my software that is no longer triggered due to the different underlying libraries. |
I also meet this problem
|
I also meet this problem. (But strangely, in alpine 3.9) |
Getting the same with Seems to get fixed by increasing uwsgi's |
In my case, I turn off the option "enable-threads".
|
Any update on this issue? I have also ran in to the same issue with |
Same issue with |
I think this error is due to uswgi config. For my Django projects, I have been using Docker (based on python:3.7-alpine) in production with no issues. Below are my Dockerfile, docker-entrypoint.sh and uswgi.ini files - which were borrowed and inspired by other online articles and research. Hope this helps other folks. Dockerfile:
docker-entrypoint.sh #!/bin/sh
while ! mysqladmin ping -h"$MYSQL_HOST" --silent; do
echo "database is unavailable - sleeping for 2 secs"
sleep 2
done
if [ "x$DJANGO_MANAGEPY_MIGRATE" = 'xon' ]; then
echo 'attempting to run "migrate" ..'
python manage.py migrate --noinput
else
echo 'DJANGO_MANAGEPY_MIGRATE is not "on", skipping'
fi
echo "copying mime.types to /etc dir .."
cp mime.types /etc/mime.types
echo "starting uwsgi.."
uwsgi uwsgi.ini uwsgi.ini [uwsgi]
strict = true
master = true
enable-threads = true
vacuum = true ; Delete sockets during shutdown
single-interpreter = true
die-on-term = true ; Shutdown when receiving SIGTERM (default is respawn)
need-app = true
disable-logging = true ; Disable built-in logging
log-4xx = true ; but log 4xx's anyway
log-5xx = true ; and 5xx's
harakiri = 120 ; forcefully kill workers after XX seconds
; py-callos-afterfork = true ; allow workers to trap signals
max-requests = 1000 ; Restart workers after this many requests
max-worker-lifetime = 3600 ; Restart workers after this many seconds
reload-on-rss = 2048 ; Restart workers after this much resident memory
worker-reload-mercy = 60 ; How long to wait before forcefully killing workers
cheaper-algo = busyness
processes = 64 ; Maximum number of workers allowed
cheaper = 8 ; Minimum number of workers allowed
cheaper-initial = 16 ; Workers created at startup
cheaper-overload = 1 ; Length of a cycle in seconds
cheaper-step = 8 ; How many workers to spawn at a time
cheaper-busyness-multiplier = 30 ; How many cycles to wait before killing workers
cheaper-busyness-min = 20 ; Below this threshold, kill workers (if stable for multiplier cycles)
cheaper-busyness-max = 70 ; Above this threshold, spawn new workers
cheaper-busyness-backlog-alert = 16 ; Spawn emergency workers if more than this many requests are waiting in the queue
cheaper-busyness-backlog-step = 2 ; How many emergency workers to create if there are too many requests in the queue
wsgi-file = /code/_base/wsgi.py
http = :8000
static-map = /static/=/code/static/
uid = 1000
gid = 2000
touch-reload = /code/reload-uwsgi |
Same problem here, using I noticed that Python 3.7 is not among the officially supported ones, so I downgraded to Python 3.5 but the error manifests nonetheless. |
@jacopofar I too am getting the same error on kubernetes but not when I run locally. My image is based on https://github.com/dockerfiles/django-uwsgi-nginx |
@jacopofar , @asherp, @aliashkar - any chance there is a stacktrace in the logs before the "killed by signal 11" line and could paste it here? It would also be very helpful if you could reveal some information about your apps: Are you by any chance using It appears there's a known issue with wheels that include their own libssl (or other libs) - see #1569 and #1590 (also this: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/) |
@awelzel I tried to reproduce but cannot get it anymore ¯_(ツ)_/¯ I don't remember any additional stacktrace, it only printed that message. This is my requirements.txt for that version:
I'm not aware of any embedded libssl except for psycopg2, sorry for not being able to provide more details :/ |
It also apparently solved my use case. Is there a way to track uwsgi stack memory consumption to be sure that it happens for out of memory reason ? |
the same error occurred when i try to run a job with frequet http requests. |
When running in staging without this set we would see workers randomly die causing 502s `DAMN ! worker 5 (pid: 429) died, killed by signal 11 :( trying respawn ...` unbit/uwsgi#1792 (comment)
I met this issue at the same env |
I almost tried every single thing explained here. Still exactly same error occurred due to uwsgi server. This is specifically for a particular flask endpoint whenever I deployed to k83 cluster and worked perfectly in dev machine.
|
I have also encountered this problem. my uwsgi version is 2.0.18 . and threads per worker sets to 6 I wonder if there is a more graceful way to handle this, for example, in simple_goodbye_cruel_world, set manage_next_request to zero before wait_for_threads ,thus stop receive requests.then in uwsgi_master_check_mercy wait for the threads to end befor killing it with signal 9. if the worker really stucsk , it can be killed by harakiri |
So far, upping |
I seem to have a similar issue. alpine 3.17.5, uwsgi 2.0.22 and python 3.10.13 compiling uwsgi on alpine and copying it in my app container.
Everything works locally on macOS/arm64. It fails in our linux/amd64 kubernetes cluster. |
I am also facing similar issue with alpine 3.17.5, uwsgi 2.0.22 and python 3.10.13. But my application works fine with lower verion python 3.9 alpine3.15. Tried all the above suggestions but no luck |
Is this issue resolved ? Any work around on this ? |
When running in staging without this set we would see workers randomly die causing 502s `DAMN ! worker 5 (pid: 429) died, killed by signal 11 :( trying respawn ...` unbit/uwsgi#1792 (comment)
nhsx/nhsx-website@eb91494 is a solution?!? |
I'm using docker... After i switched from https://github.com/phusion/baseimage-docker (
phusion/baseimage:0.10.1
with Python v3.5) to https://hub.docker.com/_/python/ (python:3.6-alpine
with Python v3.6)After this i get very often the error:
DAMN ! worker X (pid: Y) died, killed by signal 11 :( trying respawn ...
The rest of the setup is the same and used
uWSGI==2.0.17
Any idea?!?
The text was updated successfully, but these errors were encountered: