Supervisor 3.3.1: CRIT uncaptured python exception, closing channel

[madmuffin1]

We're still experiencing the problem that supervisor logs a CRIT uncaptured on every log message produced when directing logs to /dev/stdout, similar to this:

Test
2017-05-17 12:20:27,860 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140178721855392 for <Subprocess at 140178721855248 with name cause-error in state STARTING> (stdout)> (<type 'exceptions.IOError'>:[Errno 29] Illegal seek [/usr/local/lib/python2.7/dist-packages/supervisor/supervisord.py|runforever|227] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|handle_read_event|232] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|record_output|166] [/usr/local/lib/python2.7/dist-packages/supervisor/dispatchers.py|_log|142] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|info|275] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|log|293] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|emit|186] [/usr/local/lib/python2.7/dist-packages/supervisor/loggers.py|doRollover|211])

This looks similar to #638, but is still present with 3.3.1 and has a different stack trace.

I have created a minimal Dockerfile that will reproducably cause the problem on my side: https://gist.github.com/madmuffin1/33408001a402c2fd98bc0b227185b363

If you need any more details, let me know, I am happy to help.

[madmuffin1]

From further studying the documentation, I found that there is redirect_stdout and redirect_stderr, which works as expected. So this problem should problably be prevented by generating a more helpful error message.

[mnaberez]

The log message above shows that an illegal seek error occurred during log rotation.

Your config file is this:

RUN apt-get update && apt-get install -y -q locales python-pip && pip install supervisor &&
echo "[supervisord]\nnodaemon=true\n\n[program:cause-error]\ncommand=/bin/bash -c 'echo Test'\nstdout_logfile=/dev/stdout" > /etc/supervisord.conf

That config uses log rotation (it's on by default), but you've specified a log file that is not seekable:

stdout_logfile=/dev/stdout

If you really want to use stdout, it may work, but minimally you'll need to disable log rotation (set stdout_maxbytes=0 (Configuration).

From further studying the documentation, I found that there is redirect_stdout and redirect_stderr, which works as expected.

There is no redirect_stdout option, and redirect_stderr will not affect this. The redirect_stderr option will cause supervisord to log stderr to the same file as stdout. You will still see the error if you don't either change the log file to a seekable file or disable rotation.

[madmuffin1]

A late thanks to @mnaberez!

If anyone stumbles upon this error like me, be aware that the mentioned options apply to the [supervisord] section, within [program:x] sections, you need to use stdout_logfile_maxbytes = 0 and/or stderr_logfile_maxbytes = 0 respectively.

[RodriguezLucha]

I did something like this:

stdout_logfile=/dev/stdout
stderr_logfile=/dev/stderr
stdout_maxbytes=0
stderr_maxbytes=0
stdout_logfile_maxbytes = 0
stderr_logfile_maxbytes = 0