CUPS segfault when adding many ipp-printers fast #466
Comments
|
I can give you a little more information about my environment: The 250 printers I try to connect via IPP are a mix of different models:
I'd have some OKI and Kyocera printers as well. But for the sake of debugging, I excluded them from IPP for the moment. In the meantime, I also noticed, that the container does not only crash when adding this many printers at the start. It also crashes with the "2 Seconds" workaround, it just does so later. At this point I'm quite at a loss. I don't know cups cannot handle this many IPP connections or if I made a mistake while setting cups up with IPP. |
|
@SwissGipfel There are no limitations on the number of queues built into CUPS. But since every queue you are adding is using the IPP Everywhere model you might be running into a thread or stack limit. What we need is a debug CUPS error_log ("cupsctl --debug-logging" before you start adding printers) as well as a backtrace from the core dump. |
zdohnal
added
the
waiting for reporter
|
Thank you for your response. logs-level-debug.zip In my last two tries, i switched my base image to https://github.com/olbat/dockerfiles/blob/master/cupsd/Dockerfile based on I'm still figuring out how I get the core dump done on my local development environment. As soon as I got it, I'll post it here. |
|
I'm not sure if these are the correct files you were looking for. Here is the core dump together with a full backtrace (generated with and the debug log file of the same run For this run, i used the following Dockerfile FROM ubuntu:22.04
# Install basic tools, cups and a minimal set of drivers
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update >/dev/null && apt-get install -y lsb-release && apt install ubuntu-dbgsym-keyring
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN apt-get update >/dev/null \
&& apt-get install -y \
sudo \
whois \
gdb \
libcap2-bin=1:2.44-1build3 >/dev/null \
cups=2.4.1op1-1ubuntu4.1 \
cups-client=2.4.1op1-1ubuntu4.1 \
cups-bsd=2.4.1op1-1ubuntu4.1 \
cups-filters=1.28.15-0ubuntu1 \
cups-dbgsym \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
EXPOSE 631
# Add user and disable sudo password checking
RUN useradd \
--groups=sudo,lp,lpadmin \
--create-home \
--home-dir=/home/print \
--shell=/bin/bash \
--password=$(mkpasswd print) \
print
# A user with this UID also exists in the cups-configurator container. Adding it here to the lpadmin group is required for authorization to cups
RUN adduser --system -uid 1005 --shell /bin/bash --home /home/networkmaster --disabled-password networkmaster && usermod networkmaster -aG lpadmin
# Change owner of some directories to run as non-root
RUN chown -R print:lp /var/log/cups \
&& chown -R print:lp /var/spool/cups \
&& chown -R print:lp /var/cache/cups \
&& chown -R print:lp /run/ \
&& chown -R print:lp /etc/cups/ \
&& setcap 'cap_net_bind_service=+ep' /usr/sbin/cupsd
COPY cupsd.conf /etc/cups/cupsd.conf
COPY cups-files.conf /etc/cups/cups-files.conf
#USER print
CMD ["/usr/sbin/cupsd", "-f"] |
|
@SwissGipfel thank you for the files! Unfortunately the debuginfo packages were missing, so the backtrace is too generic and I don't see where the crash happened either. Can you get debuginfo packages (at least for glibc, cupsd and libcups) into the container, reproduce the issue and get a new backtrace? But it probably will be some internal thread limit we hit - the last thread in the backtrace doesn't seem to go into libcups at all (unless there is some libcups function under |
|
Here is the backtrace with the additional debuginfo packages and the corresponding logs I used this modified Dockerfile to generate this batch: FROM ubuntu:22.04
# Install basic tools, cups and a minimal set of drivers
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update >/dev/null && apt-get install -y lsb-release && apt install ubuntu-dbgsym-keyring
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN apt-get update >/dev/null
RUN apt-get install --allow-unauthenticated -y libglib2.0-0-dbgsym=2.72.1-1 libglib2.0-0=2.72.1-1 \
&& apt-get install -y \
sudo \
whois \
gdb \
libcap2-bin=1:2.44-1build3 >/dev/null \
cups=2.4.1op1-1ubuntu4.1 \
cups-client=2.4.1op1-1ubuntu4.1 \
cups-bsd=2.4.1op1-1ubuntu4.1 \
cups-filters=1.28.15-0ubuntu1 \
cups-dbgsym \
libcups2-dbgsym \
libcupsfilters1-dbgsym \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
EXPOSE 631
# Add user and disable sudo password checking
RUN useradd \
--groups=sudo,lp,lpadmin \
--create-home \
--home-dir=/home/print \
--shell=/bin/bash \
--password=$(mkpasswd print) \
print
# A user with this UID also exists in the cups-configurator container. Adding it here to the lpadmin group is required for authorization to cups
RUN adduser --system -uid 1005 --shell /bin/bash --home /home/networkmaster --disabled-password networkmaster && usermod networkmaster -aG lpadmin
# Change owner of some directories to run as non-root
RUN chown -R print:lp /var/log/cups \
&& chown -R print:lp /var/spool/cups \
&& chown -R print:lp /var/cache/cups \
&& chown -R print:lp /run/ \
&& chown -R print:lp /etc/cups/ \
&& setcap 'cap_net_bind_service=+ep' /usr/sbin/cupsd
COPY cupsd.conf /etc/cups/cupsd.conf
COPY cups-files.conf /etc/cups/cups-files.conf
#USER print
CMD ["/usr/sbin/cupsd", "-f"]Comparing the two gdb logs there are less Thanks |
zdohnal
added
bug
|
@SwissGipfel thank you for the data! In the end I've brought up the container and found out that Ubuntu packages CUPS with more subpackages, so the missing symbols were from cups-daemon-dgbsym. The sigsegv happens in Can you run cupsd within valgrind in the container? This seems like a memory problem. The command looks like: |
|
I had some issues with valgrind in my dev environment, but finally it produced some logs: However, I had to remove the FROM ubuntu:22.04
# Install basic tools, cups and a minimal set of drivers
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y \
lsb-release \
&& apt-get install -y ubuntu-dbgsym-keyring
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" >> /etc/apt/sources.list.d/ddebs.list
RUN apt-get update \
&& apt-get install -y \
sudo \
whois \
gdb \
libcap2-bin=1:2.44-1build3 >/dev/null \
cups=2.4.1op1-1ubuntu4.1 \
cups-client=2.4.1op1-1ubuntu4.1 \
cups-bsd=2.4.1op1-1ubuntu4.1 \
cups-filters=1.28.15-0ubuntu1 \
cups-dbgsym \
cups-daemon-dbgsym \
libcups2-dbgsym \
libcupsfilters1-dbgsym \
libglib2.0-0=2.72.1-1 \
libglib2.0-0-dbgsym=2.72.1-1 \
valgrind \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
EXPOSE 631
# Add user and disable sudo password checking
RUN useradd \
--groups=sudo,lp,lpadmin \
--create-home \
--home-dir=/home/print \
--shell=/bin/bash \
--password=$(mkpasswd print) \
print
# A user with this UID also exists in the cups-configurator container. Adding it here to the lpadmin group is required for authorization to cups
RUN adduser --system -uid 1005 --shell /bin/bash --home /home/networkmaster --disabled-password networkmaster && usermod networkmaster -aG lpadmin
# Change owner of some directories to run as non-root
RUN chown -R print:lp /var/log/cups \
&& chown -R print:lp /var/spool/cups \
&& chown -R print:lp /var/cache/cups \
&& chown -R print:lp /run/ \
&& chown -R print:lp /etc/cups/
# && setcap 'cap_net_bind_service=+ep' /usr/sbin/cupsd
COPY cupsd.conf /etc/cups/cupsd.conf
COPY cups-files.conf /etc/cups/cups-files.conf
#USER print
CMD ["/usr/bin/valgrind","-v","--leak-check=full","--track-origins=yes","--log-file=/var/log/cups/valgrind.txt","/usr/sbin/cupsd","-f"] |
|
Let me know if the image needs some additional debug packages to analyze this problem further. I saw some lines in the valgrind log with the warning |
zdohnal
removed
the
waiting for reporter
|
@SwissGipfel Great! The log helps a lot - now I can see that we don't protect |
scheduler: Protect MimeDatabase from parallel access (fixes #466)
scheduler/ipp.c: Fix memory leaks found during #466
I'm running cups in a Kubernetes cluster with a with a sidecar that adds and removes printers based on a database.
This worked very well until I've started to add printers via ipp.
The sidecar adds printer roughly:
lpadmin -p <printer-name> -E -v socket://<printer-ip>:9100lpadmin -p <printer-name> -E -v ipp://<printer-ip> -m everywhereThese printers will be added at the start of my cups container and the container crashes with roughly 500 printers added - each time with a segfault like this one:
I can reproduce the same behavior, if I only add my ~250 ipp printers - again with a segfault like:
My current workaround is to wait 2 seconds each time i generate the command
lpadmin -p <printer-name> -E -v ipp://<printer-ip> -m everywhere. With this sleep, cups doesn't crash consistently but the container also needs about 8.5 minutes to start up.I'm using cups with an Ubuntu:22.04 base image. The whole container looks like this:
(I also tried it without the version pinning, with the same result)
The whole cups container has a limit of 500Mi memory and 500m CPU - but it never peaks as far as I can see.
Is this a bug or an expected behavior?
Do you have an idea how I can debug this further?
The text was updated successfully, but these errors were encountered: