podman images very slow #6288
Comments
openshift-ci-robot
added
the
kind/bug
|
Checked |
|
We do have some algorithmic parsing in |
|
@vrothberg Any thoughts here? |
|
Unfortunately, I have nothing at hand. Some users made similar observations over time but such slow downs are seemingly random in the sense that we don't have a reproducer. |
|
Yeah, I pretty much regretted not taking a snapshot as soon as I had started the prune :( Should never destroy a repro... |
|
Okay, made a run between 127 images (what was left after the prune) up to 450. I used this script: while podman build -t bloat-test . >/dev/null; do
touch config/empty.yaml
/usr/bin/time -f %e/%U/%S podman images | wc -l
doneWith this Dockerfile: FROM python:alpine
ARG KUBECTL_VERSION=1.16.5
ARG HELM_VERSION=3.2.1
ARG KUBEVAL_VERSION=0.15.0
ARG CONFTEST_VERSION=0.18.2
ARG YAMLLINT_VERSION=1.23.0
WORKDIR /work
RUN apk add bash coreutils && \
a long string of wget | tar:s
COPY config /config
COPY lint.sh /
ENTRYPOINT ["/lint.sh"]I can't share the actual files we copy over, but it is ~50k of plain text in /config. The layers above are ~250M, but the only layers which are rebuilt are the bottom three, due to touching a file in /config. From first to last measurement, wall time to run Would probably be a good idea to reproduce this with a simpler image, and make a longer run. This took ~1h40m to run on my laptop, so kicking it off from a clean slate on some machine somewhere and letting it run for longer might show a clearer trend. Here is a gist with the raw data. |
|
Could be us trying to see the size of some of the images, which we should not be doing by default. |
|
@baude PTAL |
|
I also see |
|
I seem to have inadvertently reproduced this again, and now I'll try to not fix it :) It's worse now: $ time podman images | wc -l
179
real 9m0.983s
user 13m5.841s
sys 0m15.668sSo 179 images, .local/share/containers is ~51G. Let me know what I can test to nail this down! |
carlpett
changed the title
|
@vrothberg @mheon @rhatdan To add some more observations, this is also slowing down other operations that interact with the image store, such as build and pull. For example, building the below Dockerfile takes 2 seconds if I create a new user and run as that one (so there is absolutely no previous podman state), while it takes 3.5 seconds on my own user. It seems to be somehow proportional to the "complexity" of the image (ie most builds are delayed significantly longer than 1.5 seconds) FROM alpine
RUN echo hello > /fileThis is becoming a problem for my day-to-day work, and I probably can't keep the broken state around indefinitely. Please let me know if I can check something. |
|
Thanks for the ping, @carlpett. I will have a look asap. |
|
Out of curiosity, @carlpett how large is your boltdb file created by podman? |
My installation just took 18s to list 56 images, with boltdb of 2MB (2097152 bytes). |
|
How does |
Lengthy delay (see #6984 for more details - it took over 80s for ~160 images!), and then all output appears simultaneously. The duration of the delay does not seem to depend on the data being displayed - |
|
Could you install buildah and do a buildah images? This would tell us if the problem is in containers/storage or podman/libpod. |
|
do you observe the issue with rootfull and rootless? or just rootless? |
|
Curiously, today it is fast - podman lists the 186 images in 2.2s, and buildah in 1.7s. I'm not aware of having made any relevant changes since yesterday either, so not sure why this happened all of a sudden...
I only really use podman rootless, so I don't know if I'd have the same problem with rootful, I'm afraid. |
|
I'm running as root, but from a 64-bit chroot on a system with an otherwise 32-bit userland... It just took |
|
#7215 will eventually solve the issue. Thanks for everybody involved and for providing the details. |
I am currently experiencing a huge performance decrease as root. With #7215 it takes 7 sec to list 510 images (yesterday it was 0.3 sec) while a vanilla Podman took 3 min 31 sec (6 sec yesterday). @rhatdan @baude ... there's still something yet to be revealed. |
|
I'll take 7 seconds. |
|
FWIW my laptop is now back to its' slow self (without doing anything actively). 18m25s for 185 images. Anything I can check, or is the linked PR already certain to have found the underlying cause @vrothberg? |
#7215 is definitely a huge improvement. Feel free to try it out on your machine. However, yesterday I experienced something that has already been reported in this issue: seemingly random performance issues. For no apparent reason listing 510 images went up from 0.3 secs to 7 secs (with #7215) and from 6 sec to 3.5 mins with an unpatched Podman. I suspect there's still something going on, very likely disk-related in c/storage. |
|
Another theory is that another service may consume disk I/O for a certain while. The performance issue on my F32 machine was only temporary and went back to normal after 15 mins (estimated) without any interference. Maybe, F32 has some periodic disk checks running? @edsantiago do you know? |
|
I downloaded the build from the PR (https://cirrus-ci.com/task/5154042487767040), and here's the result: $ time Downloads/podman images | wc -l
186
real 0m6.974s
user 0m10.592s
sys 0m0.305sSo quite a bit better than 15-20 minutes, even though not as good as your sub-second timings. I'm looking at |
|
Thanks for testing, @carlpett! That's pretty good news. I'll run some analyses as well when I hit the issue again. I didn't expect it to disappear so quickly. |
|
Can we close this issue now? |
I prefer to keep it open until we've resolved the random performance decrease. We're way faster now but there's still something to unpack. |
|
Couldn't reproduce anymore. Listing 1210 images takes 0.71 seconds which is a decent performance. I am going to close but feel free to comment or create a new issue in case the seemingly random issues occur again. |
|
just faced this problem, but apparently |
|
I'm sorry but the problem persists here: $ time podman images | wc -l
48
real 0m13,713s
user 0m5,608s
sys 0m9,363spodman version
Version: 3.4.7
API Version: 3.4.7
Go Version: go1.16.15
Built: Thu Apr 21 15:14:26 2022
OS/Arch: linux/amd64And this is very uncomfortable with bash completion. Have I missed something ? |
|
There was another issue (#13755) which will be fixed with the upcoming Podman v4.1 release. |
|
This issue still exists. I'm using podman version |
Thanks for reaching out. Can you please open a new issue for that? |
github-actions
bot
added
the
locked - please file new issue/PR
/kind bug
Description
The bash_completion for
podman runis very slow. Running withset -x, it seems the longest operation is listing all images:I have ~460 images locally at this time (according to
podman images | wc -l, so quite a lot of<none>and versions sharing underlying layers), andpodman imagestakes takes 20 seconds:Note that this happens even when I already supplied the image, eg
podman run busybox -- <tab>or when an image is not the correct/relevant completion, egpodman run -v <tab>.Since the terminal just appears to freeze unless you know what is happening, this is pretty disruptive. I'm not sure if 460 images should be considered excessive? Otherwise it might make sense to either not complete with the images output, or investigate if
podman imagesshould be much faster?Steps to reproduce the issue:
podman run <tab>Describe the results you received:
Waiting.
Describe the results you expected:
Less waiting.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Additional environment details (AWS, VirtualBox, physical, etc.):
Physical
The text was updated successfully, but these errors were encountered: