podman system prune: Inconsistent "Total reclaimed space"

[2xB]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman system prune --all produces a message of Total reclaimed space: 273.2GB, while my file system only shows 45.0 GB less (and it never used that much for container storage). There exists a reproducible script to test this.

This was discussed in #8831 .

(I am not 100% certain that the bug was first seen with the --all keyword, but the demonstration below shows this effect with --all, making it very likely that this was also how I initially saw that error.)

Steps to reproduce the issue:

This is reproducibly packed into a single bash script, using a podman-in-podman setup to make this reproducible and relatively clean without interfering with any pre-existing containers when executing podman system prune. Just for the record, I'm using rootless Podman 3.1.0 on Manjaro Linux and inside the Fedora container is a Podman 3.1.2 during my tests. But theoretically this script should work on any system without interference with existing podman containers.

The script sets up a podman image issue8831_2 with some podman image files inside that are deleted using podman system prune -f and podman system prune -f --all, first in that order (marks 1 and 2) and then - rolling back to issue8831_2 - in the other order (marks 3 and 4).

The script (click to expand)

#!/bin/bash

# Build podman container to work in
podman build -t issue8831_1 - << EOF
FROM fedora:34

RUN dnf update -y && dnf install -y podman nano

RUN sed -i -e 's,driver = "overlay",driver = "vfs",g' /etc/containers/storage.conf && \
  rm -rf /var/lib/containers

RUN podman pull fedora:34
EOF

# Create podman files in that container, commit as issue8831_2
podman run -it --security-opt seccomp=unconfined --privileged --name issue8831_1c issue8831_1 <<EOF
cat << EOD > Dockerfile
FROM fedora:34 as target1

RUN fallocate -l 0.5GB largefile1

RUN touch smallfile1

FROM fedora:34 as target2

RUN fallocate -l 0.5GB largefile2

RUN touch smallfile2

FROM target1 as result1

COPY --from=target2 smallfile2 .

FROM fedora:34 as result2

COPY --from=target2 smallfile2 .
COPY --from=target1 smallfile1 .
EOD

podman build --target result1 -t r1 .
podman build --target result2 -t r2 .
exit
EOF
podman commit issue8831_1c issue8831_2

# Delete podman files, way 1
podman run -it --security-opt seccomp=unconfined --privileged --rm issue8831_2 <<EOF
echo "Mark 1"
podman system prune -f
echo "Mark 2"
podman system prune -f --all
exit
EOF

# Delete podman files, way 2
podman run -it --security-opt seccomp=unconfined --privileged --rm issue8831_2 <<EOF
echo "Mark 3"
podman system prune -f --all
echo "Mark 4"
podman system prune -f
exit
EOF

# Cleanup
podman rm issue8831_1c
podman rmi issue8831_2
podman rmi issue8831_1

Describe the results you received:
The result - cleaned a bit:

Mark 1
[root@d4d6a28e17f0 /]# podman system prune -f
552c1a027388a2dbc95c6eaccee2e9d95ca0b3c0f17bb61af8950fec38b0c2cf
Deleted Images
0e1e85c3af4edf74c5f4731991789561a2bfabec9e0ee2e797ae33742f9f095f
Total reclaimed space: 686.5MB

Mark 2
[root@d4d6a28e17f0 /]# podman system prune -f --all
Deleted Images
registry.fedoraproject.org/fedora:34
e32001e126b4de19d3436c0358124c9604a8f13f556e944137706f96bffeb15c
07877da981cceb7bd0064af6563c800c6e91c166265671870026ccd16c663879
localhost/r1:latest
abccf02e337b261341f0fb84d7ebd7273c2b14946e7e917c4ed365fa0516e64d
localhost/r2:latest
Total reclaimed space: 2.619GB

# ---

Mark 3
[root@e45d5fc83e68 /]# podman system prune -f --all
Deleted Images
registry.fedoraproject.org/fedora:34
e32001e126b4de19d3436c0358124c9604a8f13f556e944137706f96bffeb15c
07877da981cceb7bd0064af6563c800c6e91c166265671870026ccd16c663879
552c1a027388a2dbc95c6eaccee2e9d95ca0b3c0f17bb61af8950fec38b0c2cf
0e1e85c3af4edf74c5f4731991789561a2bfabec9e0ee2e797ae33742f9f095f
localhost/r1:latest
abccf02e337b261341f0fb84d7ebd7273c2b14946e7e917c4ed365fa0516e64d
localhost/r2:latest
Total reclaimed space: 3.992GB

Mark 4
[root@e45d5fc83e68 /]# podman system prune -f
Deleted Images
Total reclaimed space: 0B

It is clear: 686.5MB + 2.619GB != 3.992GB (+ 0B )

Describe the results you expected:

Something consistent.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Outside the container:
Version:      3.1.0
API Version:  3.1.0
Go Version:   go1.16.2
Git Commit:   9f09fb62cba8f44c18eda84db3e72aab3ae76046-dirty
Built:        Wed Mar 31 16:46:48 2021
OS/Arch:      linux/amd64

Inside the container:
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16
Built:        Thu Apr 22 13:11:28 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

Output (click to expand)

``` Inside the container: arch: amd64 buildahVersion: 1.20.1 cgroupManager: systemd cgroupVersion: v1 conmon: package: conmon-2.0.27-2.fc34.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ' cpus: 8 distribution: distribution: fedora version: "34" eventLogger: journald hostname: 13d4fc8a28bc idMappings: gidmap: null uidmap: null kernel: 5.9.16-1-MANJARO linkmode: dynamic memFree: 207339520 memTotal: 8088653824 ociRuntime: name: crun package: crun-0.19.1-2.fc34.x86_64 path: /usr/bin/crun version: |- crun version 0.19.1 commit: 1535fedf0b83fb898d449f9680000f729ba719f5 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true selinuxEnabled: false slirp4netns: executable: "" package: "" version: "" swapFree: 13671735296 swapTotal: 17179865088 uptime: 37h 54m 35.03s (Approximately 1.54 days) registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /etc/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: vfs graphOptions: {} graphRoot: /var/lib/containers/storage graphStatus: {} imageStore: number: 0 runRoot: /run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 3.1.2 Built: 1619097088 BuiltTime: Thu Apr 22 13:11:28 2021 GitCommit: "" GoVersion: go1.16 OsArch: linux/amd64 Version: 3.1.2 ```

Package info (e.g. output of rpm -q podman or apt list podman):

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Container, see script above.

[2xB]

(Discussed before in #8831 with @bblenard, @mheon and @rhatdan)

[vrothberg]

Thanks for reaching out!

I have high hopes that this should be fixed with #10147.

[bblenard]

@2xB I should have done this way earlier but I used your test script and tested with the changes @vrothberg made and I got results that seem to be correct. So I'm guessing / hoping the issue is resolved :)

# Build images with 2xB's dockerfile
➜  podman git:(master) ✗ bin/podman build --target result1 -t r1 .
...
➜  podman git:(master) ✗ podman build --target result2 -t r2 .
...
➜  podman git:(master) ✗ bin/podman system prune -f                                                                                                                                                                                           
Deleted Images                                                                                                                                                                                                                                
ee42716e17b3aea181029fe207b64bbf5ebaf5e5c1402661b498439d77292dec                                                                                                                                                                              
270c9b5ee077142741de6d697f8058ecd696e35af4089ddfb7090b70cf7170f7                                                                                                                                                                              
e1d3da56967be28ef932eb7de44fb5b5a812aca4535bb6b5996a46a8a960f358                                                                                                                                                                              
e469336ce7645e5c307dc0f6678fce0d011d18e25350fd2f5dad4422d63687dc                                                                                                                                                                              
b429a13d05c005797a0c1012e45615a9d650924fd913283dd5d72f5aa6a53754                                                                                                                                                                              
Total reclaimed space: 2.919GB                                                                                                                                                                                                                
➜  podman git:(master) ✗ bin/podman system prune -f --all                                                                                                                                                                                     
Deleted Images                                                                                                                                                                                                                                
aadc32e2a62610a251826bd9a5a0337bf84f62aeee982cb555ac0fa106c30aa2                                                                                                                                                                              
3567369c671193f96f057f76e3e136ecbd3fdc7065019cc3dd6ed5a96894f128                                                                                                                                                                              
39aec6c9e7fe45092887a52f3d31b1076892d406ae43d9719eb1a694b72f65a6                                                                                                                                                                              
9f728974f6192c84506e53b791eb57171b7c02e59db6b9b5ba1f66e5a85f649b                                                                                                                                                                              
Total reclaimed space: 1.059GB 
# Build images with 2xB's dockerfile again
➜  podman git:(master) ✗ bin/podman build --target result1 -t r1 .
...
➜  podman git:(master) ✗ podman build --target result2 -t r2 .
...                                                          
➜  podman git:(master) ✗ bin/podman system prune -f --all          
Deleted Images
3567369c671193f96f057f76e3e136ecbd3fdc7065019cc3dd6ed5a96894f128
1b547a66c2c93c3a0bc802d9649b0aac3cf2ce1ff9d28abb32c9a01e92529018
fe52b4d163005826f6e8b7a1bb63af3d4bcd6db1fa03b7901245b46dc40f917b
b32bfc7d3a035a05afde270cd08e7cff3fc0613ff73cf423c86c2abbeaaf6231
bbac351fa8c20403fa4198090dda1cab260dbf3ad42cb5c2f230751b7f5d041f
8b3c952899c3069167c62ac47b209904938f4ff9f18ee5e9f223843b8540239a
c9ae03eb73f0caa19940c16e683936675d2731f95c9356d5a71fc6a62acaca04
98ac081518b564cf2a8868ee47cdfa015a86aacabcc637ad62a9fae930254ebf
Total reclaimed space: 3.97GB

1.059GB + 2.919GB = 3.97800

[vrothberg]

Thanks for checking! Apologies for not getting back to the issue after #10147 was merged.