Error: readlink /var/lib/containers/storage/overlay: invalid argument when using Podman on WSL

[PavelSosin-320]

/kind bug

Description

Podman image inspect fails with Error: readlink /var/lib/containers/storage/overlay: invalid argument
Landscape:
Podman 2.2 on Ubuntu 20.04 distro with "Kube" and Podman installed from Ubuntu repository
Steps to reproduce the issue:
0. Install Podman on Ubuntu and check if it works - indeed, works

1. Create pod containing a single nginx container pod and test it -> everything is fine
2. Create a single Microsoft VSCode dev-container pod and test it -> everything is fine
3. Add a side-car node container to the dev-container pod with corresponding shared volume mounts and common workdir -> everything is working fine
4. Analyze nginx image using podman image inspect nginx -> OK
5. Analyze node image using podman image inspect node -> Error: readlink /var/lib/containers/storage/overlay: invalid argument
6. pull node image again using podman pull node and analyze it using podman image inspect nginx -> the same error
7. Pull node image using buildah: buildah pull node and analyze it using buildah inspect node -> wonderfull, everything is OK
8. Back to podman: podman image inspect nginx -> OK
   Describe the results you received:
   Error: readlink /var/lib/containers/storage/overlay: invalid when analyzing pulled image using podman image inspect

Describe the results you expected:
The same result as when I analyzing nginx image or node image using buildah

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.14
Built:        Thu Jan  1 02:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

root@MSI:~# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: MSI
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.72-microsoft-standard-WSL2
  linkmode: dynamic
  memFree: 11665215488
  memTotal: 12561657856
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 3221225472
  swapTotal: 3221225472
  uptime: 2h 33m 20.35s (Approximately 0.08 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 20
    paused: 0

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
WSL2 distro
Hint: Podman runs inside WSL VM and WSL file system. It has tiny rootfs inside a compressed vhdx file containing ext4 volume.
Most of the storage is in the /mnt/c,d,... filesystems:
drvfs 139271164 139119476 151688 100% /mnt/c
drvfs 87256060 72047908 15208152 83% /mnt/d
podman container inspect works OK
podman image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/node latest bd4dba13afd5 3 days ago 963 MB
2d840844f8e7 3 weeks ago 963 MB
docker.io/library/nginx
I can do podman image inspect for every image except node official image. What is wrong with it?

[vrothberg]

Thanks for reaching out, @PavelSosin-320 ! That's a curious issue. Could you share the debug logs (podman --log-level=debug image inspect ...)?

[PavelSosin-320]

I suppose you need only the tail of the log after network initialization:
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/node:latest"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@bd4dba13afd50cc847e97b58c02202103704ba2b57f59dad132cf08b1c329f3c"
DEBU[0000] exporting opaque data as blob "sha256:bd4dba13afd50cc847e97b58c02202103704ba2b57f59dad132cf08b1c329f3c"
Error: readlink /var/lib/containers/storage/overlay: invalid argument
What [overlay@/var/lib/containers mean? [overlay@/var/lib/containers/storage+/var/run/containers/storage]@bd4dba13afd50cc847e97b58c02202103704ba2b57f59dad132cf08b1c329f3c"? It is container storage but I'm asking "image inspect". I that afraid something wrong in the Podman image storage. I don't like what podman info says me:
imageStore:
number: 9
runRoot: /var/run/containers/storage

This may explain while buildah that has its own image store works, including buildah --type=image inspect variant.
Unfortunately, I didn't succeed to run buildah --log-level=3 to compare logs.

[vrothberg]

Can you try removing the image and repull?

[PavelSosin-320]

@vrothberg I did it - see step 6.
I suspect that it is a plain bug. when I ask ls -al /var/lib/containers/storage/I see:
drwx------ 74 root root 12288 Dec 17 10:02 overlay
drwx------ 23 root root 4096 Dec 17 10:02 overlay-containers
drwx------ 11 root root 4096 Dec 16 08:25 overlay-images
drwx------ 2 root root 12288 Dec 17 10:02 overlay-layers

So, podman image inspect shall look into overlay-images, instead of overlay-containers.
Its content is:
ls -al /var/lib/containers/storage/overlay-images/
total 68
drwx------ 11 root root 4096 Dec 16 08:25 .
drwx------ 10 root root 4096 Dec 6 13:00 ..
drwx------ 2 root root 4096 Dec 9 15:02 2d840844f8e7594a542b30eaed0a3451f5d84b9f85d091d09abc8e0ae75c48e4
drwx------ 2 root root 4096 Nov 2 11:13 5336a27a9b1ff3f6715422fcccd188f931fc8775f27d4c75aa6f444748b990c2
drwx------ 2 root root 4096 Sep 13 11:08 7e4d58f0e5f3b60077e9a5d96b4be1b974b5a484f54f9393000a99f3b6816e3d
drwx------ 2 root root 4096 Sep 13 11:07 80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c
drwx------ 2 root root 4096 Sep 13 12:27 9ad97b8dcd324318f70bfb3675f31a6a9fe0cc8d4048f8737d1fb6c885ad72d8
drwx------ 2 root root 4096 Oct 20 08:55 a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e
drwx------ 2 root root 4096 Dec 6 13:00 bc9a0695f5712dcaaa09a5adc415a3936ccba13fc2587dfd76b1b8aeea3f221c
drwx------ 2 root root 4096 Dec 16 08:25 bd4dba13afd50cc847e97b58c02202103704ba2b57f59dad132cf08b1c329f3c
drwx------ 2 root root 4096 Nov 1 16:13 c4a263cbda7b75995bcdd234dadddaa41ce845150c9b6064c348a60b46279203

and cat /var/lib/containers/storage/overlay-images/images.json output indeed contains node image layers. (too big to attach)
Are some data is missed?

[baude]

@PavelSosin-320 im not able to trip this bug ... i suspect it might be a difference in our commands. could you provide an exact set of podman commands to the point of tripping the first error?

[PavelSosin-320]

I just created the single container pod from VSCode dev-container image using new:my-pod-name
to use this container as a main service
Then I added another container using the official node.js image using --pod option
Podman eagerly pull nodejs image and created 2nd container in the same pod.
Node.js side-car container inherits volume mounts from the main container. Both containers have the same workdir.
Then I run my pod and both containers started, execute podman exec correctly, and can be inspected using Podman inspect
The total image count correctly reflects the number of pulled images.
When I inspect the dev-container image the output appears and it is OK
When I tried to inspect nodejs image pulled 2nd Podman doesn't see it. Th error message "Error: readlink /var/lib/containers/storage/overlay: invalid argument" appears.
The containers folder in /var/lib/containers contains the correct number of containers.
This image indeed missed in ls -al /var/lib/containers/storage/overlay-images/ output
On other hand, the /var/lib/containers/storage/overlay-images/images.json contains information about this image.
In the scenario when the container is added to the existing pod, the running container exists but its image in the local repository disappeared.
Of course, when I run or pull nodejs container separately its image is preserved and "inspectable"

[PavelSosin-320]

On my new Podman 2.0.6 the same scenario:
Works if the 2nd container is "official" Python - I can do Podman image inspect Python, but not for nodejs ????

[PavelSosin-320]

Works OK Podman 2.0.6 on Ubuntu 20.10 Groovy WSL distro

[CherrieYu]

I got the same issue，how did you fix it？thanks

[FlaviusHouk]

I've also faced this issue. It happened on virtual machine after I've accidently rebooted it during build process. After that I was not able to inspect the state of cached image layer.

Podman version

Version:      3.4.1
API Version:  3.4.1
Go Version:   go1.16.8
Built:        Wed Oct 20 17:31:56 2021
OS/Arch:      linux/amd64

[rhatdan]

Please update to podman 4.* there have some fixes in this area.

[idleroamer]

@rhatdan is there any possibility to cherry-pick the fixes into 3.x branch? Is it more of fundamental architectural rework or something potentially surgically transplant-able?

[mheon]

The 3.x releases are out of support at this point, so there will be no further bugfixes made there

[idleroamer]

@mheon thanks a lot for reply, is there nevertheless any way to find out which commits roughly fixed the issue, so we can cherry-pick manually?