"layer not known" transient error after interrupting a pull

[martinetd]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Follow-up on #8148 : with today's master, the error is now transient, but it's still scary to unaware users so simple ^C should probably not cause this.
As said there it's more of an UX issue than an actual problem

Steps to reproduce the issue:

1. alternate between podman pull smth, interrupting it, and rmi the freshly fetched image. Full reproducer in It's baaaack: podman images: Error: top layer [...] not found in layer tree #8148 (comment)

Describe the results you received:
Sometimes get this error (only once, state fixes itself after running it)

$ ./bin/podman --root /tmp/podmanroot rmi localhost:5000/a:2
WARN[0000] Top layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f of image e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2 not found in layer tree. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f not found in layer. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Top layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f of image e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2 not found in layer tree. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f not found in layer. The storage may be corrupted, consider running `podman system reset`. 
Error: 1 error occurred:
	* layer not known

Describe the results you expected:

pull should not be able to leave the directory in that state? I don't know.

[rhatdan]

@umohnani8 Just fixed this in container storage, I believe. Urvashi can you confirm?

[umohnani8]

This looks different, I fixed the symlinks not found issue. This actually may be fixed by the pull improvements that @vrothberg is working on?

[vrothberg]

The pull improvement won't help in this case. The problem is that rmi needs to (still) be made more robust against a corrupted storage.

[vrothberg]

#9617 is (hopefully) the last missing piece. We already ignored unknown-image errors so it seems logical to extend it to ignore unknown-layer errors as well - those may be caused by an incomplete pull/commit.

[martinetd]

Thanks! I'll test tomorrow

[martinetd]

tomorrow got a bit late, here's what it looks like now:

$ ./bin/podman --root /tmp/podmanroot pull localhost:5000/a:2
Trying to pull localhost:5000/a:2...
Getting image source signatures
Copying blob fc87b077f4bb skipped: already exists  
Copying blob 353a677e0e56 done  
Copying config e1b17d5af1 done  
Writing manifest to image destination
Storing signatures
^C
$ ./bin/podman --root /tmp/podmanroot rmi localhost:5000/a:2
WARN[0000] Top layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f of image e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2 not found in layer tree. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f not found in layer. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Top layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f of image e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2 not found in layer tree. The storage may be corrupted, consider running `podman system reset`. 
WARN[0000] Layer ca84fd7b90547cb7f2b5adf21c4d5270becc15b855548277feccb31732917e8f not found in layer. The storage may be corrupted, consider running `podman system reset`. 
Untagged: e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2
Deleted: e1b17d5af16529f22664f6c8f7328cad67bf10e3cabe9f7e13d75f14b992d7a2

$ echo $?
0

So the hard error is gone; I guess it's fair to keep showing the warnings. Thanks!

[vrothberg]

@martinetd, thanks a lot for sharing and for your patience. I appreciate your feedback and interactions!