tar 1.35.1 (rawhide): podman import, with modified tar: duplicates of file paths not supported

[edsantiago]

Reproducer (copy/paste/run):

#!/bin/bash

i=quay.io/libpod/testimage:20221018

bin/podman create --name a $i
bin/podman export -o /tmp/a.tar a
tar --delete -f /tmp/a.tar home/podman/pause
bin/podman import /tmp/a.tar b

(I used podman-from-source for convenience, but also fails with podman-4.5-from-rpm).

Import barfs with:

e31ce32f59ee1840830d534efa2fa4f38473b2e56c7410834523a5c554b1719a
Getting image source signatures
Copying blob 51098d81131b done   | 
Error: writing blob: adding layer with blob "sha256:51098d81131bac1a1a8539c68e4389e15fe1a800e4922eecd9cb11b0f51d9386": processing tar file(): duplicates of file paths not supported

tar-1.35-1.fc39.x86_64, works perfectly fine with 1.34-8.fc39

Almost certainly a tar bug, not podman, but (1) tracking here because it is blocking #18612, and (2) I have no idea what podman is doing with tar, or how to write a tar reproducer. I leave that to someone familiar with podman import.

[nalind]

This appears to be a bug in tar. The reproducer is:

podman create --name foo quay.io/libpod/testimage:20221018
podman export -f /tmp/a.tar foo
tar --delete home/podman/pause -f /tmp/a.tar
podman import -f /tmp/a.tar

Running tar tf on /tmp/a.tar displays unexpected changes. Running tar tf on the file, both before and after, and diffing the outputs, should help to clarify them. Running tar --delete as a filter, with stdin redirected from the file produced by podman export, and with stdout redirected to a file which will later be fed to podman import, seems to work around it.

[edsantiago]

Recipe: run the above podman export, then tar tf. Run tar --delete, and once again tar tf.

With tar 1.34

@@ -234,3 +234,2 @@
 home/podman/
-home/podman/pause
 home/podman/testimage-id
@@ -770,5 +769,2 @@
 usr/share/apk/keys/x86_64/[email protected]
-usr/share/apk/keys/x86_64/[email protected]
-usr/share/man/
-usr/share/misc/
 usr/share/udhcpc/

With tar 1.35.1

@@ -234,3 +234,2 @@
 home/podman/
-home/podman/pause
 home/podman/testimage-id
@@ -773,2 +772,14 @@
 usr/share/misc/
+usr/share/man/
+usr/share/misc/
+usr/share/udhcpc/
+usr/share/udhcpc/default.script
+var/
+var/cache/
+var/cache/apk/
+var/cache/misc/
+var/empty/
+var/lib/
+var/lib/apk/
+var/lib/arpd/
 usr/share/udhcpc/

Looks like 1.34 has a bug where it deletes random entries (symlinks maybe?). I am not concerned about that.

1.35, though, adds duplicates. This is what then causes podman (presumably the go tar library) to barf with "duplicates not allowed".

I can't make a simple reproducer. Ideally I could file a tar bug with a recipe for tar cf, tar --delete, boom. But I can't. I can't even make a reproducer using the extracted image:

# tar xf /tmp/podman-exported-image.tar
# tar cf /tmp/foo.tar .
# tar --delete ...
# tar tf   <----- deletes only what I ask, does not add any new duplicates

Giving up for today. Here's the tar git repo should anyone feel inclined to bisect.

[edsantiago]

Filed bz2230127

[edsantiago]

The cause is e89c7a4.

[cevich]

Oh wow @edsantiago you figured it out! Nice job! I don't think anybody would have followed through on that for maybe weeks or months. Good on you 👍

[rhatdan]

Can we close this or do we need to wait for the package to get updated on Rawhide?

[edsantiago]

Package was not in updates-testing this morning, but it's out of bodhi so I guess it's safe to close.

[cevich]

FWIW, I just finished building & releasing updated CI VM images this afternoon. Not sure if they include this new tar or not: containers/automation_images#294

[edsantiago]

If podman CI passes, tar is good.